<p>This part of the handbook is intended for people maintaining the troggle software. Day to day cave recording and surveying tasks are documented in the expo "survey handbook"
<p>The first thing to do is to read: "<ahref="/expofiles/documents/troggle/troggle_paper.pdf"download>Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis, CUCC.</em>
<li>that troggle is just one of several cave-survey management online software systems. CUCC EXPO is not the only caving expedition with a substantial nerd community.<br/><br/>
<p><em>Assumptions</em> (points to necessarily agree upon)
<ol>
<li>Let's NOT try to design a generic catalogue for storing all kind of data about caves of the whole world, intended for every kind of user (sports, exploration, science). Let's just settle for a generic framework. Let geeks in individual countries or individual communities write their tools operating within this framework.
<li>Let's try make it available for the layman, but still well-playable for the geeks.
<li>Let's rely on already existing, popular technologies. Let's keep it open source and multiplatform. Let's try not to reinvent the wheel.
<li>Let's not assume everyone has an Internet connection while working with their data.
<li>Let's version-control as much as possible.
<li>Let's support i18n - let's use UTF-8 everywhere and cater for data in many languages(entrance names, cave descriptions, location descriptions etc.)
<h2>The data management system conventions bit</h2>
<p>This is likely to change with structural change to the site, with style changes which we expect to implement and with the method by which the info is actually stored and served up.</p>
<p>... and it's not written yet, either :-)</p>
<ul>
<li>Structure</li>
<li>Info for each cave – automatically generated by <tt>make-indxal4.pl</tt></li>
<li>Contents lists & relative links for multi-article publications like journals. Complicated by expo articles being in a separate hierarchy from journals.</li>
<li>Translations</li>
<li>Other people's work - the noinfo hierarchy.</li>
<li>Style guide for writing cave descriptions: correct use of boldface (<em>once</em> for each passage name, at the primary definition thereof; other uses of the name should be links to this, and certainly should not be bold.) </li>
<p>CUCC still has an archive list of things that at one time were live tasks:
from <ahref="https://camcaving.uk/Documents/Expo/Legacy/Misc/Troggle%20-%20Cambridge%20University%20Caving%20Club.htm">camcaving.uk/Documents/Expo/Legacy/Misc/...</a> and that page is reproduced in the table below (so don't worry if the URL link goes dark when CUCC reorganise their legacy pages).
<p>Troggle is a system under development for keeping track of all expo data in a logical and accessible way, and displaying it on the web. At the moment, it is [no longer] under development at <u>http://troggle.cavingexpedition.com/</u>
<tt>But note that this is Aaron's version of troggle, forked from the version of troggle we use. Aaron uses this for the <ahref="https://expeditionwriter.com/new-expedition-to-mount-erebus-antarctica/">Erebus expedition</a>.</tt>
<td><p>Start at the front page, <arel="nofollow"class="external autonumber"href="http://expo.survex.com/expedition/2007">troggle.cavingexpedition.com/ [1]</a> and click to logbook for year. The logbooks have been parsed back to 1997. </p></td>
<td><p>Done; see <arel="nofollow"class="external autonumber"href="http://expo.survex.com/survexfile/caves/264">troggle.cavingexpedition.com/survey/caves/264 [2]</a></p></td>
<td><p>Yes; minimal. surveys.csv produced an html table of whose surveys were not marked “finished”</p></td>
<td><p>Yes. Makes table of surveys per expo which shows exactly what needs doing. Displays scans. Integrated with survex, scanner software, and tunnel.</p></td>
<td><p>See it at <arel="nofollow"class="external free"href="http://expo.survex.com/survey_scans/">troggle.cavingexpedition.com/survey</a> . Be sure to try a recent year when we should have data. Survex, scanner, and tunnel integration still needs doing.</p></td>
<td><p>Done; see <arel="nofollow"class="external free"href="http://expo.survex.com/expedition/2007">troggle.cavingexpedition.com/calendar/2007</a> (replace 2007 with year in question)</p></td>
<td><p>Everything can be edited through admin, at <arel="nofollow"class="external free"href="http://expo.survex.com/admin/">troggle.cavingexpedition.com/admin</a> . Ask aaron, martin, or julian for the password if you want to have a look / play around with the admin site. Any changes you make will be overwritten. Eventually, data entry will probably be done using custom forms.
<ahref="http://www.uisic.uis-speleo.org/contacts.html">www.uisic.uis-speleo.org/contacts.html</a> change link. no-one looks for list of databases under 'contacts'
graziano ferrari northern italy list (access + google earth)
</pre>
<h3>Wookey's notes on things to do</h3>
from <ahref="http://wookware.org/software/cavearchive/goliczmail">wookware.org/software/cavearchive/goliczmail</a>
<pre>
Generally I'd like to find some people (geeks) that share these technical
ideas: (1) store things in a file system, (2) use XML, (3) do not aim too high
(do not try designing a general system for handling all caving-related data
for the whole world).
If I could find some people that agree with this, then we could try to reach a
compromise on:
(1) how do we store our data in a file system,
(2) how do we use this XML (let's do a common spec, but keep it simple)
[What Rad has misunderstood here is that the database is not for speed. We use it mostly so that we can manage 'referential integrity' i.e. have all the different bits of information match up correctly. While the total size of the data is small, the interrelationships and complexity is quite large. From the justification for troggle: <p>"A less obvious but more deeply rooted problem was the lack of relational information. One table named folk.csv stored
names of all expedition members, the years in which they were present, and a link to a biography page. This was great
for displaying a table of members by expedition year, but what if you wanted to display a list of people who wrote in the
logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that
list has been recorded in the logbook, but there is no way to access it because there is no connection between the
person's name in folk.csv and the entries he wrote in the logbook".
<p>And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks,
survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent]
</em>
</div>
<p>Similarly I see little gain from doing the html - python himera
[This vastly underestimates the number of things that troggle does for us. See "<ahref="/expofiles/documents/troggle/troggle_paper.pdf"download>Troggle: a novel system for cave exploration information management</a>". And a VM is not required to run and debug troggle.
Sam has produced a docker variant which he uses extensively.
<p>Troggle today has 8,200 lines of python (including comments and blank lines), plus 600 in imagekit and 200 in flatpages. 2,200 of those are in the parsers. Django itself has a lot more, including integration with TinyMCE in-browser HTML editor. </em>]
[The effort estimate is similarly a gross underestimate because (a) he assumes one script per page of output, forgetting all the core work to create a central consistent dataset, and (b) he is missing out most
of the functionality we use without realizing it because it is built into
django's SQL system, such as multi-user operations.
<p><spanstyle="color:red">Eventually we will have to migrate from django of course</span>, as it will eventually
fail to keep up with the rest of the world. Right now we need to get ourselves onto python3
so that we can use an LTS release which has current security updates. This is
<ahref="https://docs.djangoproject.com/en/3.0/internals/release-process/#supported-versions-policy">more urgent for django</a> than for Linux. In Ubuntu terms we are on 18.04 LTS (Debian 10) which has no free maintenance updates from 2023. <spanstyle="color:red">We should plan to migrate troggle from django to another framework in about 2025. See stroggle below.</span>]
</em>
</div>
Things this [Rad's] solution doesn't solve:
<ul>
<li> no webpage-based 'wizzard' for creating caves and such (did people use it
[Creating a cave description for a new cave, and especially linking in images,
is currently so difficult that only a couple of people can do it. Fixing this is
a matter of urgency. No one should have to imagine where the path to a file will be but isn't now.
We need a file uploading system to put things in the right place; and this would help photos too.]
</em>
</div>
</div>
<h3>stroggle</h3>
<p>At one time Martin Green attempted to reimplement troggle as "stroggle" using <ahref="https://www.fullstackpython.com/flask.html">flask</a> instead of Django at
<ahref="https://en.wikipedia.org/wiki/Gitorious">git@gitorious.org:stroggle/stroggle.git</a> (but gitorious has been deleted).</p>
<p>A copy of this project is archived by Wookey on <ahref="http://wookware.org/software/cavearchive/stroggle/">wookware.org/software/cavearchive/stroggle/</a>.
<p>There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "<ahref="https://gitorious.org/">gitorious valhalla</a>"<br/>
but note that this domain has an expired certificate so https:// complains.
<p>The schema for stroggle is <ahref="http://wookware.org/software/cavearchive/stroggle/git/exampledata/expo/schema.json">a schema.json file</a>. See the comparable <ahref="datamodel.html">troggle schema file</a> which is indeed horrendously bigger.
<h3><aid="arch">Archived updates</a></h3>
<p>Since 2008 we have been keeping detailed records of all data management system updates in the version control system.
Before then we manually maintained <ahref="../computing/update.html">a list of updates</a> which are now only of historical interest.
<p>A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article <ahref="c21bs.html">Taking Expo Bullshit into the 21st Century</a> is archived here.
