Restructuring troggle notes and creation of troggle documentation directory in the handbook

This commit is contained in:
Philip Sargent 2020-04-02 14:25:58 +01:00
parent 1f24e303ef
commit a5447cf626
14 changed files with 519 additions and 139 deletions

View File

@ -89,7 +89,7 @@ editing and keeps track of all changes so we can roll back and have branches if
<li><a href="/repositories/home/expo/loser/graph/">loser</a> - the survex cave survey data (hg)</li>
<li><a href="/cgit/drawings/.git/log">drawings</a> - the tunnel and therion cave data and drawings (git)</li>
<li><a href="/repositories/home/expo/expoweb/graph">expoweb</a> - the website pages, handbook, generation scripts (hg)</li>
<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="computing/troggle-ish.html">notes on troggle</a> for further explanation (git)</li>
<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="troggle/trogintro.html">notes on troggle</a> for further explanation (git)</li>
</ul>
<p>We have migrated two of these to git but the other two still use mercurial.
@ -103,20 +103,11 @@ below for details on that.</p>
<h3><a id="howitworks">How the data management system works</a></h3>
<p>Part of the data management system is static HTML, but quite a lot is generated by scripts and troggle (a web framework built using Django).
<p>Examples of troggle-generated pages from data:
<ul>
<li><a href="http://expo.survex.com/caves">expo.survex.com/caves</a> - list of caves surveyed and links to guidebook descriptions
<li><a href="http://expo.survex.com/pubs.htm">expo.survex.com/pubs.htm</a> - reports, accounts and logbooks
<li><a href="http://expo.survex.com/expedition/2018">expo.survex.com/expedition/2018</a> - Members on expo 2018: . Scroll down for a list of all the data typed in from survey trips.
<li><a href="http://expo.survex.com/survexfile/caves/">expo.survex.com/survexfile/caves/</a> - List of caves with all the surveys done for each.
<li><a href="http://expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx">expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx</a> - CUCC cave survey data from 1983 in Schnellzughohle.
<li><a href="http://expo.survex.com/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="http://expo.survex.com/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
<p>Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the <a href="troggle/trogintro.html">troggle intro</a>.
<p>Anything you check in which affects cave data or descriptions won't appear on the site until
the data management system update scripts are run.
This happens automatically every 30 mins, but you can also kick off a manual update.
This <em>should</em> happen automatically every 30 mins (not since 2017), but you can also kick off a manual update.
See 'The expoweb-update script' below for details.</p>
<p>Also note that the ::expoweb:: web pages and cave data reports you see on the visible website

View File

@ -8,8 +8,8 @@
<body>
<h2 id="tophead">CUCC Expedition Handbook - Online systems</h2>
<h1>Expo Online Systems Overview</h1>
<p>The online data and web publishing system (i.e. "the website") is now large and complicated with a lot of aspects.
This handbook section contains info at various levels:
<p>The online data and web publishing system (i.e. "the website") is now large and complicated.
The handbook contains info at various levels:
simple 'How to add stuff' information for the typical expoer,
more detailed info for cloning it onto your own machine for more significant edits,
and structural info on how it's all put together for people who want/need to change things.
@ -20,7 +20,7 @@ on how the cave data,
handbook and public website are constructed and managed.
It contains material which will be merged into this online systems manual.
<p>These pages listed below have been reviewed recently (2018), and a
<p>These pages listed below have been reviewed recently (2019), and a
fuller list of "How do I..." instruction pages are on <a href="index.htm">the handbook opening page</a>.
<ul>
@ -30,14 +30,16 @@ fuller list of "How do I..." instruction pages are on <a href="index.htm">the ha
<li><a href="survey/newcave.html">Recording a new cave discovery</a></li>
<li><a href="survey/status.html">Monitoring the status of the cave survey workflow during and after expo</a></li>
</ul>
<p>But the systems Manual is still being actively edited to extract and simplify documentation.
<p>The systems manual is being actively edited to extract and simplify documentation.
<ul>
<li><a href="computing/newyear.html">Manual: Creating a new 'year' in the system</a></li>
<li><a href="manual.html">Manual: Expo survey data maintenance manual</a></li>
<li><a href="manual.html#update">Manual: Updating the cave guidebook descriptions</a></li>
<li><a href="manual.html#manual">Manual: Expo software and server maintenance manual</a></li>
</ul>
<h2><a id="update">Updating the online systems - overview</a></h2>
<h3 id="troggle">Troggle - what it is</a></h3>
<p>Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the <a href="troggle/trogintro.html">troggle intro</a>.
<h3>Autogenerated pages</h3>
@ -59,7 +61,7 @@ using a text editor. The public <a href="exposerver.html">expo server</a> is on
when sitting at that laptop.
<p>It's important to understand that the pages you can edit by this method
are stored in a distributed version control system (see below). This stops us losing data and
are stored in a version control system (see below). This stops us losing data and
makes it very hard for you to screw anything up permanently, so don't
worry about making changes - they can always be reverted if there is a
problem. It also means that several people can work on the site on
@ -89,7 +91,7 @@ See <a href="manual.html#editthispage">these instructions for this tidy-up</a>
<h3 id="git">Version control system</a></h3>
<p>We use a distributed revision control system (git, and formerly mercurial) for all the important data.
<p>We use a distributed revision control system (<a href="https://git-scm.com/about/distributed">git</a>, and formerly mercurial) for all the important data. (Note that we just use git: not GitHub, not GitLab, just <a href="https://git-scm.com/about/distributed">git</a>.)
This means that many people can edit and merge their changes with the expo
server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut
and those who have returned to the UK. Also anyone who is up
@ -106,42 +108,6 @@ The same goes for holiday photographs and GPS logs.</p>
<p>In 2019 we had half our <a href="manual.html#repositories">version-controlled repositories</a> under mercurial and half under git.
The intention is to move entirely to git before the 2020 expo.
<h3 id="troggle">Troggle - what it is</a></h3>
<p>
Troggle is the software collection (not really a "package") based on <a href="https://www.djangoproject.com/">Django</a>
originally intended to manage all expo data in a logical and accessible way
and publish it on the web.
<p>Examples of troggle-generated pages from data:
<ul>
<li><a href="http://expo.survex.com/caves">expo.survex.com/caves</a> - list of caves surveyed and links to guidebook descriptions
<li><a href="http://expo.survex.com/pubs.htm">expo.survex.com/pubs.htm</a> - reports, accounts and logbooks
<li><a href="http://expo.survex.com/expedition/2018">expo.survex.com/expedition/2018</a> - Members on expo 2018: . Scroll down for a list of all the data typed in from survey trips.
<li><a href="http://expo.survex.com/survexfile/caves/">expo.survex.com/survexfile/caves/</a> - List of caves with all the surveys done for each.
<li><a href="http://expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx">expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx</a> - Cave survey data from 1983 in Schnellzughohle.
<li><a href="http://expo.survex.com/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="http://expo.survex.com/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
<p>[Note that /survey_scans/ is generated by troggle and is not the same thing as /expofiles/surveyscans/ at all.]
<p>Only a small part of troggle's original plan was fully implemented and deployed.
Many of the things it was intended to replace are still operating as a motley collection written by many different people in
several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).
Today troggle is used for only three things:
<ol>
<li>Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left
hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
<li>Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description
can do this by filling-in some online forms.
<li>Providing a secondary way of editing individual pages of the handbook and historic records pages
for very quick and urgent changes.
This is the "Edit this page" capability; see <a href="#editthispage">above for
how to use it</a> and how to tidy up afterwards.
</ol>
<p>See the <a href="computing/troggle-ish.html">notes on troggle</a> page
for how and why it was developed and what needs to be done.
<hr />
</body>

View File

@ -102,7 +102,7 @@ mentioned, or when they are "defined".</p>
<p>
You will type this description, and pass it on to someone more nerdy who
will file it in the right place. This will involve "creating a new cave" using the <a href="../computing/troggle-ish.html">troggle</a> system.
will file it in the right place. This will involve "creating a new cave" using the <a href="../troggle/trogintro.html">troggle</a> system.

View File

@ -86,7 +86,7 @@ You can scan what's in the repositories (read only) using your web browser:
<li><a href="/repositories/home/expo/loser/graph/">loser</a> - the survex cave survey data (hg)</li>
<li><a href="/cgit/drawings/.git/log">drawings</a> - the tunnel and therion cave data and drawings (git)</li>
<li><a href="/repositories/home/expo/expoweb/graph">expoweb</a> - the website pages, handbook, generation scripts (hg)</li>
<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="../computing/troggle-ish.html">notes on troggle</a> for further explanation (git)</li>
<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="../troggle/trogintro.html">notes on troggle</a> for further explanation (git)</li>
</ul>>

View File

@ -0,0 +1,43 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle Intro</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle - what you may need to know</h1>
<p>Troggle is the software which runs the the expo cave survey data management and website.
<ul>
<li>For day to day cave survey recording, see: the expo <a href="../survey/newcave.html">survey handbook</a>.
<li>For more in-depth survey data management: set up <a href="basiclaptop.html">your own laptop</a>.
<li>For the history of expo use of computers, see: <a href="../website-history.html">website and troggle history</a>.
<li>For troggle maintenance, see: the <a href="../troggle/trogmanual.html">troggle maintainers manual</a>.
</ul>
<h3 id="troggle">Troggle - what it is</a></h3>
<p>Troggle manages all cave and expo data in a logical and maintainable way
and publishes it on the web.
<p>
The troggle software is written and maintained by expo members.
<p>Examples of troggle-generated pages from data:
<ul>
<li><a href="/caves">expo.survex.com/caves</a> - list of caves surveyed and links to guidebook descriptions
<li><a href="/pubs.htm">expo.survex.com/pubs.htm</a> - reports, accounts and logbooks
<li><a href="/expedition/2018">expo.survex.com/expedition/2018</a> - Members on expo 2018: . Scroll down for a list of all the data typed in from survey trips.
<li><a href="/survexfile/caves/">expo.survex.com/survexfile/caves/</a> - List of caves with all the surveys done for each.
<li><a href="/survexfile/caves-1623/115/cucc/futility.svx">expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx</a> - Cave survey data from 1983 in Schnellzughohle.
<li><a href="/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
<hr />
Return to<br />
<a href="../onlinesystems.html">expo online systems overbiew</a><br />
</body>
</html>

View File

@ -2,18 +2,19 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle Notes</title>
<title>Handbook Troggle Intro</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle - what you may need to know</h1>
<h1>Troggle - Maintainers Manuals</h1>
<p>Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.
<p>You may have arrived here by accident when where you really need to be is <a href="../website-history.html">website history</a>.
<p>This page needs to be restructured and rewritten so that it describes these things:
<p>This part of the handbook is intended for people maintaining the troggle software. Day to day cave recording and surveying tasks are documented in the expo "survey handbook"
<p>This troggle manual describes these:
<ul>
<li>Day to day troggle tasks - usually during expo. i.e. links to the "survey handbook"
<li>Annual tasks: preparing for next year, finishing last year (troggle & scripts)
<li>Architectural documentation of how it all fits together & list of active scripts
<li>How to edit and maintain troggle itself. The code is public on repository <a href="http://expo.survex.com/repositories/">::troggle::</a>
@ -22,9 +23,7 @@
<br />
<tt><em>Everything here should be updated or replaced - this page just records a lot of unfinished ideas.
Most people will not want to read this at all. This is for speleosoftwarearcheologists only.</em>
</tt>
<p>This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it.
<h3 id="troggle">Troggle - what it is</a></h3>
@ -86,6 +85,10 @@ All Survex | Scans | Tunneldata | 107 | 161 | 204 | 258 | 264 | Expo2016 | Expo2
<li>Let's support i18n - let's use UTF-8 everywhere and cater for data in many languages(entrance names, cave descriptions, location descriptions etc.)
</ol>
<tt><em>Everything here should be updated or replaced - this page just records a lot of unfinished ideas.
Most people will not want to read this at all. This is for speleosoftwarearcheologists only.</em>
</tt>
<p>Two page preliminary design document for <a href="../../documents/caca_arch2.pdf">'caca' (Cave Catalogue) rev.2 2013-07-26</a> by Wookey (copied from http://wookware.org/software/cavearchive/caca_arch2.pdf)
<h3>stroggle</h3>
@ -354,16 +357,6 @@ Storage:
<hr />
<ul id="links">
<li><a href="../index.htm">Expedition Handbook</a>
<ul>
<li><a href="../survey/index.htm">Surveying guide</a> - Overview</li>
<li><a href="../look4.htm">Prospecting guide</a> &ndash; Overview</li>
<li><a href="../rescue.htm">Rescue guide</a></li>
</ul></li>
<li><a href="../../index.htm">Expedition Intro </a></li>
<li><a href="https://camcaving.uk">CUCC Home </a></li>
</ul>
</body>
</html>

View File

@ -0,0 +1,392 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle NOTES</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle - TO BE RESTRUCTURED</h1>
<p>Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.
<h2>NOTES - taken from elsewhere</h2>
<p>This part of the handbook is intended for people maintaining the troggle software. Day to day cave recording and surveying tasks are documented in the expo "survey handbook"
<p>This troggle manual describes these:
<ul>
<li>Annual tasks: preparing for next year, finishing last year (troggle & scripts)
<li>Architectural documentation of how it all fits together & list of active scripts
<li>How to edit and maintain troggle itself. The code is public on repository <a href="http://expo.survex.com/repositories/">::troggle::</a>
</ul>
<p>This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it.
Today troggle is used for only three things:
<ol>
<li>Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left
hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
<li>Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description
can do this by filling-in some online forms. (And managing all the cave suvey data to produce this.)
<li>Providing a secondary way of editing individual pages of the handbook and historic records pages
for very quick and urgent changes.
This is the "Edit this page" capability; see <a href="../onlinesystems.html#editthispage"> for
how to use it</a> and <em>how to tidy up afterwards</em>.
</ol>
<p>[Note that /survey_scans/ is generated by troggle and is not the same thing as /expofiles/surveyscans/ at all.]
<p>Only a small part of troggle's original plan was fully implemented and deployed.
Many of the things it was intended to replace are still operating as a motley collection written by many different people in
several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).
Today troggle is used for only three things:
<ol>
<li>Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left
hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
<li>Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description
can do this by filling-in some online forms.
</ol>
<h3>The first thing to do</h3>
<p>The first thing to do is to read: "<a href="../../../troggle/docsEtc/troggle_paper.odt" download>Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis, CUCC.</em>
<p>Two things to remember are
<ul>
<li>that troggle is just one of several cave-survey management online software systems. CUCC EXPO is not the only caving expedition with a substantial nerd community.<br /><br />
<li>that troggle is part of a 40-year ongoing project and lives in a soup of several disparate scripts all working on the same data
</ul>
<h3>Troggle Login</h3>
<p>Yes you can log in to the troggle control panel: <a href="http://expo.survex.com/troggle">expo.survex.com/troggle</a>.
</p>
<p>It has this menu of commands:
<pre>
All Survex | Scans | Tunneldata | 107 | 161 | 204 | 258 | 264 | Expo2016 | Expo2017 | Expo2018 | Django admin
</pre>
<h3>Future Developments: Preamble</h3>
<p><em>Assumptions</em> (points to necessarily agree upon)
<ol>
<li>Let's NOT try to design a generic catalogue for storing all kind of data about caves of the whole world, intended for every kind of user (sports, exploration, science). Let's just settle for a generic framework. Let geeks in individual countries or individual communities write their tools operating within this framework.
<li>Let's try make it available for the layman, but still well-playable for the geeks.
<li>Let's rely on already existing, popular technologies. Let's keep it open source and multiplatform. Let's try not to reinvent the wheel.
<li>Let's not assume everyone has an Internet connection while working with their data.
<li>Let's version-control as much as possible.
<li>Let's support i18n - let's use UTF-8 everywhere and cater for data in many languages(entrance names, cave descriptions, location descriptions etc.)
</ol>
<tt><em>Everything here should be updated or replaced - this page just records a lot of unfinished ideas.
Most people will not want to read this at all. This is for speleosoftwarearcheologists only.</em>
</tt>
<p>Two page preliminary design document for <a href="../../documents/caca_arch2.pdf">'caca' (Cave Catalogue) rev.2 2013-07-26</a> by Wookey (copied from http://wookware.org/software/cavearchive/caca_arch2.pdf)
<h3>stroggle</h3>
<p>At one time Martin Green attempted to reimplement troggle as "stroggle" using <a href="https://www.fullstackpython.com/flask.html">flask</a> instead of Django at
<a href="https://en.wikipedia.org/wiki/Gitorious">git@gitorious.org:stroggle/stroggle.git</a> (but gitorious has been deleted).</p>
<p>A copy of this project is archived by Wookey on <a href="http://wookware.org/software/cavearchive/stroggle/">wookware.org/software/cavearchive/stroggle/</a>.
<p>There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "<a href="https://gitorious.org/">gitorious valhalla</a>"<br />
<a href="https://gitorious.org/stroggle/stroggle.git/">stroggle code</a></br>
<a href="https://gitorious.org/stroggle/stroggle-gitorious-wiki.git/">stroggle-gitorious-wiki</a></br>
but note that this domain has an expired ertificate so https:// complains.
h3 id="automation">Automation on expo.survex.com</h3>
<p>Ths section is entirely out of date (June 2014), and moved here for historic interest</p>.
<p>The way things normally work, python or perl scripts turn CSV input into HTML for the data management system. Note that:</p>
<p>The CSV files are actually tab-separated, not comma-separated despite the extension.</p>
<p>The scripts can be very picky and editing the CSVs with microsoft excel has broken them in the past- not sure if this is still the case.</p>
<p>Overview of the automagical scripts on the expo data management system</p>
[Clearly very out of date is it is assuming the version control is svn whereas we changed to hg years ago.]
<pre>
Script location Input file Output file Purpose
/svn/trunk/expoweb/noinfo/make-indxal4.pl /svn/trunk/expoweb/noinfo/CAVETAB2.CSV many produces all cave description pages
/svn/trunk/expoweb/scripts/make-folklist.py /svn/trunk/expoweb/folk/folk.csv http://expo.survex.com/folk/index.htm Table of all expo members
/svn/trunk/surveys/tablize-csv.pl /svn/trunk/surveys/tablizebyname-csv.pl
/svn/trunk/surveys/Surveys.csv
http://expo.survex.com/expo/surveys/surveytable.html http://expo.survex.com/surveys/surtabnam.html
Survey status page: "wall of shame" to keep track of who still needs to draw which surveys
</pre>
<h3><a id="arch">Archived updates</a></h3>
<p>Since 2008 we have been keeping detailed records of all data management system updates in the version control system.
Before then we manually maintained <a href="../update.htm">a list of updates</a> which are now only of historical interest.
<p>A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article <a href="c21bs.html">Taking Expo Bullshit into the 21st Century</a> is archived here.
<h2>The data management system conventions bit</h2>
<p>This is likely to change with structural change to the site, with style changes which we expect to implement and with the method by which the info is actually stored and served up.</p>
<p>... and it's not written yet, either :-)</p>
<ul>
<li>Structure</li>
<li>Info for each cave &ndash; automatically generated by <tt>make-indxal4.pl</tt></li>
<li>Contents lists &amp; relative links for multi-article publications like journals. Complicated by expo articles being in a separate hierarchy from journals.</li>
<li>Translations</li>
<li>Other people's work - the noinfo hierarchy.</li>
<li>Style guide for writing cave descriptions: correct use of boldface (<em>once</em> for each passage name, at the primary definition thereof; other uses of the name should be links to this, and certainly should not be bold.) </li>
</ul>
<h3>CUCC wiki on troggle</h3>
<p>CUCC still has an archive list of things that at one time were live tasks:
from <a href="https://camcaving.uk/Documents/Expo/Legacy/Misc/Troggle%20-%20Cambridge%20University%20Caving%20Club.htm">camcaving.uk/Documents/Expo/Legacy/Misc/...</a> and that page is reproduced in the table below (so don't worry if the URL link goes dark when CUCC reorganise their legacy pages).
<p>Troggle is a system under development for keeping track of all expo data in a logical and accessible way, and displaying it on the web. At the moment, it is [no longer] under development at <u>http://troggle.cavingexpedition.com/</u>
<tt>But note that this is Aaron's version of troggle, forked from the version of troggle we use. Aaron uses this for the <a href="https://expeditionwriter.com/new-expedition-to-mount-erebus-antarctica/">Erebus expedition</a>.</tt>
</p>
<p>Note that the information there is incomplete and editing is not yet enabled.
</p>
<table border="1" cellspacing="0">
<tr>
<th><p>Feature</p></th>
<th><p>Old expo website</p></th>
<th><p>Troggle: planned</p></th>
<th><p>Troggle: progress so far</p></th>
</tr>
<tr>
<td><p>Logbook</p></td>
<td><p>Yes; manually formatted each year</p></td>
<td><p>Yes; wiki-style</p></td>
<td><p>Start at the front page, <a rel="nofollow" class="external autonumber" href="http://expo.survex.com/expedition/2007">troggle.cavingexpedition.com/ [1]</a> and click to logbook for year. The logbooks have been parsed back to 1997. </p></td>
</tr>
<tr>
<td><p>Cave index and stats generated from survex file</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Done; see <a rel="nofollow" class="external autonumber" href="http://expo.survex.com/survexfile/caves/264">troggle.cavingexpedition.com/survey/caves/264 [2]</a> </p></td>
</tr>
<tr>
<td><p>Survey workflow helper</p></td>
<td><p>Yes; minimal. surveys.csv produced an html table of whose surveys were not marked “finished”</p></td>
<td><p>Yes. Makes table of surveys per expo which shows exactly what needs doing. Displays scans. Integrated with survex, scanner software, and tunnel.</p></td>
<td><p>See it at <a rel="nofollow" class="external free" href="http://expo.survex.com/survey_scans/">troggle.cavingexpedition.com/survey</a> . Be sure to try a recent year when we should have data. Survex, scanner, and tunnel integration still needs doing.</p></td>
</tr>
<tr>
<td><p>QM lists generated automatically</p></td>
<td><p>Depends on the cave. Each cave had a different system.</p></td>
<td><p>Yes; unified system.</p></td>
<td><p>Done, but only 204 and 234 Qms have been imported from old system so far. No view yet.</p></td>
</tr>
<tr>
<td><p>Automatic calendar for each year of who will be on expo when</p></td>
<td><p>No, manually produced some years</p></td>
<td><p>Yes</p></td>
<td><p>Done; see <a rel="nofollow" class="external free" href="http://expo.survex.com/expedition/2007">troggle.cavingexpedition.com/calendar/2007</a> (replace 2007 with year in question)</p></td>
</tr>
<tr>
<td><p>Web browser used to enter data</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>Everything can be edited through admin, at <a rel="nofollow" class="external free" href="http://expo.survex.com/admin/">troggle.cavingexpedition.com/admin</a> . Ask aaron, martin, or julian for the password if you want to have a look / play around with the admin site. Any changes you make will be overwritten. Eventually, data entry will probably be done using custom forms.
</p></td>
</tr>
<tr>
<td><p>Cave and passage descriptions</p></td>
<td><p>Yes, manually html coded.</p></td>
<td><p>Yes, wiki-style.</p></td>
<td><p>Not done yet.<br />
</p></td>
</tr>
<tr>
<td><p>Expo handbook</p></td>
<td><p>Yes, manually html coded.<br />
</p>Maybe. Needs to be discussed further.</td>
<td><p><br />
</p></td>
<td><p>Not done yet.</p></td>
</tr>
<tr>
<td><p>Table of who was on which expo</p></td>
<td><p>Yes</p></td>
<td><p>Yes</p></td>
<td><p>Data has been parsed, this view hasn't been written yet. </p></td>
</tr>
<tr>
<td><p>Signup form, System for keeping contact, medical and next of kin info</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p>Signup form should be ready by 20 Jan.</p></td>
</tr>
<tr>
<td><p>Automated photo upload and gallery</p></td>
<td><p>No; some manual photo galleries put together with lots of effort</p></td>
<td><p>Yes</p></td>
<td><p>Photo upload done, gallery needs writing.</p></td>
</tr>
<tr>
<td><p>Search</p></td>
<td><p>No</p></td>
<td><p>Yes</p></td>
<td><p></p></td>
</tr>
</table>
<h3>List of cave database software</h3>
from <a href="http://wookware.org/software/cavearchive/databasesoftwarelist">wookware.org/software/cavearchive/databasesoftwarelist</a>
<pre>
ckan is something like this - could we use it?
esri online
CUCC (troggle) http://cucc.survex.com/ - this site.
virgina caves database (access+arcgis) (futrell)
each country database
Austria (spelix) ( <a href="https://www.spelix.at/">www.spelix.at/</a>
UK cave registry
mendip cave registry: (access) <a href="http://www.mcra.org.uk/wiki/doku.php">www.mcra.org.uk/wiki/doku.php</a>
White mountains database (gpx + google earth)
Matienzo (?)
Fisher ridge (stephen cladiux)
hong meigui (erin) <a href="http://www.hongmeigui.net/"http://www.hongmeigui.net/</a> (ask erin later)
Wikicaves <a href="http://www.grottocenter.org/">www.grottocenter.org/</a>
multilingual, slippymap, wiki data entry. includes coordinate-free caves.
focus on sport-caving type info (access, basic gear list, overall description, bibliography)
e.g. australians only publish coordinates to nearest 10km
turkey <a href="http://www.tayproject.org">www.tayproject.org</a>.
<a href="http://www.uisic.uis-speleo.org/contacts.html">www.uisic.uis-speleo.org/contacts.html</a> change link. no-one looks for list of databases under 'contacts'
graziano ferrari northern italy list (access + google earth)
</pre>
<h3>Wookey's notes on things to do</h3>
from <a href="http://wookware.org/software/cavearchive/goliczmail">wookware.org/software/cavearchive/goliczmail</a>
<pre>
Generally I'd like to find some people (geeks) that share these technical
ideas: (1) store things in a file system, (2) use XML, (3) do not aim too high
(do not try designing a general system for handling all caving-related data
for the whole world).
If I could find some people that agree with this, then we could try to reach a
compromise on:
(1) how do we store our data in a file system,
(2) how do we use this XML (let's do a common spec, but keep it simple)
(3) how do we aim not to high and not end up dead like CaveXML :)
After we do that, everyone goes away to do their own projects and write their
own code. Or maybe we have some degree of co-operation in actually writing the
code. Normal life. But the idea is that all geeks working on "cave inventory"
and systems making extensive use of cave inventories try to adhere to this
framework as much as possible. So that we can then exchange our tools.
I think things like "which revision system do we use" or "do we use web or
Python" are really secondary. Everyone has their own views, habits,
backgrounds.
My idea is to work on this in a small group (no more than a few persons) - to
get things going fast, even if they are not perfect from the beginning. If it
works, we try to convince others to use it and maybe push it through UIS.
</pre>
<h3>Wookey's other notes on things to do</h3>
from <a href="http://wookware.org/software/cavearchive/troggle2design">wookware.org/software/cavearchive/troggle2design</a>
<pre>
forms
-----
1) members read/write folk.csv and year/members
2) cave read/write cave_data, entrance_data, surveys/pics
3) trips -> logbook , QMs, or surveys (more than one survey or location possible)
4) logbook reads/write year/logbook
5) survey
6) prospecting app
forms show who is logged in.
databases
---------
trips, read from
logbook entry
folder year#index
.svx files
description
QMs
members (cache from form)
caves
caves_data
entrance_data
storage:
expoweb
data/
cave_entrances
caves
descriptions
loser
foo.svx
</pre>
<h3>Yet more of Wookey's notes</h3>
from <a href="http://wookware.org/software/cavearchive/expoweb-design">wookware.org/software/cavearchive/expoweb-design</a>
<pre>
frontpage
---------
quick to load:
Links:
Caves number, name, location
Years
Handbook
Data Entry
Main Index
Slippy map:
Indexes to cave page
Cave page:
Access, description, photos, QMs, Survey
Years:
Logbooks/surveynotes/survexdata/people matrix
Documents
Data Entry:
Logbook entry
Survey data
Survey Notes
Cave description
QMs
Photos
New cave
Backend datafiles:
caves/
cave_entrance
cave_data
directory of info
years/
year/
logbook
pubs/
reports
admin/
lists
who_and_when
travel
jobs
surveyscans/
year/
index
#num
handbook/
(all static info)
Storage:
non-html or > 200K go in 'files' (PDF, PNG, JPEG, DOC, ODF, SVG)
convert small 800x600 version into website by default. (matching structure?
</pre>
<hr />
</body>
</html>

View File

@ -0,0 +1,48 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle Status</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle & Expo Systems - Update status</h1>
<p>Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook.
<h3>Early 2019</h3>
<p>In early 2019 the university computing service upgraded its firewall rules which took the
server offline completely.
<p>
Wookey eventually managed to find us free space (a virtual machine)
on a debian mirror server somewhere in Leicestershire (we think).
This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
<p>At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata). The other two repos expoweb and loser remained on mercurial.
</div>
<h4>Wookey: July 2019</h4>
<p>troggle has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
<p>
tunneldata has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
<p>
The loser repo and expoweb repo need more care in migration. Loser should have the old 1999-2004 CVS history restored, and maybe toms annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
<p>
Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
<h4>April 2020</h4>
<p>
Wookey is now preparing to move 'expoweb' and 'loser' from mercurial to git "as-is" and then to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up during the move.
<p>Sam continues to work on upgrading django from v1.7 . We are using python 2.7.17 and while we <a href="https://docs.djangoproject.com/en/1.10/topics/python3/">could upgrade</a> to Python v3 using the same version (1.7) of django, we would rather upgrade django as much as possible first before we tackle that. Old versions of django have unpatched security issues.
<p> "Django 1.11 is the last version to support Python 2.7. Support for Python 2.7 and Django 1.11 ends in 2020." see: <a href="https://docs.djangoproject.com/en/3.0/faq/install/">django versions</a>.
<p>
Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.
<hr />
Return to<br />
<a href="../website-history.html">Website history</a><br />
<a href="trogintro.html">Troggle intro</a><br />
<hr />
</body>
</html>

View File

@ -116,7 +116,7 @@ made sense, and so wrote Troggle.
This uses Django to generate pages.
This reads in all the logbooks and surveys and provides a nice way to access them, and enter new data.
It was separate for a while until Martin Green added code to merge the old static pages and
new troggle dynamic pages into the same site. This is now the live system running everything (in 2019). Work on developing Troggle further still continues sporadically (see <a href="computing/troggle-ish.html">Troggle notes</a>).</p>
new troggle dynamic pages into the same site. This is now the live system running everything (in 2019). Work on developing Troggle further still continues (see <a href="troggle/trogintro.html">Troggle intro</a>).</p>
<p>After Expo 2009 the version control system was updated to a <a href="onlinesystems.html#mercurial">DVCS</a> (Mercurial, aka 'hg'),
because a distributed version control system makes a great deal of sense for expo
@ -133,78 +133,25 @@ university since Feb 2014.
<h3>2018</h3>
<p>In 2018 we have 4 repositories,
<p>In 2018 we had 4 repositories:
<ul>
<li><a href="/repositories/home/expo/loser/graph/">loser</a> - the survex cave survey data (hg)</li>
<li><a href="/cgit/drawings/.git/log">drawings</a> - the tunnel and therion cave data and drawings (git)</li>
<li><a href="/repositories/home/expo/expoweb/graph">expoweb</a> - the website pages, handbook, generation scripts (hg)</li>
<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="computing/troggle-ish.html">notes on troggle</a> for further explanation (git)</li>
<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="troggle/trogintro.html">notes on troggle</a> for further explanation (git)</li>
</ul>
<p>In spring 2018 Sam, Wookey and Paul Fox updated the Linux version and the Django version (i.e. troggle) to
something vaguely acceptable to the university computing service and fixed all the problems that were then observed.
<h3>2019</h3>
<p>In early 2019 the university computing serviuce upgraded its firewall rules which took the
server offline completely. Wookey eventually managed to find us free space (a virtual machine)
on a debian mirror server somewhere in Leicestershire (we think).
This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
<p>At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata). The other two repos expoweb and loser remained on mercurial.
</div>
<h4>Wookey: 12 July 2019</h4>
<p>troggle has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
<h3>More recent</h3>
<p>
tunneldata has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
<p>
The loser repo and expoweb repo need more care in migration. Loser should have the old 1999-2004 CVS history restored, and maybe toms annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
<p>
Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
For the current situation see <a href="troggle/trogstatus.html">expo systems status</a>.
<hr />
<h3 id="automation">Automation on expo.survex.com</h3>
<p>Ths section is entirely out of date (June 2014), and moved here for historic interest</p>.
<p>The way things normally work, python or perl scripts turn CSV input into HTML for the data management system. Note that:</p>
<p>The CSV files are actually tab-separated, not comma-separated despite the extension.</p>
<p>The scripts can be very picky and editing the CSVs with microsoft excel has broken them in the past- not sure if this is still the case.</p>
<p>Overview of the automagical scripts on the expo data management system</p>
[Clearly very out of date is it is assuming the version control is svn whereas we changed to hg years ago.]
<pre>
Script location Input file Output file Purpose
/svn/trunk/expoweb/noinfo/make-indxal4.pl /svn/trunk/expoweb/noinfo/CAVETAB2.CSV many produces all cave description pages
/svn/trunk/expoweb/scripts/make-folklist.py /svn/trunk/expoweb/folk/folk.csv http://expo.survex.com/folk/index.htm Table of all expo members
/svn/trunk/surveys/tablize-csv.pl /svn/trunk/surveys/tablizebyname-csv.pl
/svn/trunk/surveys/Surveys.csv
http://expo.survex.com/expo/surveys/surveytable.html http://expo.survex.com/surveys/surtabnam.html
Survey status page: "wall of shame" to keep track of who still needs to draw which surveys
</pre>
<h3><a id="arch">Archived updates</a></h3>
<p>Since 2008 we have been keeping detailed records of all data management system updates in the version control system.
Before then we manually maintained <a href="../update.htm">a list of updates</a> which are now only of historical interest.
<p>A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article <a href="c21bs.html">Taking Expo Bullshit into the 21st Century</a> is archived here.
<h2>The data management system conventions bit</h2>
<p>This is likely to change with structural change to the site, with style changes which we expect to implement and with the method by which the info is actually stored and served up.</p>
<p>... and it's not written yet, either :-)</p>
<ul>
<li>Structure</li>
<li>Info for each cave &ndash; automatically generated by <tt>make-indxal4.pl</tt></li>
<li>Contents lists &amp; relative links for multi-article publications like journals. Complicated by expo articles being in a separate hierarchy from journals.</li>
<li>Translations</li>
<li>Other people's work - the noinfo hierarchy.</li>
<li>Style guide for writing cave descriptions: correct use of boldface (<em>once</em> for each passage name, at the primary definition thereof; other uses of the name should be links to this, and certainly should not be bold.) </li>
</ul>
Return to<br />
<a href="onlinesystems.html">Website update</a><br />
<a href="onlinesystems.html">expo online systems overbiew</a><br />
<hr />
</body>

View File

@ -827,7 +827,7 @@ ticked off 2 QM's !
<p>[and don't write up their trip at all, even
though Mike remembered it in 2009 when it wasn't
parsed into the <a href="../../handbook/computing/troggle-ish.html">troggle</a> database]
parsed into the <a href="../../handbook/troggle/trogintro.html">troggle</a> database]
<p> T/U 11 hours

View File

@ -65,7 +65,7 @@
<tr>
<th rowspan="2" scope="row">Aaron Curtis </th>
<td rowspan="2">Chief Troggle-odyte </td>
<td>Responsible for the development of <a href="../../handbook/computing/troggle-ish.html">Troggle</a></td>
<td>Responsible for the development of <a href="../../handbook/troggle/trogintro.html">Troggle</a></td>
</tr>
<tr>
<tr>

View File

@ -50,7 +50,7 @@ vicinity of E entrance. More specifically:
caving groups.</li>
<li>To update the expo website to reflect the passages found in
the past 2 or 3 years.</li>
<li>To test out <a href="http://expo.survex.com/handbook/troggle-ish.html">Troggle</a> on the expedition.</li>
<li>To test out <a href="../../handbook/troggle/trogintro.html">Troggle</a> on the expedition.</li>
<p>&nbsp; </p>
<
</body>

View File

@ -58,8 +58,8 @@ and its southern extremities were tantalisingly close to the north of
204, so this was also an exploration goal for 2009.</p>
<p>This year's expedition also had a non-caving goal (not just
drinking G&ouml;sser). Recently members of CUCC have started to
develop a piece of software called <a href="../../handbook/computing/troggle-ish.html">Troggle</a>, which aims to facilitate
drinking G&ouml;sser). Recently [since 2006] members of CUCC have started to
develop a piece of software called <a href="../../handbook/troggle/trogintro.html">Troggle</a>, which aims to facilitate
keeping track of logbook entries, typing up surveys, caves etc, and save time in a lot of the
work that goes on behind the scenes when expo is over. This year was
the first time Troggle would be tested "in th field" (well, spud hut).</p>