CUCC Expedition Handbook

Troggle - TO BE RESTRUCTURED

Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.

NOTES - taken from elsewhere

This part of the handbook is intended for people maintaining the troggle software. Day to day cave recording and surveying tasks are documented in the expo "survey handbook"

This troggle manual describes these:

• Annual tasks: preparing for next year, finishing last year (troggle & scripts)
• Architectural documentation of how it all fits together & list of active scripts
• How to edit and maintain troggle itself. The code is public on repository ::troggle::

This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it. Today troggle is used for only three things:

1. Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
2. Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description can do this by filling-in some online forms. (And managing all the cave suvey data to produce this.)
3. Providing a secondary way of editing individual pages of the handbook and historic records pages for very quick and urgent changes. This is the "Edit this page" capability; see for how to use it and how to tidy up afterwards.

[Note that /survey_scans/ is generated by troggle and is not the same thing as /expofiles/surveyscans/ at all.]

Only a small part of troggle's original plan was fully implemented and deployed. Many of the things it was intended to replace are still operating as a motley collection written by many different people in several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL). Today troggle is used for only three things:

1. Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
2. Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description can do this by filling-in some online forms.

The first thing to do

The first thing to do is to read: "Troggle: a novel system for cave exploration information management", by Aaron Curtis, CUCC.

Two things to remember are

• that troggle is just one of several cave-survey management online software systems. CUCC EXPO is not the only caving expedition with a substantial nerd community.
  
  
• that troggle is part of a 40-year ongoing project and lives in a soup of several disparate scripts all working on the same data

Troggle Login

Yes you can log in to the troggle control panel: expo.survex.com/troggle.

It has this menu of commands:

All Survex | Scans | Tunneldata | 107 | 161 | 204 | 258 | 264 | Expo2016 | Expo2017 | Expo2018 | Django admin 

Future Developments: Preamble

Assumptions (points to necessarily agree upon)

1. Let's NOT try to design a generic catalogue for storing all kind of data about caves of the whole world, intended for every kind of user (sports, exploration, science). Let's just settle for a generic framework. Let geeks in individual countries or individual communities write their tools operating within this framework.
2. Let's try make it available for the layman, but still well-playable for the geeks.
3. Let's rely on already existing, popular technologies. Let's keep it open source and multiplatform. Let's try not to reinvent the wheel.
4. Let's not assume everyone has an Internet connection while working with their data.
5. Let's version-control as much as possible.
6. Let's support i18n - let's use UTF-8 everywhere and cater for data in many languages(entrance names, cave descriptions, location descriptions etc.)

Two page preliminary design document for 'caca' (Cave Catalogue) rev.2 2013-07-26 by Wookey (copied from http://wookware.org/software/cavearchive/caca_arch2.pdf)

stroggle

At one time Martin Green attempted to reimplement troggle as "stroggle" using flask instead of Django at git@gitorious.org:stroggle/stroggle.git (but gitorious has been deleted).

A copy of this project is archived by Wookey on wookware.org/software/cavearchive/stroggle/.

There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "gitorious valhalla"
stroggle code
stroggle-gitorious-wiki
but note that this domain has an expired ertificate so https:// complains. h3 id="automation">Automation on expo.survex.com

Ths section is entirely out of date (June 2014), and moved here for historic interest

The way things normally work, python or perl scripts turn CSV input into HTML for the data management system. Note that:

The CSV files are actually tab-separated, not comma-separated despite the extension.

The scripts can be very picky and editing the CSVs with microsoft excel has broken them in the past- not sure if this is still the case.

Overview of the automagical scripts on the expo data management system

Script location 	Input file 	Output file 	Purpose
/svn/trunk/expoweb/noinfo/make-indxal4.pl 	/svn/trunk/expoweb/noinfo/CAVETAB2.CSV 	many 	produces all cave description pages
/svn/trunk/expoweb/scripts/make-folklist.py 	/svn/trunk/expoweb/folk/folk.csv 	http://expo.survex.com/folk/index.htm 	Table of all expo members

/svn/trunk/surveys/tablize-csv.pl /svn/trunk/surveys/tablizebyname-csv.pl
	/svn/trunk/surveys/Surveys.csv 	

http://expo.survex.com/expo/surveys/surveytable.html http://expo.survex.com/surveys/surtabnam.html
	Survey status page: "wall of shame" to keep track of who still needs to draw which surveys

Archived updates

Since 2008 we have been keeping detailed records of all data management system updates in the version control system. Before then we manually maintained a list of updates which are now only of historical interest.

A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article Taking Expo Bullshit into the 21st Century is archived here.

The data management system conventions bit

This is likely to change with structural change to the site, with style changes which we expect to implement and with the method by which the info is actually stored and served up.

... and it's not written yet, either :-)

• Structure
• Info for each cave – automatically generated by make-indxal4.pl
• Contents lists & relative links for multi-article publications like journals. Complicated by expo articles being in a separate hierarchy from journals.
• Translations
• Other people's work - the noinfo hierarchy.
• Style guide for writing cave descriptions: correct use of boldface (once for each passage name, at the primary definition thereof; other uses of the name should be links to this, and certainly should not be bold.)

CUCC wiki on troggle

CUCC still has an archive list of things that at one time were live tasks: from camcaving.uk/Documents/Expo/Legacy/Misc/... and that page is reproduced in the table below (so don't worry if the URL link goes dark when CUCC reorganise their legacy pages).

Troggle is a system under development for keeping track of all expo data in a logical and accessible way, and displaying it on the web. At the moment, it is [no longer] under development at http://troggle.cavingexpedition.com/ But note that this is Aaron's version of troggle, forked from the version of troggle we use. Aaron uses this for the Erebus expedition.

Note that the information there is incomplete and editing is not yet enabled.

List of cave database software

ckan is something like this - could we use it?
esri online

CUCC (troggle) http://cucc.survex.com/ - this site.
virgina caves database (access+arcgis) (futrell)
each country database
Austria (spelix) ( www.spelix.at/
UK cave registry 
mendip cave registry: (access) www.mcra.org.uk/wiki/doku.php
White mountains database (gpx + google earth)
Matienzo (?)
Fisher ridge (stephen cladiux)
hong meigui (erin)  (ask erin later)
Wikicaves www.grottocenter.org/
 multilingual, slippymap, wiki data entry. includes coordinate-free caves.
 focus on sport-caving type info (access, basic gear list, overall description, bibliography)
 e.g. australians only publish coordinates to nearest 10km
turkey www.tayproject.org.

www.uisic.uis-speleo.org/contacts.html change link. no-one looks for list of databases under 'contacts'

graziano ferrari northern italy list (access + google earth)

Wookey's notes on things to do

Generally I'd like to find some people (geeks) that share these technical
ideas: (1) store things in a file system, (2) use XML, (3) do not aim too high
(do not try designing a general system for handling all caving-related data
for the whole world).

If I could find some people that agree with this, then we could try to reach a
compromise on:
(1) how do we store our data in a file system,
(2) how do we use this XML (let's do a common spec, but keep it simple)
(3) how do we aim not to high and not end up dead like CaveXML :)

After we do that, everyone goes away to do their own projects and write their
own code. Or maybe we have some degree of co-operation in actually writing the
code. Normal life. But the idea is that all geeks working on "cave inventory"
and systems making extensive use of cave inventories try to adhere to this
framework as much as possible. So that we can then exchange our tools.

I think things like "which revision system do we use" or "do we use web or    
Python" are really secondary. Everyone has their own views, habits,
backgrounds.

My idea is to work on this in a small group (no more than a few persons) - to
get things going fast, even if they are not perfect from the beginning. If it
works, we try to convince others to use it and maybe push it through UIS. 

Wookey's other notes on things to do

forms
-----
1) members read/write folk.csv and year/members
2) cave read/write cave_data, entrance_data, surveys/pics 
3) trips -> logbook , QMs, or surveys (more than one survey or location possible)
4) logbook reads/write year/logbook
5) survey 
6) prospecting app

forms show who is logged in.

databases
---------
trips, read from 
 logbook entry
 folder year#index
 .svx files
 description
 QMs

members (cache from form)

caves
 caves_data
 entrance_data

storage:
 expoweb
 data/
 cave_entrances
 caves
 descriptions

 loser
 foo.svx

Yet more of Wookey's notes

frontpage
---------
quick to load:
Links:
 Caves number, name, location
 Years 
 Handbook
 Data Entry
 Main Index

Slippy map:
 Indexes to cave page

Cave page:
 Access, description, photos, QMs, Survey
 
Years:
 Logbooks/surveynotes/survexdata/people matrix
 Documents

Data Entry:
 Logbook entry
 Survey data
 Survey Notes
 Cave description
 QMs
 Photos
 New cave

Backend datafiles:
 caves/
 cave_entrance
 cave_data
 directory of info
 
 years/
 year/
  logbook
	 pubs/
	  reports
  admin/
   lists
		 who_and_when
		 travel
		 jobs
 
 surveyscans/
  year/
	  index
		 #num
 handbook/
  (all static info)
 
Storage:
 non-html or > 200K go in 'files' (PDF, PNG, JPEG, DOC, ODF, SVG)
 convert small 800x600 version into website by default. (matching structure?