We have migrated two of these to git but the other two still use mercurial. @@ -103,20 +103,11 @@ below for details on that.
Part of the data management system is static HTML, but quite a lot is generated by scripts and troggle (a web framework built using Django). -
Examples of troggle-generated pages from data: -
Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the troggle intro. +
Anything you check in which affects cave data or descriptions won't appear on the site until the data management system update scripts are run. -This happens automatically every 30 mins, but you can also kick off a manual update. +This should happen automatically every 30 mins (not since 2017), but you can also kick off a manual update. See 'The expoweb-update script' below for details.
Also note that the ::expoweb:: web pages and cave data reports you see on the visible website diff --git a/handbook/onlinesystems.html b/handbook/onlinesystems.html index cfb5c5ebe..c8d9aad76 100644 --- a/handbook/onlinesystems.html +++ b/handbook/onlinesystems.html @@ -8,8 +8,8 @@
The online data and web publishing system (i.e. "the website") is now large and complicated with a lot of aspects. -This handbook section contains info at various levels: +
The online data and web publishing system (i.e. "the website") is now large and complicated. +The handbook contains info at various levels: simple 'How to add stuff' information for the typical expoer, more detailed info for cloning it onto your own machine for more significant edits, and structural info on how it's all put together for people who want/need to change things. @@ -20,7 +20,7 @@ on how the cave data, handbook and public website are constructed and managed. It contains material which will be merged into this online systems manual. -
These pages listed below have been reviewed recently (2018), and a +
These pages listed below have been reviewed recently (2019), and a fuller list of "How do I..." instruction pages are on the handbook opening page.
But the systems Manual is still being actively edited to extract and simplify documentation. +
The systems manual is being actively edited to extract and simplify documentation.
Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the troggle intro.
It's important to understand that the pages you can edit by this method -are stored in a distributed version control system (see below). This stops us losing data and +are stored in a version control system (see below). This stops us losing data and makes it very hard for you to screw anything up permanently, so don't worry about making changes - they can always be reverted if there is a problem. It also means that several people can work on the site on @@ -89,7 +91,7 @@ See these instructions for this tidy-up
We use a distributed revision control system (git, and formerly mercurial) for all the important data. +
We use a distributed revision control system (git, and formerly mercurial) for all the important data. (Note that we just use git: not GitHub, not GitLab, just git.) This means that many people can edit and merge their changes with the expo server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut and those who have returned to the UK. Also anyone who is up @@ -106,42 +108,6 @@ The same goes for holiday photographs and GPS logs.
In 2019 we had half our version-controlled repositories under mercurial and half under git. The intention is to move entirely to git before the 2020 expo. -
-Troggle is the software collection (not really a "package") based on Django -originally intended to manage all expo data in a logical and accessible way -and publish it on the web. -
Examples of troggle-generated pages from data: -
[Note that /survey_scans/ is generated by troggle and is not the same thing as /expofiles/surveyscans/ at all.] -
Only a small part of troggle's original plan was fully implemented and deployed. -Many of the things it was intended to replace are still operating as a motley collection written by many different people in -several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL). -Today troggle is used for only three things: -
See the notes on troggle page -for how and why it was developed and what needs to be done. -
You will type this description, and pass it on to someone more nerdy who -will file it in the right place. This will involve "creating a new cave" using the troggle system. +will file it in the right place. This will involve "creating a new cave" using the troggle system. diff --git a/handbook/tortoise/tortoise-win.htm b/handbook/tortoise/tortoise-win.htm index b37040e02..0a48134a0 100644 --- a/handbook/tortoise/tortoise-win.htm +++ b/handbook/tortoise/tortoise-win.htm @@ -86,7 +86,7 @@ You can scan what's in the repositories (read only) using your web browser:
Troggle is the software which runs the the expo cave survey data management and website. + +
Troggle manages all cave and expo data in a logical and maintainable way +and publishes it on the web. +
+The troggle software is written and maintained by expo members. + +
Examples of troggle-generated pages from data: +
Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.
You may have arrived here by accident when where you really need to be is website history. -
This page needs to be restructured and rewritten so that it describes these things: +
This part of the handbook is intended for people maintaining the troggle software. Day to day cave recording and surveying tasks are documented in the expo "survey handbook" + +
This troggle manual describes these:
This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it.
Two page preliminary design document for 'caca' (Cave Catalogue) rev.2 2013-07-26 by Wookey (copied from http://wookware.org/software/cavearchive/caca_arch2.pdf)
Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook. + +
This part of the handbook is intended for people maintaining the troggle software. Day to day cave recording and surveying tasks are documented in the expo "survey handbook" + +
This troggle manual describes these: +
This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it. + +Today troggle is used for only three things: +
[Note that /survey_scans/ is generated by troggle and is not the same thing as /expofiles/surveyscans/ at all.] +
Only a small part of troggle's original plan was fully implemented and deployed. +Many of the things it was intended to replace are still operating as a motley collection written by many different people in +several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL). +Today troggle is used for only three things: +
The first thing to do is to read: "Troggle: a novel system for cave exploration information management", by Aaron Curtis, CUCC. +
Two things to remember are +
Yes you can log in to the troggle control panel: expo.survex.com/troggle. +
+It has this menu of commands: +
+All Survex | Scans | Tunneldata | 107 | 161 | 204 | 258 | 264 | Expo2016 | Expo2017 | Expo2018 | Django admin ++ +
Assumptions (points to necessarily agree upon) +
Two page preliminary design document for 'caca' (Cave Catalogue) rev.2 2013-07-26 by Wookey (copied from http://wookware.org/software/cavearchive/caca_arch2.pdf) + +
At one time Martin Green attempted to reimplement troggle as "stroggle" using flask instead of Django at +git@gitorious.org:stroggle/stroggle.git (but gitorious has been deleted).
+A copy of this project is archived by Wookey on wookware.org/software/cavearchive/stroggle/. +
There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "gitorious valhalla"
+stroggle code
+stroggle-gitorious-wiki
+but note that this domain has an expired ertificate so https:// complains.
+
+h3 id="automation">Automation on expo.survex.com
+
+
Ths section is entirely out of date (June 2014), and moved here for historic interest
. + +The way things normally work, python or perl scripts turn CSV input into HTML for the data management system. Note that:
+The CSV files are actually tab-separated, not comma-separated despite the extension.
+The scripts can be very picky and editing the CSVs with microsoft excel has broken them in the past- not sure if this is still the case.
+Overview of the automagical scripts on the expo data management system
+[Clearly very out of date is it is assuming the version control is svn whereas we changed to hg years ago.] ++Script location Input file Output file Purpose +/svn/trunk/expoweb/noinfo/make-indxal4.pl /svn/trunk/expoweb/noinfo/CAVETAB2.CSV many produces all cave description pages +/svn/trunk/expoweb/scripts/make-folklist.py /svn/trunk/expoweb/folk/folk.csv http://expo.survex.com/folk/index.htm Table of all expo members + +/svn/trunk/surveys/tablize-csv.pl /svn/trunk/surveys/tablizebyname-csv.pl + /svn/trunk/surveys/Surveys.csv + +http://expo.survex.com/expo/surveys/surveytable.html http://expo.survex.com/surveys/surtabnam.html + Survey status page: "wall of shame" to keep track of who still needs to draw which surveys + ++ +
Since 2008 we have been keeping detailed records of all data management system updates in the version control system. +Before then we manually maintained a list of updates which are now only of historical interest. +
A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article Taking Expo Bullshit into the 21st Century is archived here. + +
This is likely to change with structural change to the site, with style changes which we expect to implement and with the method by which the info is actually stored and served up.
+... and it's not written yet, either :-)
+CUCC still has an archive list of things that at one time were live tasks: +from camcaving.uk/Documents/Expo/Legacy/Misc/... and that page is reproduced in the table below (so don't worry if the URL link goes dark when CUCC reorganise their legacy pages). +
Troggle is a system under development for keeping track of all expo data in a logical and accessible way, and displaying it on the web. At the moment, it is [no longer] under development at http://troggle.cavingexpedition.com/ + +But note that this is Aaron's version of troggle, forked from the version of troggle we use. Aaron uses this for the Erebus expedition. +
+Note that the information there is incomplete and editing is not yet enabled. +
+Feature |
+ Old expo website |
+ Troggle: planned |
+ Troggle: progress so far |
+
|---|---|---|---|
Logbook |
+ Yes; manually formatted each year |
+ Yes; wiki-style |
+ Start at the front page, troggle.cavingexpedition.com/ [1] and click to logbook for year. The logbooks have been parsed back to 1997. |
+
Cave index and stats generated from survex file |
+ Yes |
+ Yes |
+ + |
Survey workflow helper |
+ Yes; minimal. surveys.csv produced an html table of whose surveys were not marked “finished” |
+ Yes. Makes table of surveys per expo which shows exactly what needs doing. Displays scans. Integrated with survex, scanner software, and tunnel. |
+ See it at troggle.cavingexpedition.com/survey . Be sure to try a recent year when we should have data. Survex, scanner, and tunnel integration still needs doing. |
+
QM lists generated automatically |
+ Depends on the cave. Each cave had a different system. |
+ Yes; unified system. |
+ Done, but only 204 and 234 Qms have been imported from old system so far. No view yet. |
+
Automatic calendar for each year of who will be on expo when |
+ No, manually produced some years |
+ Yes |
+ Done; see troggle.cavingexpedition.com/calendar/2007 (replace 2007 with year in question) |
+
Web browser used to enter data |
+ No |
+ Yes |
+ Everything can be edited through admin, at troggle.cavingexpedition.com/admin . Ask aaron, martin, or julian for the password if you want to have a look / play around with the admin site. Any changes you make will be overwritten. Eventually, data entry will probably be done using custom forms. + |
+
Cave and passage descriptions |
+ Yes, manually html coded. |
+ Yes, wiki-style. |
+ Not done yet. |
+
Expo handbook |
+ Yes, manually html coded. |
+
|
+ Not done yet. |
+
Table of who was on which expo |
+ Yes |
+ Yes |
+ Data has been parsed, this view hasn't been written yet. |
+
Signup form, System for keeping contact, medical and next of kin info |
+ No |
+ Yes |
+ Signup form should be ready by 20 Jan. |
+
Automated photo upload and gallery |
+ No; some manual photo galleries put together with lots of effort |
+ Yes |
+ Photo upload done, gallery needs writing. |
+
Search |
+ No |
+ Yes |
+ + |
+ckan is something like this - could we use it? +esri online + +CUCC (troggle) http://cucc.survex.com/ - this site. +virgina caves database (access+arcgis) (futrell) +each country database +Austria (spelix) ( www.spelix.at/ +UK cave registry +mendip cave registry: (access) www.mcra.org.uk/wiki/doku.php +White mountains database (gpx + google earth) +Matienzo (?) +Fisher ridge (stephen cladiux) +hong meigui (erin) (ask erin later) +Wikicaves www.grottocenter.org/ + multilingual, slippymap, wiki data entry. includes coordinate-free caves. + focus on sport-caving type info (access, basic gear list, overall description, bibliography) + e.g. australians only publish coordinates to nearest 10km +turkey www.tayproject.org. + +www.uisic.uis-speleo.org/contacts.html change link. no-one looks for list of databases under 'contacts' + +graziano ferrari northern italy list (access + google earth) ++ +
+Generally I'd like to find some people (geeks) that share these technical +ideas: (1) store things in a file system, (2) use XML, (3) do not aim too high +(do not try designing a general system for handling all caving-related data +for the whole world). + +If I could find some people that agree with this, then we could try to reach a +compromise on: +(1) how do we store our data in a file system, +(2) how do we use this XML (let's do a common spec, but keep it simple) +(3) how do we aim not to high and not end up dead like CaveXML :) + +After we do that, everyone goes away to do their own projects and write their +own code. Or maybe we have some degree of co-operation in actually writing the +code. Normal life. But the idea is that all geeks working on "cave inventory" +and systems making extensive use of cave inventories try to adhere to this +framework as much as possible. So that we can then exchange our tools. + +I think things like "which revision system do we use" or "do we use web or +Python" are really secondary. Everyone has their own views, habits, +backgrounds. + +My idea is to work on this in a small group (no more than a few persons) - to +get things going fast, even if they are not perfect from the beginning. If it +works, we try to convince others to use it and maybe push it through UIS. ++ +
+forms +----- +1) members read/write folk.csv and year/members +2) cave read/write cave_data, entrance_data, surveys/pics +3) trips -> logbook , QMs, or surveys (more than one survey or location possible) +4) logbook reads/write year/logbook +5) survey +6) prospecting app + +forms show who is logged in. + +databases +--------- +trips, read from + logbook entry + folder year#index + .svx files + description + QMs + +members (cache from form) + +caves + caves_data + entrance_data + +storage: + expoweb + data/ + cave_entrances + caves + descriptions + + loser + foo.svx ++ + +
+frontpage +--------- +quick to load: +Links: + Caves number, name, location + Years + Handbook + Data Entry + Main Index + +Slippy map: + Indexes to cave page + +Cave page: + Access, description, photos, QMs, Survey + +Years: + Logbooks/surveynotes/survexdata/people matrix + Documents + +Data Entry: + Logbook entry + Survey data + Survey Notes + Cave description + QMs + Photos + New cave + +Backend datafiles: + caves/ + cave_entrance + cave_data + directory of info + + years/ + year/ + logbook + pubs/ + reports + admin/ + lists + who_and_when + travel + jobs + + surveyscans/ + year/ + index + #num + handbook/ + (all static info) + +Storage: + non-html or > 200K go in 'files' (PDF, PNG, JPEG, DOC, ODF, SVG) + convert small 800x600 version into website by default. (matching structure? ++ +
Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. + +
In early 2019 the university computing service upgraded its firewall rules which took the +server offline completely. +
+Wookey eventually managed to find us free space (a virtual machine) +on a debian mirror server somewhere in Leicestershire (we think). +This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling. + +
At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata). The other two repos expoweb and loser remained on mercurial. + + +
troggle has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used. +
+tunneldata has also been migrated to git, and renamed 'drawings' as it includes therion data too these days. +
+The loser repo and expoweb repo need more care in migration. Loser should have the old 1999-2004 CVS history restored, and maybe toms annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion. +
+Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down. + +
+Wookey is now preparing to move 'expoweb' and 'loser' from mercurial to git "as-is" and then to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up during the move. +
Sam continues to work on upgrading django from v1.7 . We are using python 2.7.17 and while we could upgrade to Python v3 using the same version (1.7) of django, we would rather upgrade django as much as possible first before we tackle that. Old versions of django have unpatched security issues. +
"Django 1.11 is the last version to support Python 2.7. Support for Python 2.7 and Django 1.11 ends in 2020." see: django versions. +
+Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything. +
After Expo 2009 the version control system was updated to a DVCS (Mercurial, aka 'hg'), because a distributed version control system makes a great deal of sense for expo @@ -133,78 +133,25 @@ university since Feb 2014.
In 2018 we have 4 repositories, +
In 2018 we had 4 repositories:
In spring 2018 Sam, Wookey and Paul Fox updated the Linux version and the Django version (i.e. troggle) to something vaguely acceptable to the university computing service and fixed all the problems that were then observed. -
In early 2019 the university computing serviuce upgraded its firewall rules which took the -server offline completely. Wookey eventually managed to find us free space (a virtual machine) -on a debian mirror server somewhere in Leicestershire (we think). -This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling. - -
At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata). The other two repos expoweb and loser remained on mercurial. - - -
troggle has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used. +
-tunneldata has also been migrated to git, and renamed 'drawings' as it includes therion data too these days. -
-The loser repo and expoweb repo need more care in migration. Loser should have the old 1999-2004 CVS history restored, and maybe toms annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion. -
-Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down. +For the current situation see expo systems status.
Ths section is entirely out of date (June 2014), and moved here for historic interest
. - -The way things normally work, python or perl scripts turn CSV input into HTML for the data management system. Note that:
-The CSV files are actually tab-separated, not comma-separated despite the extension.
-The scripts can be very picky and editing the CSVs with microsoft excel has broken them in the past- not sure if this is still the case.
-Overview of the automagical scripts on the expo data management system
-[Clearly very out of date is it is assuming the version control is svn whereas we changed to hg years ago.] --Script location Input file Output file Purpose -/svn/trunk/expoweb/noinfo/make-indxal4.pl /svn/trunk/expoweb/noinfo/CAVETAB2.CSV many produces all cave description pages -/svn/trunk/expoweb/scripts/make-folklist.py /svn/trunk/expoweb/folk/folk.csv http://expo.survex.com/folk/index.htm Table of all expo members - -/svn/trunk/surveys/tablize-csv.pl /svn/trunk/surveys/tablizebyname-csv.pl - /svn/trunk/surveys/Surveys.csv - -http://expo.survex.com/expo/surveys/surveytable.html http://expo.survex.com/surveys/surtabnam.html - Survey status page: "wall of shame" to keep track of who still needs to draw which surveys - -- -
Since 2008 we have been keeping detailed records of all data management system updates in the version control system. -Before then we manually maintained a list of updates which are now only of historical interest. -
A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article Taking Expo Bullshit into the 21st Century is archived here. - -
This is likely to change with structural change to the site, with style changes which we expect to implement and with the method by which the info is actually stored and served up.
-... and it's not written yet, either :-)
-