diff --git a/handbook/datamgt.html b/handbook/datamgt.html index 3201d9fba..2f4133f90 100644 --- a/handbook/datamgt.html +++ b/handbook/datamgt.html @@ -22,6 +22,8 @@ data. Aside from the data collection requirements of the game itself, setting up expedition) of cave exploration often involves collection of personal information ranging from dates available to medical information to the desire to purchase an expedition t-shirt.
+ + If an expedition will only happen once, low-tech methods are usually adequate to record information. Any events that need to be recorded can go in a logbook. Survey notes must be turned into finished cave sketches, without undue concern for the future expansion of those sketches. @@ -49,8 +51,10 @@ or even useful recipes for locally available food. The more of this information wishes to keep, the more valuable an effective and user-friendly system of data management. -
From " -Troggle: a novel system for cave exploration information management", by Aaron Curtis, CUCC. +
From "Troggle: +a novel system for cave exploration information management", by Aaron Curtis (2006) and +updated as "Troggle: +a revised system for cave data management" in 2020.
You will be familiar with documenting newly found QMs in the survex file when you type it in. But QMs are only useful if they can be easily scanned by people planning the next pushing trip. That's what we are discussing here.
There are four (and a half) ways we have used to manage QMs:
diff --git a/handbook/troggle/trogarch.html b/handbook/troggle/trogarch.html
index aa7e98e2b..8581453bd 100644
--- a/handbook/troggle/trogarch.html
+++ b/handbook/troggle/trogarch.html
@@ -36,9 +36,9 @@ ALSO there have been tables added to the core representation which are not antic
Read the proposal: "Troggle: a novel system for cave exploration information management", by Aaron Curtis. But remember that this paper is an over-ambitious proposal. Only the core data management features have been built. We have none of the "person management" features, none of the "wallet progress" stuff and only two forms in use: for entering cave and cave entrance data. +
The reasons why we have an online system at all are described in our website history. +
There is an introductory article "Troggle: a revised system for cave data management".
diff --git a/handbook/troggle/trogindex.html b/handbook/troggle/trogindex.html index 4d0bbbe63..96bceb3d2 100644 --- a/handbook/troggle/trogindex.html +++ b/handbook/troggle/trogindex.html @@ -26,7 +26,6 @@
The second, the django administration user, is part of the core django Users/Groups system'django.contrib.auth'.
The original design of Troggle envisaged that expo cavers would have their own individual logins and manage their own accounts, which is what 'django-registration' is designed for. It easily handles the hundreds of accounts which would be necessary. (See "Troggle: a novel system for cave exploration information management", by Aaron Curtis) +
The original design of Troggle envisaged that expo cavers would have their own individual logins and manage their own accounts, which is what 'django-registration' is designed for. It easily handles the hundreds of accounts which would be necessary. (See " +Troggle: a revised system for cave data management")
The django administrator account is built-in by default and created when the database is initiated.
diff --git a/handbook/troggle/trogsimpler.html b/handbook/troggle/trogsimpler.html
index 958c65a2f..1ec0775e1 100644
--- a/handbook/troggle/trogsimpler.html
+++ b/handbook/troggle/trogsimpler.html
@@ -52,7 +52,7 @@ for displaying a table of members by expedition year, but what if you wanted to
logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that
list has been recorded in the logbook, but there is no way to access it because there is no connection between the
person's name in folk.csv and the entries he wrote in the logbook".
-[Adrian Curtis]
+[Aaron Curtis]
And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks,
survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent]
Troggle today has 8,200 lines of python (including comments and blank lines), plus 600 in imagekit and 200 in flatpages. 2,200 of those are in the parsers. Django itself has a lot more, including integration with TinyMCE in-browser HTML editor. ]
+ Troggle today has 6,400 non-comment lines of python and 2,500 non-comment lines of django HTML template code. Plus there is the integration with the in-browser HTML editor in JavaScript. Half of the python is in the parsers which will not change whatever we do. Django itself is much, much bigger and includes all the security middleware necessary in the web today.
+ But maintaining the code with the regular Django updates is a heavy job.]
Troggle is the software which runs the the expo cave survey data management and website.
- In early 2019 the university computing service upgraded its firewall rules which took the
-server offline completely.
-
-Wookey eventually managed to find us free space (a virtual machine)
-on a debian mirror server somewhere in Leicestershire (we think).
-This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
-
- At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata).
-
-
- The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
-
-The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
-
-The loser repo and expoweb repo need more care in migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
-
-Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
+ For earlier history see Website history.
-Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working on loser with him.
- Sam continues to work on upgrading django from v1.7 on python 2.7.17 . We would like to upgrade django as quickly as possible because old versions of django have unpatched security issues.
-Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.
+Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working with him on the loser (survex files) migration to git.
+ In May we were on django 1.7 and python 2.7.17. Sam continued to work on upgrading django from v1.7 . We wanted to upgrade django as quickly as possible because old versions of django had unpatched security issues.
+[Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.]
Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.
Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django incompatibility.
+ Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django plugin incompatibility.
In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.
Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted.
diff --git a/handbook/website-history.html b/handbook/website-history.html
index bf15f75e2..8d31545c6 100644
--- a/handbook/website-history.html
+++ b/handbook/website-history.html
@@ -92,12 +92,16 @@ haphazard nature of the development of the "script and spreadsheet" method meant
had an individual system for storing QMs. Without a standard system, it was sometimes unclear
how to correctly enter data.
- From "
-Troggle: a novel system for cave exploration information management", by Aaron Curtis, CUCC [with some additions]
+ From "
+Troggle: a revised system for cave data management", by Philip Sargent and Aaron Curtis, CUCC [with some additions].
+
+Original (2006) paper: "
+Troggle: a novel system for cave exploration information management", by Aaron Curtis.
The CUCC Website, which publishes the cave data, was originally created by
Andy Waddington in the early 1990s and was hosted by Wookey.
@@ -130,8 +134,6 @@ troggle, the survey data, the tunnel data. Seagrass was turned off at
the end of 2013, and the site has been hosted by Sam Wenham at the
university since Feb 2014.
-
-
In 2018 we had 4 repositories, 2 mercurial, 2 git
@@ -145,6 +147,26 @@ university since Feb 2014.
In spring 2018 Sam, Wookey and Paul Fox updated the Linux version and the Django version (i.e. troggle) to
something vaguely acceptable to the university computing service and fixed all the problems that were then observed.
+ In early 2019 the university computing service upgraded its firewall rules which took the
+server offline completely.
+
+Wookey eventually managed to find us free space (a virtual machine)
+on a debian mirror server somewhere in Leicestershire (we think).
+This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
+
+ At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata).
+
+
+ The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
+
+The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
+
+The loser repo and expoweb repo need more care in hg->git migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
+
+Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
+
For the current situation see expo systems status.
diff --git a/handbook/troggle/trogstatus.html b/handbook/troggle/trogstatus.html
index bb13ba525..df70d7a63 100644
--- a/handbook/troggle/trogstatus.html
+++ b/handbook/troggle/trogstatus.html
@@ -10,31 +10,13 @@
Troggle & Expo Systems - status update
Early 2019
-Wookey: July 2019
-May 2020 and django versions
June 2020
-
History in summary
+
2018
Early 2019
+Wookey: July 2019
+More recent