From e417d28272d82b3c27d9dd5e9dda2de54f768221 Mon Sep 17 00:00:00 2001 From: Philip Sargent Date: Fri, 31 Jul 2020 23:20:20 +0100 Subject: [PATCH] Updating troggle docum. links in handbook --- handbook/datamgt.html | 8 ++++-- handbook/{troggle => i}/mtv_drawing2.jpg | Bin handbook/{troggle => i}/qm-image.jpg | Bin handbook/survey/index.htm | 2 +- handbook/t/riscos.jpg | Bin 0 -> 41099 bytes handbook/troggle/scriptsqms.html | 2 +- handbook/troggle/trogarch.html | 6 ++--- handbook/troggle/trogindex.html | 1 - handbook/troggle/trogregistr.html | 3 ++- handbook/troggle/trogsimpler.html | 9 ++++--- handbook/troggle/trogstatus.html | 28 ++++----------------- handbook/website-history.html | 30 ++++++++++++++++++++--- 12 files changed, 49 insertions(+), 40 deletions(-) rename handbook/{troggle => i}/mtv_drawing2.jpg (100%) rename handbook/{troggle => i}/qm-image.jpg (100%) create mode 100644 handbook/t/riscos.jpg diff --git a/handbook/datamgt.html b/handbook/datamgt.html index 3201d9fba..2f4133f90 100644 --- a/handbook/datamgt.html +++ b/handbook/datamgt.html @@ -22,6 +22,8 @@ data. Aside from the data collection requirements of the game itself, setting up expedition) of cave exploration often involves collection of personal information ranging from dates available to medical information to the desire to purchase an expedition t-shirt.

+ + If an expedition will only happen once, low-tech methods are usually adequate to record information. Any events that need to be recorded can go in a logbook. Survey notes must be turned into finished cave sketches, without undue concern for the future expansion of those sketches. @@ -49,8 +51,10 @@ or even useful recipes for locally available food. The more of this information wishes to keep, the more valuable an effective and user-friendly system of data management. -

From " -Troggle: a novel system for cave exploration information management", by Aaron Curtis, CUCC. +

From "Troggle: +a novel system for cave exploration information management", by Aaron Curtis (2006) and +updated as "Troggle: +a revised system for cave data management" in 2020.

diff --git a/handbook/troggle/mtv_drawing2.jpg b/handbook/i/mtv_drawing2.jpg similarity index 100% rename from handbook/troggle/mtv_drawing2.jpg rename to handbook/i/mtv_drawing2.jpg diff --git a/handbook/troggle/qm-image.jpg b/handbook/i/qm-image.jpg similarity index 100% rename from handbook/troggle/qm-image.jpg rename to handbook/i/qm-image.jpg diff --git a/handbook/survey/index.htm b/handbook/survey/index.htm index a05b169d4..9d2dbea0f 100644 --- a/handbook/survey/index.htm +++ b/handbook/survey/index.htm @@ -62,7 +62,7 @@ detailed topics.

  • More resources on surveying
    • -
  • History of cave data archiving here (broken links).
    • +
  • History of CUCC cave data archiving.
  • More obscure methods: -Troggle status - Systems status and upgrade progress
    Troggle server configuration - how to get troggle running on a new machine (incoimplete!)
    Troggle and django - Upgrading to later django versions
    Troggle unit tests - test suite for programmers
    diff --git a/handbook/troggle/trogregistr.html b/handbook/troggle/trogregistr.html index a4ef005af..85fed1140 100644 --- a/handbook/troggle/trogregistr.html +++ b/handbook/troggle/trogregistr.html @@ -26,7 +26,8 @@

    The second, the django administration user, is part of the core django Users/Groups system'django.contrib.auth'.

    Why this is how it is

    -

    The original design of Troggle envisaged that expo cavers would have their own individual logins and manage their own accounts, which is what 'django-registration' is designed for. It easily handles the hundreds of accounts which would be necessary. (See "Troggle: a novel system for cave exploration information management", by Aaron Curtis) +

    The original design of Troggle envisaged that expo cavers would have their own individual logins and manage their own accounts, which is what 'django-registration' is designed for. It easily handles the hundreds of accounts which would be necessary. (See " +Troggle: a revised system for cave data management")

    The django administrator account is built-in by default and created when the database is initiated. diff --git a/handbook/troggle/trogsimpler.html b/handbook/troggle/trogsimpler.html index 958c65a2f..1ec0775e1 100644 --- a/handbook/troggle/trogsimpler.html +++ b/handbook/troggle/trogsimpler.html @@ -52,7 +52,7 @@ for displaying a table of members by expedition year, but what if you wanted to logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that list has been recorded in the logbook, but there is no way to access it because there is no connection between the person's name in folk.csv and the entries he wrote in the logbook". -[Adrian Curtis] +[Aaron Curtis]

    And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks, survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent] @@ -119,10 +119,11 @@ exist

    [This vastly underestimates the number of things that troggle does for us. -See "Troggle: a novel system for -cave exploration information management". And a VM is not required to run and debug troggle. +See " +Troggle: a revised system for cave data management".] And a VM is not required to run and debug troggle. Sam has produced a docker variant which he uses extensively. -

    Troggle today has 8,200 lines of python (including comments and blank lines), plus 600 in imagekit and 200 in flatpages. 2,200 of those are in the parsers. Django itself has a lot more, including integration with TinyMCE in-browser HTML editor. ] +

    Troggle today has 6,400 non-comment lines of python and 2,500 non-comment lines of django HTML template code. Plus there is the integration with the in-browser HTML editor in JavaScript. Half of the python is in the parsers which will not change whatever we do. Django itself is much, much bigger and includes all the security middleware necessary in the web today. +

    But maintaining the code with the regular Django updates is a heavy job.]

    How much work would this actually take:
      diff --git a/handbook/troggle/trogstatus.html b/handbook/troggle/trogstatus.html index bb13ba525..df70d7a63 100644 --- a/handbook/troggle/trogstatus.html +++ b/handbook/troggle/trogstatus.html @@ -10,31 +10,13 @@

      Troggle & Expo Systems - status update

      Troggle is the software which runs the the expo cave survey data management and website. -

      Early 2019

      -

      In early 2019 the university computing service upgraded its firewall rules which took the -server offline completely. -

      -Wookey eventually managed to find us free space (a virtual machine) -on a debian mirror server somewhere in Leicestershire (we think). -This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling. - -

      At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata). - - -

      Wookey: July 2019

      -

      The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used. -

      -The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days. -

      -The loser repo and expoweb repo need more care in migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion. -

      -Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down. +

      For earlier history see Website history.

      May 2020 and django versions

      -Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working on loser with him. -

      Sam continues to work on upgrading django from v1.7 on python 2.7.17 . We would like to upgrade django as quickly as possible because old versions of django have unpatched security issues. -Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases. +Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working with him on the loser (survex files) migration to git. +

      In May we were on django 1.7 and python 2.7.17. Sam continued to work on upgrading django from v1.7 . We wanted to upgrade django as quickly as possible because old versions of django had unpatched security issues. +[Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.]

      • "Django 1.11 is the last version to support Python 2.7. Support for Django 1.11 ends in 2020." see: django versions. You will notice that we are really outstaying our welcome here, especially as python2.7 was declared dead in January this year. @@ -55,7 +37,7 @@ tackling the next step: thinking deeply about when we migrate from django

        Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.

        June 2020

        -

        Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django incompatibility. +

        Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django plugin incompatibility.

        In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.

        Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted. diff --git a/handbook/website-history.html b/handbook/website-history.html index bf15f75e2..8d31545c6 100644 --- a/handbook/website-history.html +++ b/handbook/website-history.html @@ -92,12 +92,16 @@ haphazard nature of the development of the "script and spreadsheet" method meant had an individual system for storing QMs. Without a standard system, it was sometimes unclear how to correctly enter data. -

        From " -Troggle: a novel system for cave exploration information management", by Aaron Curtis, CUCC [with some additions] +

        From " +Troggle: a revised system for cave data management", by Philip Sargent and Aaron Curtis, CUCC [with some additions]. + +Original (2006) paper: " +Troggle: a novel system for cave exploration information management", by Aaron Curtis.

        History in summary

        +

        The CUCC Website, which publishes the cave data, was originally created by Andy Waddington in the early 1990s and was hosted by Wookey. @@ -130,8 +134,6 @@ troggle, the survey data, the tunnel data. Seagrass was turned off at the end of 2013, and the site has been hosted by Sam Wenham at the university since Feb 2014. - -

        2018

        In 2018 we had 4 repositories, 2 mercurial, 2 git @@ -145,6 +147,26 @@ university since Feb 2014.

        In spring 2018 Sam, Wookey and Paul Fox updated the Linux version and the Django version (i.e. troggle) to something vaguely acceptable to the university computing service and fixed all the problems that were then observed. +

        Early 2019

        +

        In early 2019 the university computing service upgraded its firewall rules which took the +server offline completely. +

        +Wookey eventually managed to find us free space (a virtual machine) +on a debian mirror server somewhere in Leicestershire (we think). +This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling. + +

        At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata). + + +

        Wookey: July 2019

        +

        The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used. +

        +The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days. +

        +The loser repo and expoweb repo need more care in hg->git migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion. +

        +Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down. +

        More recent

        For the current situation see expo systems status.