Troggle is the software which runs the the expo cave survey data management and website.
In early 2019 the university computing service upgraded its firewall rules which took the server offline completely.
Wookey eventually managed to find us free space (a virtual machine) on a debian mirror server somewhere in Leicestershire (we think). This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata).
The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
The loser repo and expoweb repo need more care in migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working on loser with him.
Sam continues to work on upgrading django from v1.7 on python 2.7.17 . We would like to upgrade django as quickly as possible because old versions of django have unpatched security issues. Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.
We planned to upgrade from django 1.7 to django 1.11, then port from python2 to python3 on the same version of django, then upgrade to as recent a version of django as we could. But we have discovered that django1.7 works just fine with python3, so we will move the development version to python3 during June and then upgrade the server operating system from Debian stretch to buster before tackling the next step: thinking deeply about when we migrate from django to something else.
Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.
Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django incompatibility.
In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.
Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted.
Wookey upgraded debian on the server from 9 stretch to 10 buster and we got the python3 development of troggle running as the public version (with some http:// and https:// glitches) by 23rd July. Buster will be in-support definitely until June 2024 so we are rather pleased to be on a "not ancient" version of the operating system at last. This concided with a last tweak at improving the full cave data file import so now it runs on the server in ~200 seconds. Which is considerably more useful than the ~5 hours it was taking earlier this year.
We plan to stick with debian 10 buster, django 1.11.29 and python 3.7.5 (the standard on buster) until spring 2021 when we may upgrade debian to the forthcoming stable release 11 bullseye. At that point debian will have python 3.8 as standard and we will migrate to django 2.x, hopefully getting as far as django 2.2 which is an LTS and actually in support until April 2022.
With any luck that will be the last of our involvement with django migrations as we may not move on from using django 2.2 until we stop using django altogether, see troggle architecture speculations.
Return to: