Troggle is the software which runs the the expo cave survey data management and website.
For earlier history see Website history.
Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working with him on the loser (survex files) migration to git.
In May we were on django 1.7 and python 2.7.17. Sam continued to work on upgrading django from v1.7 . We wanted to upgrade django as quickly as possible because old versions of django had unpatched security issues. [Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.]
We planned to upgrade from django 1.7 to django 1.11, then port from python2 to python3 on the same version of django, then upgrade to as recent a version of django as we could. But we have discovered that django1.7 works just fine with python3, so we will move the development version to python3 during June and then upgrade the server operating system from Debian stretch to buster before tackling the next step: thinking deeply about when we migrate from django to something else.
Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.
Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django plugin incompatibility.
In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.
Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted.
Wookey upgraded debian on the server from 9 stretch to 10 buster and we got the python3 development of troggle running as the public version (with some http:// and https:// glitches) by 23rd July. Buster will be in-support definitely until June 2024 so we are rather pleased to be on a "not ancient" version of the operating system at last. This concided with a last tweak at improving the full cave data file import so now it runs on the server in ~80 seconds. Which is considerably more useful than the ~5 hours it was taking earlier this year.
Lockdown has been good to troggle. During March and April Philip migrated troggle up to Django 2.2.19, excising the ancient and unused user registration system on the way. Django 2.2 LTS is a long-term stable relase which will be in-support by Django until April next year. Wookie discovered and ran the Django system testsuite on the Debian server thus enabling us to use a necessary obstensibly outdated link between Django and the database MariaDB. As of April 9th troggle is now running on software which is actually 'in date'.
We plan to stick with this configuration for a year.
Next April (2022) we may to the forthcoming debian stable release 11 bullseye. At that point debian will have python 3.8 as standard and we also hope to migrate to django 3.2 LTS which will be a year old by then and which will be supported until April 2025.
With any luck that will be the last of our involvement with django migrations as we may not move on from using django 3.2 until we stop using django altogether, see troggle architecture speculations.
Return to: