expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

HTML5 <details> & <summary> reformatting

Browse Source
This commit is contained in:
Philip Sargent 2021-04-09 16:15:51 +01:00
parent f2d83f1926
commit f52b3d87ae
3 changed files with 112 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
handbook/troggle/trogspeculate.html
View File

@ -47,10 +47,12 @@ browser (or a progressive web app) using a python interpreter running in
javascript. [yeah, expofiles would need some subset labelled as needing to
be forcibly downloaded, but the rest coming only on demand.] Some django
enthusiast must have done this already surely ? Ah yes, Brython.<br>
<a href="https://github.com/brython-dev/brython">github.com/brython-dev/brython</a><br>
<a href="https://www.brython.info/">www.brython.info</a>
<a href="https://github.com/brython-dev/brython">github.com/brython-dev/brython</a> &nbsp; &nbsp;
<a href="https://www.brython.info/">www.brython.info</a> and <br>
<a href="https://skulpt.org/gallery.html">Skulpt</a> (which has, since 2017, a full-blown
<a href="https://anvil.works/features">commerical system(</a>) built on top of it - by a CambridgeCL spinout)</br>
<p>
Which is fun, but not useful. And not just because it is immature. None of
Which is fun, but not useful. And not just because it may be immature. None of
this addresses <strong>our biggest problem: devising something that can be
maintained by fewer, less-expert people who can only devote short snippets
of time and not long-duration immersion</strong>.
@ -106,7 +108,10 @@ not a function within the python codebase.
(We could retain an HTML templating engine though.)
<h3><em>Addendum</em></h3>
<h3><em>Addendum 1</em></h3>
<p>The above discussion is extremely ignorant in a couple of respects. Now (April 2021) we can properly appreciate that the part of Django that interacts with a database is actually a small part of the system. The http request/response engine is not easily replaced. And the 90 or so HTML templates do not just reformat the data given to them in python dictionaries: they directly query and traverse the database to produce tabular output. So if we 'took out' the database, most of our templates would fail utterly and need completely rewriting. It could be done, but the manpower requirement is not trivial.
<h3><em>Addendum 2</em></h3>
<p>There is a templating engine <a href="https://mozilla.github.io/nunjucks/">Nunjucks</a>
which is a port to JavaScript of the Django templating system we use
(via <a href="https://palletsprojects.com/p/jinja/">Jinja</a> - these are the same people who do Flask). This would be an obvious thing to use if we needed to go in that direction.

28
handbook/troggle/trogstatus.html
View File

@ -12,32 +12,10 @@
<p>For earlier history see <a href="../website-history.html">Website history</a>.
<h4>May 2020 and django versions</h4>
<p>
Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working with him on the loser (survex files) migration to git.
<p>In May we were on django 1.7 and python 2.7.17. Sam continued to work on upgrading django from v1.7 . We wanted to upgrade django as quickly as possible because old versions of django had unpatched security issues.
[Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.]
<ul>
<li>"Django 1.11 is the last version to support Python 2.7. Support for Django 1.11 ends in 2020." see: <a href="https://docs.djangoproject.com/en/3.0/faq/install/">django versions</a>. You will notice that we are really outstaying our welcome here, especially as python2.7 was <a href="https://python-release-cycle.glitch.me/">declared dead in January</a> this year.
<li>For a table displaying the various versions of django and support expiry dates
see <a href="https://www.djangoproject.com/download/">the django download</a> page.
Django 1.7 expired in December 2015.
Django: <a href="https://www.djangoproject.com/download/#supported-versions">full deprecation timeline</a>.
<li>Ubuntu 20.04 came out on 23rd April but it does not support python2 at all. So we cannot use it for software maintenance (well be can, but only using non-recommended software, which is what we are trying to get away from).
</ul>
<p>We planned to upgrade from django 1.7 to django 1.11, then port from python2 to python3 on
the same version of django, then upgrade to as recent a version of django as we could. But we have
discovered that django1.7 works just fine with <a href="https://docs.djangoproject.com/en/1.10/topics/python3/">python3</a>, so we will move the development version to python3 during June and
then upgrade the server operating system from Debian <var>stretch</var> to <var>buster</var> before
tackling the next step: thinking deeply about when we migrate from django
<a href="trogdesign.html">to something else</a>.
</p>
<p>
Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.
<h4>June 2020</h4>
<p>Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django plugin incompatibility.
<p>Sam was a bit overworked in trying to get an entire university to work remotely during Covid lockdown so Philip [Sargent] started on the python2/3 conversion and got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django plugin incompatibility.
<p>
In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.
<p>Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted.
@ -46,9 +24,9 @@ In the course of these migrations several unused or partly-used django plugins w
<h4>April 2021</h4>
<p>Lockdown has been good to troggle. During March and April Philip migrated troggle up to Django 2.2.19, excising the ancient and unused user registration system on the way. Django 2.2 LTS is a long-term stable relase which will be in-support by Django until April next year. Wookie discovered and ran the Django system testsuite on the Debian server thus enabling us to use a necessary obstensibly outdated link between Django and the database MariaDB. As of April 9th troggle is now running on software which is actually 'in date'.</p>
<p>Lockdown has been good to troggle. During March and April Philip migrated troggle up to Django 2.2.19, excising the ancient and unused user registration system on the way. Django 2.2 LTS is a long-term stable relase which will be in-support by Django until April next year. Wookie discovered and ran the Django system testsuite on the Debian server thus enabling us to use a necessary (but obstensibly outdated) link between Django and the database MariaDB. As of April 9th troggle is now running on software which is actually 'in date'.</p>
<p>We plan to stick with this configuration for a year.</p>
<p>Next April (2022) we may to the forthcoming debian stable release 11 <var>bullseye</var>. At that point debian will have python 3.8 as standard and we also hope to migrate to django 3.2 LTS which will be a year old by then and which will be <a href="https://www.djangoproject.com/download/#supported-versions">supported until April 2025</a>.
<p>Next April (2022) we may upgrade to the forthcoming debian stable release 11 <var>bullseye</var>. At that point debian will have python 3.8 as standard. We also hope to migrate to django 3.2 LTS which will be a year old by then and which will be <a href="https://www.djangoproject.com/download/#supported-versions">supported until April 2025</a>.
<p>With any luck that will be the last of our involvement with django migrations as we may not move on from using django 3.2 until we stop using django altogether, see <a href="trogspeculate.html">troggle architecture speculations</a>.
<hr />
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>

130
handbook/website-history.html
View File

@ -15,6 +15,22 @@ Justified text is hard to read:
https://designshack.net/articles/mobile/the-importance-of-designing-for-readability/
https://designforhackers.com/blog/justify-text-html-css/
-->
<style>
summary::-webkit-details-marker {
color: red;
font-size: 200%;
}
summary {
color: #005151;
font-family: Tahoma,'Trebuchet MS','Lucida Grande',Verdana,Arial,Sans-Serif;
font-size: 12pt;
margin-block-start: 1.33em;
margin-block-end: 1.33em;
margin-inline-start: 0px;
margin-inline-end: 0px;
font-weight: bold;
}
</style>
<h2>Early history</h2>
<p>
Over 42 years, CUCC has developed methods for handling such information. Refinements in data
@ -59,22 +75,29 @@ used to generate a prospecting map as a JPEG image. The system of automatically
webpages from data files reduced the need for repetitive manual HTML coding. Centralized storage
of all caves in a large .CSV file with a cave on each row made the storage of new information
more straightforward.
<hr />
<h3>HTML and browser incompatibilities</h3>
<p>This was a much bigger problem in the past than it is possible to imagine now
(most of the early editing is done on an Acorn platform using !Zap). So much so
that we have a separate page all about it: "<a href="computing/useany.html">Platform portability</a> - making the website work widely".
<h2>HTML and the website</h2>
<a href="https://en.wikipedia.org/wiki/RISC_OS"><img class="onright" src="t/riscos.jpg" width="100px"/></a>
<p>From having a set of HTML files, it was a small step to publish a website. The CUCC Expo Website, which publishes the cave data, was originally created by
Andy Waddington in the early 1990s and was hosted by Wookey.
<details>
<summary>1999 scripts and spreadsheets</summary>
The version control system was <a href="https://www.nongnu.org/cvs/">CVS</a>. The whole site was just static HTML, carefully
designed to be RISCOS-compatible (hence the short 10-character filenames)
as both Wadders and Wookey were <a href="https://en.wikipedia.org/wiki/RISC_OS">RISCOS"</a> people then (in the early 1990s).
Wadders wrote a huge amount of info collecting expo history, photos, cave data etc.</p>
<h3>Version control</h3>
<p>Another important element of this system was <a href="computing/repos.html">version control</a>. The entire data structure was
stored initially in a <a href="https://en.wikipedia.org/wiki/Concurrent_Versions_System">Concurrent Version System</a> repository, and later migrated to
<a href="https://en.wikipedia.org/wiki/Apache_Subversion">Subversion</a> [<em>now using <a href="computing/repos.html">git</a> in 2020</em>].
Any edits to the spreadsheets which caused the scripts to fail, breaking the
website, could be easily reversed.
<p>Martin Green added the <em>survtab.csv</em> file to contain tabulated data for many caves around 1999, and a
script to generate the index pages from it. Dave Loeffler added scripts and programs to generate the
prospecting maps in 2004. The server moved to Mark Shinwell's machine in the early
2000s, and the version control system was updated to <a href="https://subversion.apache.org/">subversion</a>.</p>
</details>
<h3>Other types of data</h3>
<p>However, not all types of data could be stored in spreadsheets or survey files. In order a
<details>
<summary>Breaking out of spreadsheet cells into HTML</summary>
<p>Not all types of data could be stored in spreadsheets or survey files. In order a
display descriptions on the webpage for an individual cave, the entire description, written in
HTML, had to be typed into a spreadsheet cell. A spreadsheet cell makes for an extremely awkward
HTML editing environment. To work around this project, descriptions for large caves were written
@ -103,24 +126,28 @@ Troggle: a revised system for cave data management</a>", by Philip Sargent and A
<em>Original (2006) paper: "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>
Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis</em>.
<hr />
</details>
<h2>History in summary</h2>
<details>
<summary>HTML and browser incompatibilities</summary>
<p>This was a much bigger problem in the past than it is possible to imagine now
(most of the early editing is done on an Acorn platform using !Zap). So much so
that we have a separate page all about it: "<a href="computing/useany.html">Platform portability</a> - making the website work widely".
</details>
<a href="https://en.wikipedia.org/wiki/RISC_OS"><img class="onright" src="t/riscos.jpg" width="100px"/></a>
<p>The CUCC Website, which publishes the cave data, was originally created by
Andy Waddington in the early 1990s and was hosted by Wookey.
<details>
<summary>Version control</summary>
<p>Another important element of this system was <a href="computing/repos.html">version control</a>. The entire data structure was
stored initially in a <a href="https://en.wikipedia.org/wiki/Concurrent_Versions_System">Concurrent Version System</a> repository, and later migrated to
<a href="https://en.wikipedia.org/wiki/Apache_Subversion">Subversion</a> [<em>now using <a href="computing/repos.html">git</a> in 2020</em>].
Any edits to the spreadsheets which caused the scripts to fail, breaking the
website, could be easily reversed.
</details>
The version control system was <a href="https://www.nongnu.org/cvs/">CVS</a>. The whole site was just static HTML, carefully
designed to be RISCOS-compatible (hence the short 10-character filenames)
as both Wadders and Wookey were <a href="https://en.wikipedia.org/wiki/RISC_OS">RISCOS"</a> people then (in the early 1990s).
Wadders wrote a huge amount of info collecting expo history, photos, cave data etc.</p>
<p>Martin Green added the <em>survtab.csv</em> file to contain tabulated data for many caves around 1999, and a
script to generate the index pages from it. Dave Loeffler added scripts and programs to generate the
prospecting maps in 2004. The server moved to Mark Shinwell's machine in the early
2000s, and the version control system was updated to <a href="https://subversion.apache.org/">subversion</a>.</p>
<details>
<summary>2006 and troggle</summary
<p>In 2006 Aaron Curtis decided that a more modern set of generated, database-based pages
made sense, and so wrote Troggle.
This uses Django to generate pages.
@ -140,7 +167,16 @@ troggle, the survey data, the tunnel data. Seagrass was turned off at
the end of 2013, and the site has been hosted by Sam Wenham at the
university since Feb 2014.
<h3>2018</h3>
<p><em>From "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>
Troggle: a revised system for cave data management</a>", by Philip Sargent and Aaron Curtis, CUCC [with some additions]</em>.
<em>Original (2006) paper: "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>
Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis</em>.
</details>
<details>
<summary>2018 Four repositories</summary>
<p>In 2018 we had 4 repositories, 2 mercurial, 2 git
<ul>
@ -152,8 +188,11 @@ university since Feb 2014.
<p>In spring 2018 Sam, Wookey and Paul Fox updated the Linux version and the Django version (i.e. troggle) to
something vaguely acceptable to the university computing service and fixed all the problems that were then observed.
</details>
<h3>Early 2019</h3>
<details>
<summary>Early 2019 - Leaving the Computing service</summary>
<p>In early 2019 the university computing service upgraded its firewall rules which took the
server offline completely.
<p>
@ -162,9 +201,12 @@ on a debian mirror server somewhere in Leicestershire (we think).
This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
<p>At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata).
</div>
<h4>Wookey: July 2019</h4>
</details>
<details>
<summary>Wookey: July 2019</summary>
<p>The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
<p>
The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
@ -172,12 +214,40 @@ The tunneldata repo has also been migrated to git, and renamed 'drawings' as it
The loser repo and expoweb repo need more care in hg->git migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
<p>
Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
</details>
<details>
<summary id='#may2020'>May 2020 and django</summary>
<p>
Wookey has now moved 'expoweb' from mercurial to git largely "as-is". Mark Shinwell has said that he will help on the loser (survex files) migration to git.
<p>In May we were on django 1.7 and python 2.7.17. Sam continued to work on upgrading django from v1.7 . We wanted to upgrade django as quickly as possible because old versions of django had unpatched security issues.
[Upgrading to later django versions <a href="troggle/trogdjangup.html">is a real pig</a> - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.]
<ul>
<li>"Django 1.11 is the last version to support Python 2.7. Support for Django 1.11 ends in 2020." see: <a href="https://docs.djangoproject.com/en/3.0/faq/install/">django versions</a>. You will notice that we are really outstaying our welcome here, especially as python2.7 was <a href="https://python-release-cycle.glitch.me/">declared dead in January</a> this year.
<li>For a table displaying the various versions of django and support expiry dates
see <a href="https://www.djangoproject.com/download/">the django download</a> page.
Django 1.7 expired in December 2015.
Django: <a href="https://www.djangoproject.com/download/#supported-versions">full deprecation timeline</a>.
<li>Ubuntu 20.04 came out on 23rd April but it does not support python2 at all. So we cannot use it for software maintenance (well be can, but only using non-recommended software, which is what we are trying to get away from).
</ul>
<p>We planned to upgrade from django 1.7 to django 1.11, then port from python2 to python3 on
the same version of django, then upgrade to as recent a version of django as we could. But we have
discovered that django1.7 works just fine with <a href="https://docs.djangoproject.com/en/1.10/topics/python3/">python3</a>, so we will move the development version to python3 during June and
then upgrade the server operating system from Debian <var>stretch</var> to <var>buster</var> before
tackling the next step: thinking deeply about when we migrate from django
<a href="trogdesign.html">to something else</a>.
</p>
<p>
Enforced time at home under covid lockdown is giving us a new impetus to writing and restructuring the documentation for everything.
</details>
<h3>More recent</h3>
<p>
For the current situation see <a href="troggle/trogstatus.html">expo systems status</a>.
<hr />
</div>
Return to<br />
<a href="computing/onlinesystems.html">expo online systems overbiew</a><br />