Updating troggle docum. links in handbook

This commit is contained in:
Philip Sargent 2020-07-31 23:20:20 +01:00
parent c5dde279d8
commit e417d28272
12 changed files with 49 additions and 40 deletions

View File

@ -22,6 +22,8 @@ data. Aside from the data collection requirements of the game itself, setting up
expedition) of cave exploration often involves collection of personal information ranging from
dates available to medical information to the desire to purchase an expedition t-shirt.
<p>
<img class="onleft" src ="i/qm-image.jpg" />
If an expedition will only happen once, low-tech methods are usually adequate to record
information. Any events that need to be recorded can go in a logbook. Survey notes must be
turned into finished cave sketches, without undue concern for the future expansion of those sketches.
@ -49,8 +51,10 @@ or even useful recipes for locally available food. The more of this information
wishes to keep, the more valuable an effective and user-friendly system of data management.
</div>
<p><em>From "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>
Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis, CUCC.</em>
<p><em>From "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Troggle:
a novel system for cave exploration information management</a>", by Aaron Curtis (2006) and
updated as "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>Troggle:
a revised system for cave data management</a>" in 2020.</em>
<hr />

View File

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 22 KiB

View File

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 53 KiB

View File

@ -62,7 +62,7 @@ detailed topics.</p>
</ul></li>
<li><a href="more.htm">More resources</a> on surveying</li>
<li>History of <a href="survey-broken.html">cave data archiving</a> here (broken links).</li>
<li>History of <a href="../website-history.html">CUCC cave data archiving</a>.</li>
<li>More obscure methods:
<ul>
<li>The <a href="lasers.htm">Loser plateau laser fixed points</a></li>

BIN
handbook/t/riscos.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 40 KiB

View File

@ -29,7 +29,7 @@ tl;dr - use <em>svx2qm.py</em>. Look at the output at:<br>
<a href="/expofiles/writeups/2019/qms2019.csv">qms2019.csv</a><br>
<h2>QMs - the fourfold path</h2>
<img class="onright" src ="qm-image.jpg" />
<img class="onright" src ="../i/qm-image.jpg" />
<p>You will be familiar with <a href="../survey/qmentry.html">documenting newly found QMs</a> in the survex file when you type it in. But QMs are only useful if they can be easily scanned by people planning the next pushing trip. That's what we are discussing here.
<p>There are four (and a half) ways we have used to manage QMs:

View File

@ -36,9 +36,9 @@ ALSO there have been tables added to the core representation which are not antic
<br><figcaption>Class Diagram (draft)</figcaption>
</figure>
</div>
<h3>Architecture and purpose</h3>
<p>Read the proposal: "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis</em>. But remember that this paper is an over-ambitious proposal. Only the core data management features have been built. We have none of the "person management" features, none of the "wallet progress" stuff and only two forms in use: for entering <var>cave</var> and <var>cave entrance</var> data.
<h3>Purpose</h3>
<p>The reasons why we have an online system at all are described in our <a href="../website-history.html">website history</a>.
<p>There is an introductory article "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>Troggle: a revised system for cave data management</a>".
<h3>Implementation in software</h3>
<p><a href="http://www.djangoproject.com/"><img class="onright" src="https://www.djangoproject.com/m/img/badges/djangopowered126x54.gif" border="0" alt="Powered by Django." title="Powered by Django." /></a>

View File

@ -26,7 +26,6 @@
<li><a href="trogspeculate.html">Troggle Architecture Speculations</a> - as in April 2020<br>
<li><a href="trog2030.html">Troggle in 2025-2030</a> - architectural evolution proposal<br>
</ul>
<a href="trogstatus.html">Troggle status</a> - Systems status and upgrade progress<br>
<a href="serverconfig.html">Troggle server configuration</a> - how to get troggle running on a new machine (incoimplete!)<br>
<a href="trogdjango.html">Troggle and django</a> - Upgrading to later django versions<br>
<a href="unittests.html">Troggle unit tests</a> - test suite for programmers<br>

View File

@ -26,7 +26,8 @@
<p> The second, the django administration user, is part of the core django Users/Groups system'<a href="https://docs.djangoproject.com/en/1.11/ref/contrib/auth/">django.contrib.auth</a>'.
<h2>Why this is how it is</h2>
<p>The original design of Troggle envisaged that expo cavers would have their own individual logins and manage their own accounts, which is what 'django-registration' is designed for. It easily handles the hundreds of accounts which would be necessary. (See "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis)
<p>The original design of Troggle envisaged that expo cavers would have their own individual logins and manage their own accounts, which is what 'django-registration' is designed for. It easily handles the hundreds of accounts which would be necessary. (See "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>
Troggle: a revised system for cave data management</a>")
<p>The django administrator account is built-in by default and created when the database is initiated.

View File

@ -52,7 +52,7 @@ for displaying a table of members by expedition year, but what if you wanted to
logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that
list has been recorded in the logbook, but there is no way to access it because there is no connection between the
person's name in folk.csv and the entries he wrote in the logbook".
[<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Adrian Curtis</a>]<em>
[<a href="/expofiles/documents/troggle/troggle2020.pdf" download>Aaron Curtis</a>]<em>
<p>And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks,
survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent]
</em>
@ -119,10 +119,11 @@ exist
</ul>
<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[This vastly underestimates the number of things that troggle does for us.
See "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Troggle: a novel system for
cave exploration information management</a>". And a VM is not required to run and debug troggle.
See "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>
Troggle: a revised system for cave data management</a>"</em>.] And a VM is not required to run and debug troggle.
Sam has produced a docker variant which he uses extensively.
<p>Troggle today has 8,200 lines of python (including comments and blank lines), plus 600 in imagekit and 200 in flatpages. 2,200 of those are in the parsers. Django itself has a lot more, including integration with TinyMCE in-browser HTML editor. </em>]
<p>Troggle today has 6,400 non-comment lines of python and 2,500 non-comment lines of django HTML template code. Plus there is the integration with the in-browser HTML editor in JavaScript. Half of the python is in the parsers which will not change whatever we do. Django itself is much, much bigger and includes all the security middleware necessary in the web today.
<p>But maintaining the code with the regular Django updates is <a href="trogdjango.html">a heavy job</a>.]
</div>
How much work would this actually take:
<ul>

View File

@ -10,31 +10,13 @@
<h1>Troggle & Expo Systems - status update</h1>
<p>Troggle is the software which runs the the expo cave survey data management and website.
<h3>Early 2019</h3>
<p>In early 2019 the university computing service upgraded its firewall rules which took the
server offline completely.
<p>
Wookey eventually managed to find us free space (a virtual machine)
on a debian mirror server somewhere in Leicestershire (we think).
This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
<p>At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata).
</div>
<h4>Wookey: July 2019</h4>
<p>The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
<p>
The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
<p>
The loser repo and expoweb repo need more care in migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
<p>
Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
<p>For earlier history see <a href="../website-history.html">Website history</a>.
<h4>May 2020 and django versions</h4>
<p>
Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working on loser with him.
<p>Sam continues to work on upgrading django from v1.7 on python 2.7.17 . We would like to upgrade django as quickly as possible because old versions of django have unpatched security issues.
Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.
Wookey has now moved 'expoweb' from mercurial to git largely "as-is" and will to use the git tools to patch up the history and to remove redundancies, rather than the original plan to tidy them up "at the time of conversion". Mark Shinwell is working with him on the loser (survex files) migration to git.
<p>In May we were on django 1.7 and python 2.7.17. Sam continued to work on upgrading django from v1.7 . We wanted to upgrade django as quickly as possible because old versions of django had unpatched security issues.
[Upgrading to later django versions is a real pig - not helped by the fact that all the tools to help do it are now out of date for these very old django releases.]
<ul>
<li>"Django 1.11 is the last version to support Python 2.7. Support for Django 1.11 ends in 2020." see: <a href="https://docs.djangoproject.com/en/3.0/faq/install/">django versions</a>. You will notice that we are really outstaying our welcome here, especially as python2.7 was <a href="https://python-release-cycle.glitch.me/">declared dead in January</a> this year.
@ -55,7 +37,7 @@ tackling the next step: thinking deeply about when we migrate from django
<p>
Enforced time at home is giving us a new impetus to writing and restructuring the documentation for everything.
<h4>June 2020</h4>
<p>Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django incompatibility.
<p>Sam was a bit overworked in trying to get an entire university to work remotely so Philip [Sargent] got troggle on django 1.7 to work on python 3.5 and then 3.8. He then did the slog of migrating it through the django versions up to 1.11.29 - the last version before django 2.0 . 1.11.29 is an LTS (long term support) version of django. In doing this we had to retreat to python3.7 due to a django plugin incompatibility.
<p>
In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.
<p>Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted.

View File

@ -92,12 +92,16 @@ haphazard nature of the development of the "script and spreadsheet" method meant
had an individual system for storing QMs. Without a standard system, it was sometimes unclear
how to correctly enter data.
<p><em>From "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>
Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis, CUCC [with some additions]</em>
<p><em>From "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>
Troggle: a revised system for cave data management</a>", by Philip Sargent and Aaron Curtis, CUCC [with some additions]</em>.
<em>Original (2006) paper: "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>
Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis</em>.
<hr />
<h2>History in summary</h2>
<a href="https://en.wikipedia.org/wiki/RISC_OS"><img class="onright" src="t/riscos.jpg" width="100px"/></a>
<p>The CUCC Website, which publishes the cave data, was originally created by
Andy Waddington in the early 1990s and was hosted by Wookey.
@ -130,8 +134,6 @@ troggle, the survey data, the tunnel data. Seagrass was turned off at
the end of 2013, and the site has been hosted by Sam Wenham at the
university since Feb 2014.
<h3>2018</h3>
<p>In 2018 we had 4 repositories, 2 mercurial, 2 git
@ -145,6 +147,26 @@ university since Feb 2014.
<p>In spring 2018 Sam, Wookey and Paul Fox updated the Linux version and the Django version (i.e. troggle) to
something vaguely acceptable to the university computing service and fixed all the problems that were then observed.
<h3>Early 2019</h3>
<p>In early 2019 the university computing service upgraded its firewall rules which took the
server offline completely.
<p>
Wookey eventually managed to find us free space (a virtual machine)
on a debian mirror server somewhere in Leicestershire (we think).
This move to a different secure server means that all ssh to the server now needs to use cryptographic keys tied to individual machines. There is an expo-nerds email list (all mailing lists are now hosted on wookware.org as the university list system restricted what non-Raven-users could do) to coordinate server fettling.
<p>At the beginning of the 2019 expo two repos had been moved from mercurial to git: troggle and drawings (formerly called tunneldata).
</div>
<h4>Wookey: July 2019</h4>
<p>The troggle software has been migrated to git, and the old erebus and cvs branches (pre 2010) removed. Some decrufting was done to get rid of log files, old copies of embedded javascript (codemirror, jquery etc) and some fat images no longer used.
<p>
The tunneldata repo has also been migrated to git, and renamed 'drawings' as it includes therion data too these days.
<p>
The loser repo and expoweb repo need more care in hg->git migration (expoweb is the website content - which is published by troggle). Loser should have the old 1999-2004 CVS history restored, and maybe Tom's annual snapshots from before that, so ancient history can usefully be researched (sometimes useful). It's also a good idea to add the 2015, 2016 and 2017 ARGE data we got (in 2017) added in the correct years so that it's possible to go back to an 'end of this year' checkout and get an accurate view of what was found (for making plots and length stats). All of that requires some history rewriting, which is best done at the time of conversion.
<p>
Similarly expoweb is full of bloat from fat images and surveys and one 82MB thesis that got checked in and then removed. Clearing that out is a good idea. I have a set of 'unused fat blob' lists which can be stripped out with git-gilter. It's not hard to make a 'do the conversion' script, ready for sometime after expo 2019 has calmed down.
<h3>More recent</h3>
<p>
For the current situation see <a href="troggle/trogstatus.html">expo systems status</a>.