expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2025-12-08 14:54:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

Troggle programmers guide - beginnings

Browse Source
This commit is contained in:
Philip Sargent
2021-04-29 17:52:28 +01:00
parent b09a66b6ff
commit 07b5b8aa8e
13 changed files with 195 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
handbook/troggle/trogindex.html
View File

@@ -36,7 +36,7 @@
<a href="trogstatus.html">Troggle & expo systems - status update</a> - where we are now<br>
<br>
<a href="trognotes.html">Troggle Notes</a> - manual notes - (to be moved/edited)<br>
<a href="trognotes.html">Troggle Programmers's Notes</a> - improving, updating, fixing, refactoring<br>
<a href="trogassumptions.html">Pre-Troggle Assumptions</a> - list of assumptions<br>
<a href="archnotes.html">Archive Notes</a> - old ideas and original discussions<br>
<br>

11
handbook/troggle/trogintro.html
View File

@@ -13,9 +13,10 @@
<ul>
<li>For day to day cave survey work on the expo laptop, see: the expo <a href="../survey/newcave.html">survey handbook</a>.
<li>For survey data management on your laptop: set up <a href="../computing/basiclaptop.html">your own laptop</a>.
<li>For most survey data management: <a href="../computing/manual.html">expo data management manual</a>.
<li>For data management using your laptop: set up <a href="../computing/basiclaptop.html">your own laptop</a>.
<li>For the history of expo use of computers, see: <a href="../website-history.html">website and troggle history</a>.
<li>For troggle maintenance, see: the <a href="../troggle/trogmanual.html">troggle maintainers manual</a>.
<li>For system maintenance, see: the <a href="../troggle/trogmanual.html">system maintainers manual</a>.
</ul>
<h3 id="what">Troggle - what it does</a></h3>
@@ -36,7 +37,9 @@ The troggle software is written and maintained by expo members.
<li><a href="/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
<p>If you want to find out how to do something using troggle, then you may find it the <a href="../troggle/trogmanual.html">troggle maintainers and advanced users manual</a
<p>If you want to find out how to do something using troggle, then you may find it the
<a href="../computing/manual.html">data maintenance manual</a> or the
<a href="../troggle/trogmanual.html">system maintenance manual</a>
<h3 id="troggle">Troggle - why we developed it</a></h3>
@@ -65,7 +68,7 @@ how to use it</a> and <em>how to tidy up afterwards</em>].
<p>
All the data of all kinds is stored in files. When troggle starts up it imports that data from the files. There are other scripts doing useful things (folk, wallets) and these too get their data from files.
<p>There is never any need to back up or archive the database as it is rebuilt from files. Rebuilding troggle and re-importing all the data on the server takes about 4 minutes (as of July 2020).
<p>There is never any need to back up or archive the database as it is rebuilt from files. Rebuilding troggle and re-importing all the data on the server takes about 10 minutes (as of May 2021).
<hr />
Go on to:

18
handbook/troggle/trogmanual.html
View File

@@ -11,8 +11,11 @@
<h1>Troggle - Maintenance Manuals</h1>
<p>Troggle is the software which runs the the expo cave survey data management and website.
<p>If you are looking for how to manage or correct cave survey data then this is not the right place. You want the
<a href="../computing/manual.html"><em>Data</em> Maintenance</a> manual.
<p>This part of the handbook is intended for people maintaining the troggle software:
<p>This part of the handbook is intended for people maintaining the troggle <em>software</em>:
<a href="https://www.goodreads.com/book/show/1155986.The_Tricky_Troggle">
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
<ul>
@@ -25,18 +28,13 @@
<li><a href="trogdjango.html">Troggle and django</a> - Upgrading to later django versions
<li><a href="trognotes.html">Uncategorised notes</a> and past speculations
<li><a href="trognotes.html">Programmers' guide</a> - and index to other documentation
</ul>
<p>Troggle is completely unlike any other django installation: it has a database, but the database is rebuilt from files every time it starts.
<p>Most of the data entry into troggle happens during or just after the expedition.
<p>Most of the data entry into troggle happens during or just after the expedition: i.e. once a year.
<h4>Advanced Users' Manual</h4>
<p>We don't have one of these. You may find what you are looking for in <a href="scriptsother.html">Other scripts</a> above. But there are a few things which are not really 'maintenance' and are not really cave data management either, e.g.
<ul>
<li><a href="exportjson.html">JSON export</a> - how to extract cave and expo data in JSON format.
<li><a href="exporttgz.html">Compressed data export</a> - how to extract cave and expo data in gzip/tar format.
</ul>
but do scan
><p>but do also scan
<ul>
<li><a href="trogindex.html">Index of all troggle documents</a> - list of everything you can do with troggle.
</ul>

100
handbook/troggle/trognotes.html
View File

@@ -7,49 +7,91 @@
</head>
<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle - the beginnings of a manual</h1>
<p>Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.
<h3>Rewrite from here on...</h3>
<h1>Troggle - programmers' guide</h1>
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
<p>This troggle manual describes these:
<h2>Programming Guide</h2>
<p>If you want to roll up your sleves and fix some horrifying annoyance that is bugging the hell out of you then you have come to the right place.
<h3>you will already have tried all these..</h3>
<ul>
<li>Annual tasks: preparing for next year, finishing last year (troggle & scripts)
<li>Architectural documentation of how it all fits together & list of active scripts
<li>How to edit and maintain troggle itself. The code is public on repository <a href="../computing/repos.html">:troggle:</a>
<li><a href="../computing/manual.html">the <em>Data</em> Maintenance</a> manual.
<li><a href="trogmanual.html">Troggle - Maintenance</a> - list of maintenance tasks<br>
<li><a href="trogarch.html">Troggle - Architecture</a> - diagrams, files, structure<br>
</ul>
<p>This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it.
<p>and you will have dismissed them as too trivial for your needs. You will have skipped over all that nasty confusing Django stuff:
<ul>
<a href="trogdjango.html">Troggle and Django</a> - The Django web framework we use<br>
<a href="trogdjangup.html">Troggle: updating Django</a> - Upgrading troggle to use a later Django version<br>
</ul>
Today troggle is used for only three things:
and you will also also have merely skimmed as irrelevant or trivial:
<ul>
<li><a href="serverconfig.html">Troggle server configuration</a> - troggle running on a new machine<br>
<li><a href="scriptsother.html">Other scripts</a>
<li><a href="scriptscurrent.html">Additional Scripts</a> - more detail<br>
<li><a href="../computing/repos.html">Version control</a> - why we have repos<br>
</ul>
<p>You want to dive into the real stuff and add missing fields to the core data model:
<ul>
<li><a href="datamodel.html">Troggle - Data Model</a> - syntax-coloured list of classes, instance variables and foreign keys<br>
</ul>
<p>We admire your drive and enthusiasm. We were like you once.
<p>First, you need to get the "expoadmin" password from a nerd. Then you will be able to dive into the online
admin system where you can explore and poke the live system and
database using the <a href="/admin/">system admin control panel</a>.
This has a data structure browser for the classes and fields of the system as
live and in-use: all the foreign keys and relationship cardinalities.
(This is not the same thing as the <a href="/troggle">troggle control panel</a> which is for deeper data management: bulk uploading and export of data files.)
<p>And, you may be surprised to discover, we have given a lot of thought into how to make it easier for you
to <a href="../computing/contribute.html#programming">get involved in the programming</a> while not screwing up this multi-decade software project.
<h3>Perhaps an external add-on?</h3>
<p>Perhaps you can program something external to troggle, in JavaScript say, using troggle data?
<ul>
<li><a href="exportjson.html">JSON export</a> - how to extract cave and expo data in JSON format.
<li><a href="exporttgz.html">Compressed data export</a> - how to extract cave and expo data in gzip/tar format.
</ul>
<h3>Creating your own system development machine</h3>
<p>It's a simple sequence:
<ol>
<li>Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left
hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
<li>Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description
can do this by filling-in some online forms. (And managing all the cave suvey data to produce this.)
<li>Use the online forms and reports.
<li>Explore the public pages of the <a href="/repositories">4 git repositories</a> including <a href="/repositories/troggle/.git/tree/">all the source code</a>
<li>Use the online forms and reports together with 'survex' and 'therion' installed on your own laptop.
<li>Get the 'expoadmin' password and explore the live system online: watch the data change in real time
<li>Configure <a href="../computing/keyexchange.html">an ssh key</a> and <a href="../computing/fzconfig.html">Filezilla</a> or scp/sFTP.
<li>Configure a <a href="../computing/basiclaptop.html">minimal expo laptop</a>
<li>Configure a <a href="../computing/yourlaptop.html">basic expo laptop</a>
<li>Configure a <a href="../survey/nerd.html">troggle development machine</a> - a laptop where you have cloned the repos and installed django. Installing is the same as updating Django. See <a href="trogdjangup.html">Troggle: updating Django</a>.
</ol>
<li>Providing a secondary way of editing individual pages of the handbook and historic records pages
for very quick and urgent changes.
This is the "Edit this page" capability; see <a href="../computing/onlinesystems.html#editthispage"> for
how to use it</a> and <em>how to tidy up afterwards</em>.
</ol>
<h3>Fixing broken URL dispatch</h3>
<p>This <em>happens all the time</em>. If ever troggle is crashing on you very mysteriously, it is almost certainly due to a typo in troggle/urls.py . Odd but true.
<h3>Fixing a broken HTML template</h3>
<p>[Note that /survey_scans/ is generated by troggle and is not the same thing as /expofiles/surveyscans/ at all.]
<h3>Running databaseReset</h3>
<ul>
<li><a href="trogimport.html">Troggle - Data Import</a> - reset and import data<br>
</ul>
<h3>Running the test suite</h3>
<ul>
<li><a href="unittests.html">Troggle - Page tests and unit tests</a> <br>
</ul>
<h3>This manual is very incomplete - notes</h3>
<p>[Note that /survey_scans/ is a url published by troggle and is not the same thing as the real directory /expofiles/surveyscans/ at all.]
<p>Only a small part of troggle's original plan was fully implemented and deployed.
Many of the things it was intended to replace are still operating as a motley collection written by many different people in
several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).
Today troggle is used for only three things:
<ol>
<li>Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left
hand corner. This is particularly true of the handbook you are reading now and the historic records of past expeditions.
<li>Publishing the "guidebook descriptions" of caves. The user who is creating a new guidebook description
can do this by filling-in some online forms.
</ol>
<h3>Troggle Login</h3>
<p>Yes you can log in to the troggle control panel: <a href="http://expo.survex.com/troggle">expo.survex.com/troggle</a>.
</p>
<hr />

2
handbook/troggle/trogstatus.html
View File

@@ -21,7 +21,7 @@
In the course of these migrations several unused or partly-used django plugins were dropped as they caused migration problems (notably staticfiles) and the plug-ins pillow, django-registration, six and sqlparse were brought up to recent versions. This was all done with pip in a python venv (virtual environment) on a Windows 10 machine running ubuntu 20.04 under WSL (Windows Systems for Linux) v1.
<p>Missing troggle functions were repaired and partly-implemented pages, such as the list of all cavers and their surveyed passages, were finished and made to work. The logbook parsing acquired a cacheing system to re-load pre-parsed files. The survex file parsing was completely rebuilt to reduce the excessive memory footprint. While doing so the parser was extended to cover nearly the full range of survex syntax and modified to parse, but not store, all the survey stations locations. A great many unused classes and some partly written code ideas were deleted.
<h4>July 2020</h4>
<p>Wookey upgraded debian on the server from 9 <var>stretch</var> to 10 <var>buster</var> and we got the python3 development of troggle running as the public version (with some http:// and https:// glitches) by 23rd July. <var>Buster</var> will be in-support definitely until June 2024 so we are rather pleased to be on a "not ancient" version of the operating system at last. This concided with a last tweak at improving the full cave data file import so now it runs on the server in ~80 seconds. Which is considerably more useful than the ~5 hours it was taking earlier this year.
<p>Wookey upgraded debian on the server from 9 <var>stretch</var> to 10 <var>buster</var> and we got the python3 development of troggle running as the public version (with some http:// and https:// glitches) by 23rd July. <var>Buster</var> will be in-support definitely until June 2024 so we are rather pleased to be on a "not ancient" version of the operating system at last. This concided with a last tweak at improving the full cave data file import so now it runs on the development system in ~80 seconds. Which is considerably more useful than the ~5 hours it was taking earlier this year.
<h4>April 2021</h4>