Documenting troggle reports & fix url refs

handbook/computing/basiclaptop.html

@@ -34,7 +34,10 @@
 <p>We are actively working on increasing the number of expo activities that can be done with just a browser and no installed software.
 
 <h3>Which laptop do you need?</h3>
-<p>If you have not actively used troggle since 2018, you are probably not aware of <a href="#any">all the things</a> you can now do with just a browser. Many of these capabilities are not new, but they weren't documented and had been forgotten over the past 10+ years. Now <a href="manual.html">these capabilities are documented</a>, though writing better documentation is an unending job.
+<p>If you have not actively used troggle since 2018, you are probably not aware of <a href="#any">all the things</a> you can now do with just a 
+browser. Many of these capabilities are not new, but they weren't documented and had been forgotten over the past 10+ years. Now <a 
+href="manual.html">these capabilities are documented</a>, though writing better documentation is an unending job, and we have a <a 
+href="manual.html">data maintenance manual</a>.
 <ul>
 <li><a href="#any">Any laptop</a> with a broswer and email. 
 <li>An <a href="#basic">expo basic laptop</a>

handbook/troggle/trog2030.html

@@ -60,7 +60,8 @@ and has stopped breaking so much with every new release.
 <p>A real architectural option would be to move to <a href="https://redis.io/">Redis</a> as an in-memory database 
 instead of using MariaDB. We don't need a fully-fledged database and these new frameworks are less closely tied to 
 having a SQL database and object-request broker (ORM) than Django is. This ties in neatly which Option 2 above: 
-reduce our use of the database functions within Django to vanishing point.
+reduce our use of the database functions within Django to vanishing point. [Oops. That would be a mistake. It is the SQL database that handles 
+all the multi-user non-interference, not Django. So unless we want to roll our own real-time software synchronisation.. Better not go there.]
 <p>ASGI, which Dango supports from v3.2 too, has the interesting effect that we no longer need a webserver like Apache or nginx to buffer requests. We can use very lightweight <a href="https://www.uvicorn.org/">uvicorn</a> instead. 
 
 <h3>Option Zero</h2>
@@ -91,7 +92,7 @@ everything in in-memory dictionaries we may need to write our own asyncio python
 we are trying to make future maintenance easier, not harder. But it looks like Redis could be the solution for us.
 
 <h3>Things that could be a bit sticky 2 - front-end code</h3>
-<p>There is not yet a front-end (javascript) framework on the client, i.e. a phone app or webpage, which is stable enough for us to commit
+<p>There is not yet a front-end (javascript or <a href="https://en.wikipedia.org/wiki/WebAssembly">WebAssembly</a>) framework on the client, i.e. a phone app or webpage, which is stable enough for us to commit
 effort to (we managed to remove all the jQuery by using recent HTML5 capabilities). Flask looks interesting
 (but <a href="https://adamj.eu/tech/2019/04/03/django-versus-flask-with-single-file-applications/">maybe is only simpler when 
 starting a new project and doesn't scale to complexity</a> the way Django does, but maybe in 2025 we 

handbook/troggle/trogdjango.html

@@ -35,16 +35,22 @@ note that the suffix on a name '<var>'_set</var> actually denotes a function cal
 <li>Django release schedule: <a href="https://www.djangoproject.com/download/">supported versions timetable</a>
 <li>Significant new Django versions are released every 8 months. Deprecated features after removed after the two subsequent versions.
 </ul>
+<p>This may seem overly pedantic and heavyweight, but this is exactly what we need when we are supporting software over decades. When we use software in troggle which is not so well organised, we really feel the pain.
 
 <h4>Django plugins</h4>
 <p>We do not just use our own code and django. We also use django plugins (known as "apps") too. These can do such things as  image re-sizing, user authentication with captchas, PayPal processing or mailing list integration. Unfortunately django plugins are volunteer efforts and do not have the same defined update management process as django itself, so release schedules are rather random. They cause merry hell when upgrading troggle to use a new version of django.
 
-<p>Every extra plugin increases the "vulnerability surface" of troggle with respect to django upgrade problems so we now only install a new plugin if it is really, really necessary and we have removed as many as we could. For example, when django-staticfiles broke during one upgrade we discovered that we could use our own troggle.expopages as a workaround, so we are not planning on reinstalling staticfiles if we don't have to.
+<p>Every extra plugin increases the "vulnerability surface" of troggle with respect to django upgrade problems so we now only install a new 
+plugin if it is really, really necessary and we have removed as many as we could. For example, when django-staticfiles broke during one upgrade 
+we discovered that we weren't really using it and could work around it, so we are not planning on reinstalling staticfiles.
 
 <h4 id="4layer">Why we still use django</h4>
-<p>Well we <a href="trog2030.html">might not use django indefinitely</a>, but unlike many frameworks the necessary functions are separately replaceable. So this gives us an evolution path. We can incrementally reduce the amount of django we use, replacing it with our own simpler python that does only what we need.
+<p>Well we <a href="trog2030.html">might not use django indefinitely</a>, but unlike many frameworks the necessary functions are separately 
-<p>The separate functions within the framework form a "stack". These functions exist in all web application frameworks but in Django they are <a href="https://docs.djangoproject.com/en/dev/misc/design-philosophies/">loosely coupled</a>.
+replaceable. So this gives us an evolution path. We can incrementally reduce the amount of django we use, replacing it with our own simpler 
-<p>The stack is: 
+python that does only what we need.
+<p>The separate functions within the framework form a "stack". These functions exist in all web application frameworks but in Django they are <a 
+href="https://docs.djangoproject.com/en/dev/misc/design-philosophies/">loosely coupled</a>.
+<p id="stack"><b>The stack is:</b> 
 <ol>
 <li>SQL database: <a href="https://docs.djangoproject.com/en/dev/topics/db/">mapping</a> to/from python objects and indexing/access routines for all the entries (sqlite, MySQL, MariaDB, postgres all work). This is where the multi-user synchronisation happens too.
 <li>request/response (<a href="https://docs.djangoproject.com/en/dev/ref/request-response/">http GET/POST</a>) including <a href="https://docs.djangoproject.com/en/dev/topics/auth/default/#auth-web-requests">access control</a> and <a href="https://www.moesif.com/blog/engineering/middleware/What-Is-HTTP-Middleware/">security middleware</a>. We could never keep up with web security if we didn't use a well-supported system for this.

handbook/troggle/trogdocm.html

@@ -13,8 +13,10 @@
 <p><em>This page will describe the usual things that go wrong: the parsers that break when
 faced with user-generated data and how to fix them. Common error messages. </em>
 <p>It concentrates on showing how to find and fix import errors so that the troggle-generated webpages are complete and not full of gaps.
+
 <p><b>Intended audience:</b> Ideally someone who is not a python programmer will be able to use this page to help them 
 fix import errors by re-editing the input files: survex, tunnel, or logbook.
+
 <p>Here we will explain the useful diagnostic detailed data pages for finding mis-matched files:<br>
 <img  border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
 <a href="http://expo.survex.com/statistics">statistics</a> - survey legs and distances for each expo<br>
@@ -24,7 +26,7 @@ fix import errors by re-editing the input files: survex, tunnel, or logbook.
 <a href="/logbookentry/2018-07-19/balkon-to-honeycomb-hangmans">logbook entry</a> - who, where and what<br>
 <a href="/survey_scans/">survey_scans</a> - all wallets of scanned surveys and corresponding survex files<br>
 <a href="/survey_scans/2014%252347/">per-wallet links</a> - links to scan files and survex files from one wallet<br>
-<a href="/dwgdata/">dwgdata</a> - all the tunnel & therion drawings and the wallets and scanned images they derive from<br>
+<a href="/dwgfiles/">dwgfiles</a> - all the tunnel & therion drawings and the wallets and scanned images they derive from<br>
 <br>
 <a href="/admin/core/dataissue/l">/admin/core/dataissue/</a> - where the data import errors and warnings are recorded<br>
 <br>

handbook/troggle/trogindex.html

@@ -13,6 +13,7 @@
 <p><img  border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
 <a href="trogintro.html">Troggle - Introduction</a> - what it is<br>
 <a href="trogusers.html">Troggle - the Users</a> - Who needs to know What and When<br>
+<a href="trogreports.html">Troggle - Surveys & Caves</a> - Trips, surveys, explorers, sketches, data...<br>
 <a href="trogdocm.html">Troggle - Fixing things</a> - users' manuals for data import<br>
 <a href="trogmanual.html">Troggle - Maintenance</a> - list of maintenance tasks<br>
 <a href="trogarch.html">Troggle - Architecture</a> - diagrams, files, structure<br>

handbook/troggle/trogkill.html

@@ -58,6 +58,7 @@ nearly half of it is directly relevant to us:
 <hr />
 Return to: <a href="trogintro.html">Troggle intro</a><br />
 Return to: <a href="trognotes.html">Troggle Programming Guide</a><br />
-<hr />
+Troggle index: 
+<a href="trogindex.html">Index of all troggle documents</a><br /><hr />
 </body>
 </html>

handbook/troggle/trogreports.html

@@ -0,0 +1,94 @@
+<!DOCTYPE html>
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>Handbook Troggle Survey Reports</title>
+<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
+</head>
+<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
+<h2 id="tophead">CUCC Expedition Handbook</h2>
+<h1>Handbook Troggle - Survey Reports</h1>
+
+<p><b>Intended audience:</b> All people entering or reading cave survey data. 
+
+<p>These are consoidated reports from many data sources, which are consolidated to tell you everything about a particular cave, or a particular 
+date on expo, or on how a specific exploration trip's survey numbers have been processed into which drawn-up survey diagrams.
+<p><img  border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
+
+<h3>All about a cave</h3>
+<a href="/survexfile/161">all data one cave</a> - all the survex data for one cave and who was on each trip<br>
+<a href="/survexfile/caves/">all cave survey data</a> - more complex caves first, then a simple list<br>
+<a href="/dwgfiles/">dwgfiles</a> - all the tunnel & therion drawings, wallets and scanned images<br>
+<a href="/survey_scans/2014%252347/">per-wallet links</a> - links to scan files and survex files from one wallet<br>
+<a href="/survey_scans/">survey_scans</a> - all wallets of scanned surveys and corresponding survex files<br>
+<a href="/survexfile/">survexfiles</a> - caves with only partial cave descriptions and <em>maybe</em> survex data<br>
+<a href="/survey_scans/">survey scan wallets</a> - all the wallets - referencing the cave they refer to<br>
+
+<a href="/1623/290/290.html">cave description</a> - Cave description (not a troggle report)<br>
+<a href="/indxal.htm">cave index</a> - list of all caves (not a troggle report)<br>
+
+
+
+<h3>All about People</h3>
+<a href="/people">survey lengths</a> - who surveyed how much cave<br>
+<a href="/folk/">members and guests</a> - who came and how many expos<br>
+<a href="/personexpedition/MartinGreen/2002">per-person caving</a> - list of logbook trips and surveys for one person<br>
+<a href="/statistics">statistics</a> - survey legs and distances for each expo<br>
+<a href="/expofiles/surveyscans/2018/Becka%20Lawson.html">outstanding wallet jobs for one person</a> - <a href="/handbook/troggle/scriptscurrent.html#wallets">(external script)</a><br>
+
+<h3>All about one Expo on one year</h3>
+<a href="/expedition/2019">expo</a> - who, where, when, what<br>
+<a href="/years/2018/index.html">expo planning</a> - planning and results (not a troggle report)<br>
+<a href="/expofiles/surveyscans/2018/walletindex.html">wallet status</a> - progress of wallets for one year <a href="/handbook/troggle/scriptscurrent.html#wallets">(external script)</a><br>
+
+
+<h3>All about survex cave data files</h3>
+/survexfile/caves-1623/161/north/satansitter.svx
+<a href="/survexfile/caves-1623/161/north/satansitter.svx">satansitter.svx</a> - direct link to editable survex file<br>
+<a href="/survexfile/">survexfiles</a> - caves with only partial cave descriptions and <em>maybe</em> survex data<br>
+<a href="/survexfile/161">all data one cave</a> - all the survex data for one cave and who was on each trip<br>
+<a href="/survexfile/caves/">all cave survey data</a> - more complex caves first, then a simple list<br>
+<a href="/statistics">statistics</a> - how much surveyed each year in survex files<br>
+<a href="/expedition/2019">expo</a> - all survex surveys on a specific expo<br>
+
+<h3>All about the scanned survey drawings</h3>
+<a href="/dwgfiles/">dwgfiles</a> - all the tunnel & therion drawings, wallets and scanned images<br>
+<a href="/survey_scans/2014%252347/">per-wallet links</a> - links to scan files and survex files from one wallet<br>
+<a href="/survey_scans/">survey_scans</a> - all wallets of scanned surveys and corresponding survex files<br>
+
+<h3>All about survey wallets</h3>
+<a href="/survey_scans/">survey scan wallets</a> - all the <a href="/handbook/survey/newwallet.html">wallets</a> and the survex data blocks associated with that survey trip<br>
+<a href="/dwgfiles/">dwgfiles</a> - all the tunnel & therion drawings, wallets and scanned images<br>
+<a href="/expedition/2019">expo</a> - all wallets on a specific expo<br>
+<a href="/expofiles/surveyscans/2018/walletindex.html">wallet status</a> - progress of <a href="/handbook/survey/newwallet.html">wallets</a> for one year <a href="/handbook/troggle/scriptscurrent.html#wallets">(external script)</a><br>
+<a href="/expofiles/surveyscans/2018/Becka%20Lawson.html">outstanding wallet jobs for one person</a> - <a href="/handbook/troggle/scriptscurrent.html#wallets">(external script)</a><br>
+<a href="/expofiles/surveyscans/2018/2018%2316/walletindex.html">outstanding wallet jobs for one wallet</a> - <a href="/handbook/troggle/scriptscurrent.html#wallets">(external script)</a><br>
+
+<h3>Data about all expos</h3>
+<a href="/folk/">members and guests</a> - who came and how many expos<br>
+<a href="/statistics">statistics</a> - how much surveyed each year in survex files<br>
+
+<h3>All about specific trips</h3>
+<a href="/logbookentry/2018-07-19/balkon-to-honeycomb-hangmans">logbook entry</a> - who, where and what<br>
+<a href="/expedition/2019">expo</a> - all logged trips on a specific expo<br>
+
+<h3>All about entrances</h3>
+<a href="/eastings">GPS and cartographic locations</a> - all the entrances, but in UTM coordinates not GPS<br>
+
+<br>
+<h3>All about the parsing errors - where things have gone wrong on data import</h3>
+<a href="/dataissues">dataissues</a> - where the data import errors and warnings are recorded<br>
+
+<h3>To do lists</h3>
+<a href="/handbook/computing/todo-data.html">data maintenance to-do list</a> - where the cave data is incomplete or inconsistent<br>
+<a href="/handbook/computing/todo-data.html">software to-do list</a> - where the software is incomplete or inconsistent<br>
+
+<br>
+<hr />
+Go on to: <a href="trogintro.html">Troggle intro</a><br>
+Go on to: <a href="trogdocm.html">Troggle - fixing things when the reports are incoherent</a><br>
+
+Return to: <a href="trogindex.html">Index of all troggle documents</a><br />
+<hr />
+</body>
+</html>

handbook/troggle/trogsimpler.html

@@ -93,9 +93,16 @@ content we want
 [A reasonable proposal, but needs quantifying with all the things troggle does 
 which Rad was unaware of. This will not be a "small number" but it needs estimating. We don't need
 everything troggle does for us of course, but that doesn't mean that removing django/troggle
-will reduce the total amount of code. The input data parsers will be nearly the same size obviously.]
+will reduce the total amount of code. The input data parsers will be nearly the same size obviously.
+<br><br>
+He is actually proposing building a <a href="http://www.laputan.org/mud/">shantytown</a> 'built from common, inexpensive materials and simple tools. Shantytowns can be built using 
+relatively unskilled labor. Even though the labor force is "unskilled" in the customary sense, the construction and maintenance of this sort of 
+housing can be quite labor intensive. There is little specialization. Each housing unit is constructed and maintained primarily by its 
+inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure 
+requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth.']
 </em>
 </div>
+<br><br>
 Why do this:
 <ul>
 <li> we don't have multiple intermediate layers all of which are difficult to

handbook/troggle/trogspeculate.html

@@ -120,7 +120,8 @@ not a function within the python codebase.
 <p>There is a templating engine <a href="https://mozilla.github.io/nunjucks/">Nunjucks</a> 
 which is a port to JavaScript of the Django templating system we use 
 (via <a href="https://palletsprojects.com/p/jinja/">Jinja</a> - these are the same people who do Flask). This would be an obvious thing to use if we needed to go in that direction.
-<p>We need a templating engine because so much of the troggle coorindation output is in tables of data from diffrerent sources, e.g. see <a href=/survexfile/264">all survey data for 264</a>.
+<p>We need a templating engine because so much of the troggle coorindation output is in tables of data from diffrerent sources, 
+e.g. see <a href="/survexfile/264">all survey data for 264</a>.
 <p>Several organisations have moved their user-interface layer to the browser using
 Nunjucks including <a href="https://service-manual.nhs.uk/design-system/prototyping">
 the NHS digital service</a> and Firefox.