From e3fc406f8023c3fa6cd261ca6e65db38ca906ce3 Mon Sep 17 00:00:00 2001 From: Philip Sargent Date: Sat, 19 Feb 2022 21:00:28 +0000 Subject: [PATCH] Documenting troggle reports & fix url refs --- handbook/computing/basiclaptop.html | 5 +- handbook/troggle/trog2030.html | 5 +- handbook/troggle/trogdjango.html | 14 +++-- handbook/troggle/trogdocm.html | 4 +- handbook/troggle/trogindex.html | 1 + handbook/troggle/trogkill.html | 3 +- handbook/troggle/trogreports.html | 94 +++++++++++++++++++++++++++++ handbook/troggle/trogsimpler.html | 9 ++- handbook/troggle/trogspeculate.html | 3 +- 9 files changed, 127 insertions(+), 11 deletions(-) create mode 100644 handbook/troggle/trogreports.html diff --git a/handbook/computing/basiclaptop.html b/handbook/computing/basiclaptop.html index ec93053c1..c709ca4b5 100644 --- a/handbook/computing/basiclaptop.html +++ b/handbook/computing/basiclaptop.html @@ -34,7 +34,10 @@

We are actively working on increasing the number of expo activities that can be done with just a browser and no installed software.

Which laptop do you need?

-

If you have not actively used troggle since 2018, you are probably not aware of all the things you can now do with just a browser. Many of these capabilities are not new, but they weren't documented and had been forgotten over the past 10+ years. Now these capabilities are documented, though writing better documentation is an unending job. +

If you have not actively used troggle since 2018, you are probably not aware of all the things you can now do with just a +browser. Many of these capabilities are not new, but they weren't documented and had been forgotten over the past 10+ years. Now these capabilities are documented, though writing better documentation is an unending job, and we have a data maintenance manual.

+

This may seem overly pedantic and heavyweight, but this is exactly what we need when we are supporting software over decades. When we use software in troggle which is not so well organised, we really feel the pain.

Django plugins

We do not just use our own code and django. We also use django plugins (known as "apps") too. These can do such things as image re-sizing, user authentication with captchas, PayPal processing or mailing list integration. Unfortunately django plugins are volunteer efforts and do not have the same defined update management process as django itself, so release schedules are rather random. They cause merry hell when upgrading troggle to use a new version of django. -

Every extra plugin increases the "vulnerability surface" of troggle with respect to django upgrade problems so we now only install a new plugin if it is really, really necessary and we have removed as many as we could. For example, when django-staticfiles broke during one upgrade we discovered that we could use our own troggle.expopages as a workaround, so we are not planning on reinstalling staticfiles if we don't have to. +

Every extra plugin increases the "vulnerability surface" of troggle with respect to django upgrade problems so we now only install a new +plugin if it is really, really necessary and we have removed as many as we could. For example, when django-staticfiles broke during one upgrade +we discovered that we weren't really using it and could work around it, so we are not planning on reinstalling staticfiles.

Why we still use django

-

Well we might not use django indefinitely, but unlike many frameworks the necessary functions are separately replaceable. So this gives us an evolution path. We can incrementally reduce the amount of django we use, replacing it with our own simpler python that does only what we need. -

The separate functions within the framework form a "stack". These functions exist in all web application frameworks but in Django they are loosely coupled. -

The stack is: +

Well we might not use django indefinitely, but unlike many frameworks the necessary functions are separately +replaceable. So this gives us an evolution path. We can incrementally reduce the amount of django we use, replacing it with our own simpler +python that does only what we need. +

The separate functions within the framework form a "stack". These functions exist in all web application frameworks but in Django they are loosely coupled. +

The stack is:

  1. SQL database: mapping to/from python objects and indexing/access routines for all the entries (sqlite, MySQL, MariaDB, postgres all work). This is where the multi-user synchronisation happens too.
  2. request/response (http GET/POST) including access control and security middleware. We could never keep up with web security if we didn't use a well-supported system for this. diff --git a/handbook/troggle/trogdocm.html b/handbook/troggle/trogdocm.html index d7af01cd3..8e061ddc7 100644 --- a/handbook/troggle/trogdocm.html +++ b/handbook/troggle/trogdocm.html @@ -13,8 +13,10 @@

    This page will describe the usual things that go wrong: the parsers that break when faced with user-generated data and how to fix them. Common error messages.

    It concentrates on showing how to find and fix import errors so that the troggle-generated webpages are complete and not full of gaps. +

    Intended audience: Ideally someone who is not a python programmer will be able to use this page to help them fix import errors by re-editing the input files: survex, tunnel, or logbook. +

    Here we will explain the useful diagnostic detailed data pages for finding mis-matched files:
    git logo statistics - survey legs and distances for each expo
    @@ -24,7 +26,7 @@ fix import errors by re-editing the input files: survex, tunnel, or logbook. logbook entry - who, where and what
    survey_scans - all wallets of scanned surveys and corresponding survex files
    per-wallet links - links to scan files and survex files from one wallet
    -dwgdata - all the tunnel & therion drawings and the wallets and scanned images they derive from
    +dwgfiles - all the tunnel & therion drawings and the wallets and scanned images they derive from

    /admin/core/dataissue/ - where the data import errors and warnings are recorded

    diff --git a/handbook/troggle/trogindex.html b/handbook/troggle/trogindex.html index 9b0e58b95..f38a958b0 100644 --- a/handbook/troggle/trogindex.html +++ b/handbook/troggle/trogindex.html @@ -13,6 +13,7 @@

    git logo Troggle - Introduction - what it is
    Troggle - the Users - Who needs to know What and When
    +Troggle - Surveys & Caves - Trips, surveys, explorers, sketches, data...
    Troggle - Fixing things - users' manuals for data import
    Troggle - Maintenance - list of maintenance tasks
    Troggle - Architecture - diagrams, files, structure
    diff --git a/handbook/troggle/trogkill.html b/handbook/troggle/trogkill.html index 7e359171a..a11d9913e 100644 --- a/handbook/troggle/trogkill.html +++ b/handbook/troggle/trogkill.html @@ -58,6 +58,7 @@ nearly half of it is directly relevant to us:

    Return to: Troggle intro
    Return to: Troggle Programming Guide
    -
    +Troggle index: +Index of all troggle documents
    diff --git a/handbook/troggle/trogreports.html b/handbook/troggle/trogreports.html new file mode 100644 index 000000000..9ea8c1f8e --- /dev/null +++ b/handbook/troggle/trogreports.html @@ -0,0 +1,94 @@ + + + + +Handbook Troggle Survey Reports + + + +

    CUCC Expedition Handbook

    +

    Handbook Troggle - Survey Reports

    + +

    Intended audience: All people entering or reading cave survey data. + +

    These are consoidated reports from many data sources, which are consolidated to tell you everything about a particular cave, or a particular +date on expo, or on how a specific exploration trip's survey numbers have been processed into which drawn-up survey diagrams. +

    git logo + +

    All about a cave

    +all data one cave - all the survex data for one cave and who was on each trip
    +all cave survey data - more complex caves first, then a simple list
    +dwgfiles - all the tunnel & therion drawings, wallets and scanned images
    +per-wallet links - links to scan files and survex files from one wallet
    +survey_scans - all wallets of scanned surveys and corresponding survex files
    +survexfiles - caves with only partial cave descriptions and maybe survex data
    +survey scan wallets - all the wallets - referencing the cave they refer to
    + +cave description - Cave description (not a troggle report)
    +cave index - list of all caves (not a troggle report)
    + + + +

    All about People

    +survey lengths - who surveyed how much cave
    +members and guests - who came and how many expos
    +per-person caving - list of logbook trips and surveys for one person
    +statistics - survey legs and distances for each expo
    +outstanding wallet jobs for one person - (external script)
    + +

    All about one Expo on one year

    +expo - who, where, when, what
    +expo planning - planning and results (not a troggle report)
    +wallet status - progress of wallets for one year (external script)
    + + +

    All about survex cave data files

    +/survexfile/caves-1623/161/north/satansitter.svx +satansitter.svx - direct link to editable survex file
    +survexfiles - caves with only partial cave descriptions and maybe survex data
    +all data one cave - all the survex data for one cave and who was on each trip
    +all cave survey data - more complex caves first, then a simple list
    +statistics - how much surveyed each year in survex files
    +expo - all survex surveys on a specific expo
    + +

    All about the scanned survey drawings

    +dwgfiles - all the tunnel & therion drawings, wallets and scanned images
    +per-wallet links - links to scan files and survex files from one wallet
    +survey_scans - all wallets of scanned surveys and corresponding survex files
    + +

    All about survey wallets

    +survey scan wallets - all the wallets and the survex data blocks associated with that survey trip
    +dwgfiles - all the tunnel & therion drawings, wallets and scanned images
    +expo - all wallets on a specific expo
    +wallet status - progress of wallets for one year (external script)
    +outstanding wallet jobs for one person - (external script)
    +outstanding wallet jobs for one wallet - (external script)
    + +

    Data about all expos

    +members and guests - who came and how many expos
    +statistics - how much surveyed each year in survex files
    + +

    All about specific trips

    +logbook entry - who, where and what
    +expo - all logged trips on a specific expo
    + +

    All about entrances

    +GPS and cartographic locations - all the entrances, but in UTM coordinates not GPS
    + +
    +

    All about the parsing errors - where things have gone wrong on data import

    +dataissues - where the data import errors and warnings are recorded
    + +

    To do lists

    +data maintenance to-do list - where the cave data is incomplete or inconsistent
    +software to-do list - where the software is incomplete or inconsistent
    + +
    +
    +Go on to: Troggle intro
    +Go on to: Troggle - fixing things when the reports are incoherent
    + +Return to: Index of all troggle documents
    +
    + + diff --git a/handbook/troggle/trogsimpler.html b/handbook/troggle/trogsimpler.html index 48d2fd764..fffd78920 100644 --- a/handbook/troggle/trogsimpler.html +++ b/handbook/troggle/trogsimpler.html @@ -93,9 +93,16 @@ content we want [A reasonable proposal, but needs quantifying with all the things troggle does which Rad was unaware of. This will not be a "small number" but it needs estimating. We don't need everything troggle does for us of course, but that doesn't mean that removing django/troggle -will reduce the total amount of code. The input data parsers will be nearly the same size obviously.] +will reduce the total amount of code. The input data parsers will be nearly the same size obviously. +

    +He is actually proposing building a shantytown 'built from common, inexpensive materials and simple tools. Shantytowns can be built using +relatively unskilled labor. Even though the labor force is "unskilled" in the customary sense, the construction and maintenance of this sort of +housing can be quite labor intensive. There is little specialization. Each housing unit is constructed and maintained primarily by its +inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure +requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth.'] +

    Why do this:
    • we don't have multiple intermediate layers all of which are difficult to diff --git a/handbook/troggle/trogspeculate.html b/handbook/troggle/trogspeculate.html index f81d78976..3476d97af 100644 --- a/handbook/troggle/trogspeculate.html +++ b/handbook/troggle/trogspeculate.html @@ -120,7 +120,8 @@ not a function within the python codebase.

      There is a templating engine Nunjucks which is a port to JavaScript of the Django templating system we use (via Jinja - these are the same people who do Flask). This would be an obvious thing to use if we needed to go in that direction. -

      We need a templating engine because so much of the troggle coorindation output is in tables of data from diffrerent sources, e.g. see all survey data for 264. +

      We need a templating engine because so much of the troggle coorindation output is in tables of data from diffrerent sources, +e.g. see all survey data for 264.

      Several organisations have moved their user-interface layer to the browser using Nunjucks including the NHS digital service and Firefox.