diff --git a/handbook/troggle/trogdesign.html b/handbook/troggle/trogdesign.html index 1e126cdea..b5607367f 100644 --- a/handbook/troggle/trogdesign.html +++ b/handbook/troggle/trogdesign.html @@ -11,11 +11,12 @@

Open issues being worked on:

Go on to: Troggle architecture
-Go on to: Troggle design: architecture speculation
Return to: Troggle intro
diff --git a/handbook/troggle/trogmanual.html b/handbook/troggle/trogmanual.html index a03f0caea..24f8aaec7 100644 --- a/handbook/troggle/trogmanual.html +++ b/handbook/troggle/trogmanual.html @@ -18,7 +18,7 @@
  • Regular tasks: housekeeping as surveys are digitised.
  • Other scripts - photos, folk, wallets - not integral parts of troggle.
  • Troggle and database architecture: how it all fits together -
  • Design decisions: e.g. new systems for website menus +
  • Design decisions inc. troggle replacement e.g. new systems for website menus
  • Maintain troggle itself. The code is public on repository :troggle:
  • Uncategorised notes and past speculations diff --git a/handbook/troggle/trognotes.html b/handbook/troggle/trognotes.html index 17e77b721..d979d722a 100644 --- a/handbook/troggle/trognotes.html +++ b/handbook/troggle/trognotes.html @@ -352,141 +352,16 @@ Storage:
    -

    Radost's proposal

    -
    -

    Radost Waszkiewicz (CUCC member 2016-2019) proposed a plan for superceding troggle

    - -

    -Hey,
    - -on the design sesh we've looked a bit on the way data is organised in the -loser repo and how to access it via troggle. -

    -A proposal arised that all this database shannaingans is essentially -unnecessary - we have about 200 caves, about 250 entrances, about 200 -people and couple dozen expos. We don't need efficient lookups at all. We -can write something which will be 'slow' and make only things we actually -care about. -

    -[What Rad has misunderstood here is that the database is not for speed. We use it mostly so that we can manage 'referential integrity' i.e. have all the different bits of information match up correctly. While the total size of the data is small, the interrelationships and complexity is quite large. From the justification for troggle:

    "A less obvious but more deeply rooted problem was the lack of relational information. One table named folk.csv stored -names of all expedition members, the years in which they were present, and a link to a biography page. This was great -for displaying a table of members by expedition year, but what if you wanted to display a list of people who wrote in the -logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that -list has been recorded in the logbook, but there is no way to access it because there is no connection between the -person's name in folk.csv and the entries he wrote in the logbook". -

    And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks, -survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent] -

    -

    Similarly I see little gain from doing the html - python himera -template pages. These contain mainly nested for loops which could just as -well be written in e.g. python. - -

    -[He could indeed. But for most people producing HTML while writing in python is just unnecessarily difficult.] - -
    - -I'd advocate following solution: - -
    -[A reasonable proposal, but needs quantifying with all the things troggle does -which Rad was unaware of. This will not be a "small" number but it needs estimating. We don't need -everything troggle does for us of course, but that doesn't mean that removing django/troggle -will reduce the total amount of code. The input data parsers will be nearly the same size obviously.] - -
    -Why do this: - -
    -[This vastly underestimates the number of things that troggle does for us. See "Troggle: a novel system for cave exploration information management". And a VM is not required to run and debug troggle. -Sam has produced a docker variant which he uses extensively. -

    Troggle today has 8,200 lines of python (including comments and blank lines), plus 600 in imagekit and 200 in flatpages. 2,200 of those are in the parsers. Django itself has a lot more, including integration with TinyMCE in-browser HTML editor. ] -

    -How much work would this actually take: - -
    -[The effort estimate is similarly a gross underestimate because (a) he assumes one script per page of output, forgetting all the core work to create a central consistent dataset, and (b) he is missing out most -of the functionality we use without realizing it because it is built into -django's SQL system, such as multi-user operations. -

    Eventually we will have to migrate from django of course, as it will eventually -fail to keep up with the rest of the world. Right now we need to get ourselves onto python3 -so that we can use an LTS release which has current security updates. This is -more urgent for django than for Linux. In Ubuntu terms we are on 18.04 LTS (Debian 10) which has no free maintenance updates from 2023. We should plan to migrate troggle from django to another framework in about 2025. See stroggle below.] - -

    -Things this [Rad's] solution doesn't solve: - -Rad -
    -[Creating a cave description for a new cave, and especially linking in images, -is currently so difficult that only a couple of people can do it. Fixing this is -a matter of urgency. No one should have to imagine where the path to a file will be but isn't now. -We need a file uploading system to put things in the right place; and this would help photos too.] - -
    - -
    -

    stroggle

    -

    At one time Martin Green attempted to reimplement troggle as "stroggle" using flask instead of Django at -git@gitorious.org:stroggle/stroggle.git (but gitorious has been deleted).

    -

    A copy of this project is archived by Wookey on wookware.org/software/cavearchive/stroggle/. -

    There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "gitorious valhalla"
    -stroggle code
    -stroggle-gitorious-wiki
    -but note that this domain has an expired certificate so https:// complains. -

    The schema for stroggle is a schema.json file. See the comparable troggle schema file which is indeed horrendously bigger. -

    Archived updates

    Since 2008 we have been keeping detailed records of all data management system updates in the version control system. Before then we manually maintained a list of updates which are now only of historical interest.

    A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article Taking Expo Bullshit into the 21st Century is archived here. - - +

    + Go on to: Troggle architecture
    + Return to: Troggle design notes
    +
    + diff --git a/handbook/troggle/trogsimpler.html b/handbook/troggle/trogsimpler.html new file mode 100644 index 000000000..9a2195f25 --- /dev/null +++ b/handbook/troggle/trogsimpler.html @@ -0,0 +1,157 @@ + + + + +Handbook Troggle - a kinder simpler troggle? + + + +

    CUCC Expedition Handbook

    +

    Troggle - a kinder simpler troggle?

    +

    Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook. + +

    stroggle

    +

    At one time Martin Green attempted to reimplement troggle as "stroggle" using flask instead of Django at + git@gitorious.org:stroggle/stroggle.git (but gitorious has been deleted).

    +

    A copy of this project is archived by Wookey on wookware.org/software/cavearchive/stroggle/. +

    There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "gitorious valhalla"
    + stroggle code
    + stroggle-gitorious-wiki
    + but note that this domain has an expired certificate so https:// complains. +

    The schema for stroggle is a schema.json file. See the comparable troggle schema file which is indeed horrendously bigger. + +

    Radost's proposal

    + +

    This part of the handbook is intended for people + maintaining the troggle software. + Day to day cave recording and surveying tasks are documented + in the expo "survey handbook" + +

    +

    Radost Waszkiewicz (CUCC member 2016-2019) proposed a plan for superceding troggle

    + +

    +Hey,
    + +on the design sesh we've looked a bit on the way data is organised in the +loser repo and how to access it via troggle. +

    +A proposal arised that all this database shannaingans is essentially +unnecessary - we have about 200 caves, about 250 entrances, about 200 +people and couple dozen expos. We don't need efficient lookups at all. We +can write something which will be 'slow' and make only things we actually +care about. +

    +[What Rad has misunderstood here is that the database is not for speed. We use it mostly so that we can manage 'referential integrity' i.e. have all the different bits of information match up correctly. While the total size of the data is small, the interrelationships and complexity is quite large. From the justification for troggle:

    "A less obvious but more deeply rooted problem was the lack of relational information. One table named folk.csv stored +names of all expedition members, the years in which they were present, and a link to a biography page. This was great +for displaying a table of members by expedition year, but what if you wanted to display a list of people who wrote in the +logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that +list has been recorded in the logbook, but there is no way to access it because there is no connection between the +person's name in folk.csv and the entries he wrote in the logbook". +

    And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks, +survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent] + +

    + +

    Similarly I see little gain from doing the html - python himera +template pages. These contain mainly nested for loops which could just as +well be written in e.g. python. + +

    +[He could indeed. But for most people producing HTML while writing in python is just unnecessarily difficult.] + +
    + +I'd advocate following solution: + +
    +[A reasonable proposal, but needs quantifying with all the things troggle does +which Rad was unaware of. This will not be a "small" number but it needs estimating. We don't need +everything troggle does for us of course, but that doesn't mean that removing django/troggle +will reduce the total amount of code. The input data parsers will be nearly the same size obviously.] + +
    +Why do this: + +
    +[This vastly underestimates the number of things that troggle does for us. See "Troggle: a novel system for cave exploration information management". And a VM is not required to run and debug troggle. +Sam has produced a docker variant which he uses extensively. +

    Troggle today has 8,200 lines of python (including comments and blank lines), plus 600 in imagekit and 200 in flatpages. 2,200 of those are in the parsers. Django itself has a lot more, including integration with TinyMCE in-browser HTML editor. ] +

    +How much work would this actually take: + +
    +[The effort estimate is similarly a gross underestimate because (a) he assumes one script per page of output, forgetting all the core work to create a central consistent dataset, and (b) he is missing out most +of the functionality we use without realizing it because it is built into +django's SQL system, such as multi-user operations. +

    Eventually we will have to migrate from django of course, as it will eventually +fail to keep up with the rest of the world. Right now we need to get ourselves onto python3 +so that we can use an LTS release which has current security updates. This is +more urgent for django than for Linux. In Ubuntu terms we are on 18.04 LTS (Debian 10) which has no free maintenance updates from 2023. We should plan to migrate troggle from django to another framework in about 2025. See stroggle below.] + +

    +Things this [Rad's] solution doesn't solve: + +Rad +
    +[Creating a cave description for a new cave, and especially linking in images, +is currently so difficult that only a couple of people can do it. Fixing this is +a matter of urgency. No one should have to imagine where the path to a file will be but isn't now. +We need a file uploading system to put things in the right place; and this would help photos too.] + +
    + +
    + + + +
    +Go on to: Troggle architecture
    +Return to: Troggle design notes
    +
    + +