CUCC Expedition Handbook

Troggle Architecture Speculations

From: Philip Sargent (Gmail) [mailto:philip.sargent@gmail.com] 
Sent: 19 April 2020 01:28 [original - since edited with extra refs.]
To: expo-tech@lists.wookware.org
Subject: vague thoughts about future troggle architecture

At our last virtual pub Sam confirmed that using today's tools to re-partition troggle with all the user interface in the user's browser would be utterly horrible using current tools (javascript frameworks: react, angular etc.).

These front-end frameworks get out of date in couple of years or so. So they don't give us the decade-long stability we need to match available maintenance effort. [ See Wikipedia list of javascript frameworks.] With our deep historical perspective ("cough"), we can expect this menagerie to sort itself out into a stable, standardised foundation within the next 10 years.

A web API to expose the troggle database (read-only) would allow some keen person to write a special-purpose app on a phone, e.g. an entrance-locator app, talking directly to the database. But replacing the whole user interface does not seem feasible yet. In 10 years: probably.

It did occur to me that we are missing a trick: 99+% of the database doesn't change except for survey data updates which, apart from during expo, happen only every week or so. And the database is only 10 MB so is entirely feasible to copy absolutely everything into the browser except for scanned images and photos.

So we could partition troggle so that all the user display bits run in the browser (or a progressive web app) using a python interpreter running in javascript. [yeah, expofiles would need some subset labelled as needing to be forcibly downloaded, but the rest coming only on demand.] Some django enthusiast must have done this already surely ? Ah yes, Brython.
github.com/brython-dev/brython
www.brython.info

Which is fun, but not useful. And not just because it is immature. None of this addresses our biggest problem: devising something that can be maintained by fewer, less-expert people who can only devote short snippets of time and not long-duration immersion.

Our biggest problem

We need:

something that can be maintained by fewer, less-expert people
who can only devote short snippets of time
without requiring weeks of long-duration deep immersion

Federation of independent scripts

I know Wookey has been thinking of a loose federation of independent scripts working on the same data, but the more I look at troggle and the tasks it does the less I feel that would work. At the core there is a common data model that everything must understand - and the only unambiguous way of presenting that data model is working code, e.g. see Troggle architecture and click on the image to see a bigger copy. [It is out of date - if someone can quickly generate an update that would be nice. It's on my to-do list..] Much of what wallets.py does (originally by Martin Green) is in troggle already - but better. [There is a many:many relationship between svx files and wallet directories in reality, not 1:1]

troggle now

Troggle is very nearly fully working (not with as many functions as originally envisaged admittedly) but very nearly [it is now: 8 July 2020]. The QM data display needs writing; but other than that it's in pretty good shape. [Ah, yes, we should really add "drawings" as a core concept as well as "surveyscans". That will be a bit of work.]

Need for separate data-import checking scripts

The one thing external scripts would be really useful for is syntax checking and reference checking prior to import. I have found some weird and wonderful filename paths inside the tunnel and therion drawings, and in survex *ref paths.

Non-django troggle

Another possibility is ripping django out of troggle and leaving bare python plus a SQL database [see Trog2030 proposal]. This means that programmers would need to understand more SQL but would not need to understand "django". Arguably this could mean that we could gain.

Writing our own multi-user code would not be sensible, hence the database. But we could move to a read-only system where the only writing happens on data-import. Then we could use python 'pickle()' or 'json()' read-only data structures, but we would need to create all our own indexing and cross-referencing code.

There would be more lower-level code, but the different segments of the system could be in caving-sensible modules not django-meaningful modules. And we would not have all the extra language-like constructs that django introduces e.g. X.objects.set_all(), which modern editors complain about because it is a django idiom and not a function within the python codebase. (We could retain an HTML templating engine though.)

Addendum

There is a templating engine Nunjucks which is a port to JavaScript of the Django templating system we use (via Jinja - these are the same people who do Flask). This would be an obvious thing to use if we needed to go in that direction.

We need a templating engine because so much of the troggle coorindation output is in tables of data from diffrerent sources, e.g. see all survey data for 264.

Several organisations have moved their user-interface layer to the browser using Nunjucks including the NHS digital service and Firefox.

Go on to: Troggle architecture
Return to: Troggle Design Decisions