expoweb/handbook/troggle/trogsimpler.html

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle - a kinder simpler troggle?</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle - a kinder simpler troggle?</h1>
<p>Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.
<p>This part of the handbook is intended for people
maintaining the troggle software. 
Day to day cave recording and surveying tasks are documented 
in the expo "survey handbook"
	<h3>stroggle</h3>
	<p>At one time Martin Green attempted to reimplement troggle as "stroggle" using <a href="https://www.fullstackpython.com/flask.html">flask</a> instead of Django at 
	<a href="https://en.wikipedia.org/wiki/Gitorious">git@gitorious.org:stroggle/stroggle.git</a> (but gitorious has been deleted).</p>
	<p>A copy of this project is archived by Wookey on <a href="http://wookware.org/software/cavearchive/stroggle/">wookware.org/software/cavearchive/stroggle/</a>.
	<p>There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "<a href="https://gitorious.org/">gitorious valhalla</a>"<br />
	<a href="https://gitorious.org/stroggle/stroggle.git/">stroggle code</a></br>
	<a href="https://gitorious.org/stroggle/stroggle-gitorious-wiki.git/">stroggle-gitorious-wiki</a></br>
	but note that this domain has an expired certificate so https:// complains.
	<p>The schema for stroggle is <a href="http://wookware.org/software/cavearchive/stroggle/git/exampledata/expo/schema.json">a schema.json file</a>. See the comparable <a href="datamodel.html">troggle schema file</a> which is indeed horrendously bigger.
	
<h3>Radost's proposal</h3>


<div style="font-family: monospace; font-size: large">
<p>Radost Waszkiewicz (CUCC member 2016-2019) proposed a plan for superceding troggle</p>

<p>
Hey,<br>

on the design sesh we've looked a bit on the way data is organised in the
loser repo and how to access it via troggle.
<p>
A proposal arised that all this database shannaingans is essentially
unnecessary - we have about 200 caves, about 250 entrances, about 200
people and couple dozen expos. We don't need efficient lookups at all. We
can write something which will be 'slow' and make only things we actually
care about. 
<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[What Rad has misunderstood here is that the database is not for speed. We use it mostly so that we can manage 
'referential integrity' i.e. have all the different bits of information match up correctly. 
While the total size of the data is small, the interrelationships and complexity is quite large. 
From the justification for troggle: 
<p></em>"A less obvious but more deeply rooted problem was the lack of relational information. One table named folk.csv stored
names of all expedition members, the years in which they were present, and a link to a biography page. This was great
for displaying a table of members by expedition year, but what if you wanted to display a list of people who wrote in the
logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that
list has been recorded in the logbook, but there is no way to access it because there is no connection between the
person's name in folk.csv and the entries he wrote in the logbook". 
[<a href="/expofiles/documents/troggle/troggle2020.pdf" download>Aaron Curtis</a>]<em>
<p>And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks,
survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent]
</em>
</div>

<p>Similarly I see little gain from doing the html - python himera
template pages. These contain mainly nested for loops which could just as
well be written in e.g. python.

<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[He could indeed. But for most people producing HTML while writing in python is just unnecessarily difficult.
But it has to be said that the 
<a href="https://docs.djangoproject.com/en/2.1/ref/templates/builtins/#std:templatefilter-dictsort">django
 HTML templating mechanism</a> is sufficiently powerful that it does almost
amount to an additional language to learn.
<p>Troggle has 66 different url recognisersand there are 71 HTML django 
template files which the recognisers direct to. 
Not all page templates are currently used but still some kind of templating system would seem to be 
probably necessary.</p>
<p>The django system is sufficiently well-thought-of
that it forms the basis for the framework-independent templaing engine 
<a href="https://jinja.palletsprojects.com/en/2.11.x/">Jinja</a> - and that site has a good discussion
on whether templating is a good thing or not. There are about 
<a href="https://wiki.python.org/moin/Templating">20 different python template engines</a>.]
</em>
</div>

I'd advocate following solution:
<ul>
<li> give up on fixing things in troggle
<li> have a small number of .py scripts which generate static webpages with
content we want
</ul>
<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[A reasonable proposal, but needs quantifying with all the things troggle does 
which Rad was unaware of. This will not be a "small number" but it needs estimating. We don't need
everything troggle does for us of course, but that doesn't mean that removing django/troggle
will reduce the total amount of code. The input data parsers will be nearly the same size obviously.]
</em>
</div>
Why do this:
<ul>
<li> we don't have multiple intermediate layers all of which are difficult to
maintain
<li> anyone can run this on their machine without need for local vm's
<li> to change something there is one thing you change - script that generates
pages belonging to particular pattern. currently you need to take care of
parsers, view script, url script, and template
<li> it runs offline (just like that as pages are just blocks of literal text
+ some javascript)
<li> we never end up with no working copy during expo - run scripts once
before expedition and put results in /backup and just leave it there till
next year. Something goes wrong on the daily updated version - we have a
very easy fallback
<li> just one (or very close to it) programming language
<li> change to python3 to parse silly chars
<li> wayyyy fewer lines of code and substantially better code to comment ratio
(if this is us writing this now)
<li> compatible with all the already existing static pages
<li> we can host this on a potato hosting as we're not running anything fancy
anymore
<li> get's rid of the horrifying url rewrites that correspond to no files that
exist
</ul>
<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[This vastly underestimates the number of things that troggle does for us. 
See "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>
Troggle: a revised system for cave data management</a>"</em>.] And a VM is not required to run and debug troggle. 
Sam has produced a docker variant which he uses extensively.
<p>Troggle today has 6,400 non-comment lines of python and 2,500 non-comment lines of django HTML template code. Plus there is the integration with the in-browser HTML editor in JavaScript. Half of the python is in the parsers which will not change whatever we do. Django itself is much, much bigger and includes all the security middleware necessary in the web today.
<p>But maintaining the code with the regular Django updates is <a href="trogdjango.html">a heavy job</a>.]
</div>
How much work would this actually take:
<ul>
<li> most likely one script per page type, so far page types that are
obviously needed:
<ul>
<li>cave index
<li>individual cave descriptions
<li>logbooks
</ul>
<li> more than half of the parsers are already rewriten by me and can be
changed to do this instead of modifying SQL database with minimal effort
<li> html/css side of things already exists, but it would be nice to go for a
more modern look there as well
</ul>
<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[The effort estimate is similarly a gross underestimate because (a)  he assumes one script per page of output, forgetting all the core work to create a central consistent dataset, and (b) he is missing out most 
of the functionality we use without realizing it because it is built into 
django's SQL system, such as multi-user operations.
<p><span style="color:red">Eventually we will have to <a href="trogdesign.html">migrate from django</a> of course</span>, as it will eventually
fail to keep up with the rest of the world. Right now we need to get ourselves onto python3
so that we can use an LTS release which has current security updates. This is 
<a href="https://docs.djangoproject.com/en/3.0/internals/release-process/#supported-versions-policy">more urgent for django</a> than for Linux. In Ubuntu terms we are on 18.04 LTS (Debian 10) which has no free maintenance updates from 2023. <span style="color:red">We should plan to migrate troggle from django to another framework in about 2025. See stroggle below.</span>]
</em>
</div>
Things this [Rad's] solution doesn't solve:
<ul>
<li> no webpage-based 'wizzard' for creating caves and such (did people use it
much? do we care?) -> maybe 'send an email to this address' is the ultimate
solution to this
<li> uploading photos is still difficult (in the sense that they don't get
minified automatically)
</ul>
Rad
<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>
[Creating a cave description for a new cave, and especially linking in images,
is currently so difficult that only a couple of people can do it. Fixing this is
a matter of urgency. No one should have to imagine where the path to a file will be but isn't now.
We need a file uploading system to put things in the right place; and this would help photos too.]
</em>
</div>

</div>


<hr />
Return to: <a href="trogdesign.html">Troggle design and future implementations</a><br />
Return to: <a href="trogintro.html">Troggle intro</a><br />
Troggle index: 
<a href="trogindex.html">Index of all troggle documents</a><br />

<hr />
</body>
</html>
new troggle discussions 2020-05-12 15:53:37 +01:00			`<!DOCTYPE html>`
			`<html>`
			`<head>`
			`<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />`
			`<title>Handbook Troggle - a kinder simpler troggle?</title>`
			`<link rel="stylesheet" type="text/css" href="../../css/main2.css" />`
			`</head>`
			`<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>`
			`<h2 id="tophead">CUCC Expedition Handbook</h2>`
			`<h1>Troggle - a kinder simpler troggle?</h1>`
			`<p>Troggle runs much of the the cave survey data management, presents the data on the website and manages the Expo Handbook.`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<p>This part of the handbook is intended for people`
			`maintaining the troggle software.`
			`Day to day cave recording and surveying tasks are documented`
			`in the expo "survey handbook"`
new troggle discussions 2020-05-12 15:53:37 +01:00			`<h3>stroggle</h3>`
			`<p>At one time Martin Green attempted to reimplement troggle as "stroggle" using <a href="https://www.fullstackpython.com/flask.html">flask</a> instead of Django at`
			`<a href="https://en.wikipedia.org/wiki/Gitorious">git@gitorious.org:stroggle/stroggle.git</a> (but gitorious has been deleted).</p>`
			`<p>A copy of this project is archived by Wookey on <a href="http://wookware.org/software/cavearchive/stroggle/">wookware.org/software/cavearchive/stroggle/</a>.`
			`<p>There is also a copy of stroggle on the backed-up, read-only copy of gitorious on "<a href="https://gitorious.org/">gitorious valhalla</a>"<br />`
			`<a href="https://gitorious.org/stroggle/stroggle.git/">stroggle code</a></br>`
			`<a href="https://gitorious.org/stroggle/stroggle-gitorious-wiki.git/">stroggle-gitorious-wiki</a></br>`
			`but note that this domain has an expired certificate so https:// complains.`
			`<p>The schema for stroggle is <a href="http://wookware.org/software/cavearchive/stroggle/git/exampledata/expo/schema.json">a schema.json file</a>. See the comparable <a href="datamodel.html">troggle schema file</a> which is indeed horrendously bigger.`

			`<h3>Radost's proposal</h3>`

More emphasis and explanation and links 2020-05-12 19:59:10 +01:00
new troggle discussions 2020-05-12 15:53:37 +01:00
			`<div style="font-family: monospace; font-size: large">`
			`<p>Radost Waszkiewicz (CUCC member 2016-2019) proposed a plan for superceding troggle</p>`

			`<p>`
			`Hey,<br>`

			`on the design sesh we've looked a bit on the way data is organised in the`
			`loser repo and how to access it via troggle.`
			`<p>`
			`A proposal arised that all this database shannaingans is essentially`
			`unnecessary - we have about 200 caves, about 250 entrances, about 200`
			`people and couple dozen expos. We don't need efficient lookups at all. We`
			`can write something which will be 'slow' and make only things we actually`
			`care about.`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>`
			`[What Rad has misunderstood here is that the database is not for speed. We use it mostly so that we can manage`
			`'referential integrity' i.e. have all the different bits of information match up correctly.`
			`While the total size of the data is small, the interrelationships and complexity is quite large.`
			`From the justification for troggle:`
			`<p></em>"A less obvious but more deeply rooted problem was the lack of relational information. One table named folk.csv stored`
new troggle discussions 2020-05-12 15:53:37 +01:00			`names of all expedition members, the years in which they were present, and a link to a biography page. This was great`
			`for displaying a table of members by expedition year, but what if you wanted to display a list of people who wrote in the`
			`logbook about a certain cave in a certain expedition year? Theoretically, all of the necessary information to produce that`
			`list has been recorded in the logbook, but there is no way to access it because there is no connection between the`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`person's name in folk.csv and the entries he wrote in the logbook".`
Updating troggle docum. links in handbook 2020-07-31 23:20:20 +01:00			`[<a href="/expofiles/documents/troggle/troggle2020.pdf" download>Aaron Curtis</a>]<em>`
new troggle discussions 2020-05-12 15:53:37 +01:00			`<p>And for ensuring survey data does not get lost we need to coordinate people, trips, survex blocks,`
			`survex files, drawing files (several formats), QMs, wallet-progress pages, rigging guides, entrance photos, GPS tracks, kataster boundaries, scans of sketches, scans of underground notes, and dates for all those - Philip Sargent]`
			`</em>`
			`</div>`

			`<p>Similarly I see little gain from doing the html - python himera`
			`template pages. These contain mainly nested for loops which could just as`
			`well be written in e.g. python.`

More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>`
			`[He could indeed. But for most people producing HTML while writing in python is just unnecessarily difficult.`
			`But it has to be said that the`
			`<a href="https://docs.djangoproject.com/en/2.1/ref/templates/builtins/#std:templatefilter-dictsort">django`
			`HTML templating mechanism</a> is sufficiently powerful that it does almost`
			`amount to an additional language to learn.`
			`<p>Troggle has 66 different url recognisersand there are 71 HTML django`
			`template files which the recognisers direct to.`
			`Not all page templates are currently used but still some kind of templating system would seem to be`
			`probably necessary.</p>`
			`<p>The django system is sufficiently well-thought-of`
			`that it forms the basis for the framework-independent templaing engine`
			`<a href="https://jinja.palletsprojects.com/en/2.11.x/">Jinja</a> - and that site has a good discussion`
			`on whether templating is a good thing or not. There are about`
			`<a href="https://wiki.python.org/moin/Templating">20 different python template engines</a>.]`
new troggle discussions 2020-05-12 15:53:37 +01:00			`</em>`
			`</div>`

			`I'd advocate following solution:`
			`<ul>`
			`<li> give up on fixing things in troggle`
			`<li> have a small number of .py scripts which generate static webpages with`
			`content we want`
			`</ul>`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>`
new troggle discussions 2020-05-12 15:53:37 +01:00			`[A reasonable proposal, but needs quantifying with all the things troggle does`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`which Rad was unaware of. This will not be a "small number" but it needs estimating. We don't need`
new troggle discussions 2020-05-12 15:53:37 +01:00			`everything troggle does for us of course, but that doesn't mean that removing django/troggle`
			`will reduce the total amount of code. The input data parsers will be nearly the same size obviously.]`
			`</em>`
			`</div>`
			`Why do this:`
			`<ul>`
			`<li> we don't have multiple intermediate layers all of which are difficult to`
			`maintain`
			`<li> anyone can run this on their machine without need for local vm's`
			`<li> to change something there is one thing you change - script that generates`
			`pages belonging to particular pattern. currently you need to take care of`
			`parsers, view script, url script, and template`
			`<li> it runs offline (just like that as pages are just blocks of literal text`
			`+ some javascript)`
			`<li> we never end up with no working copy during expo - run scripts once`
			`before expedition and put results in /backup and just leave it there till`
			`next year. Something goes wrong on the daily updated version - we have a`
			`very easy fallback`
			`<li> just one (or very close to it) programming language`
			`<li> change to python3 to parse silly chars`
			`<li> wayyyy fewer lines of code and substantially better code to comment ratio`
			`(if this is us writing this now)`
			`<li> compatible with all the already existing static pages`
			`<li> we can host this on a potato hosting as we're not running anything fancy`
			`anymore`
			`<li> get's rid of the horrifying url rewrites that correspond to no files that`
			`exist`
			`</ul>`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>`
			`[This vastly underestimates the number of things that troggle does for us.`
Updating troggle docum. links in handbook 2020-07-31 23:20:20 +01:00			`See "<a href="/expofiles/documents/troggle/troggle2020.pdf" download>`
			`Troggle: a revised system for cave data management</a>"</em>.] And a VM is not required to run and debug troggle.`
new troggle discussions 2020-05-12 15:53:37 +01:00			`Sam has produced a docker variant which he uses extensively.`
Updating troggle docum. links in handbook 2020-07-31 23:20:20 +01:00			`<p>Troggle today has 6,400 non-comment lines of python and 2,500 non-comment lines of django HTML template code. Plus there is the integration with the in-browser HTML editor in JavaScript. Half of the python is in the parsers which will not change whatever we do. Django itself is much, much bigger and includes all the security middleware necessary in the web today.`
			`<p>But maintaining the code with the regular Django updates is <a href="trogdjango.html">a heavy job</a>.]`
new troggle discussions 2020-05-12 15:53:37 +01:00			`</div>`
			`How much work would this actually take:`
			`<ul>`
			`<li> most likely one script per page type, so far page types that are`
			`obviously needed:`
			`<ul>`
			`<li>cave index`
			`<li>individual cave descriptions`
			`<li>logbooks`
			`</ul>`
			`<li> more than half of the parsers are already rewriten by me and can be`
			`changed to do this instead of modifying SQL database with minimal effort`
			`<li> html/css side of things already exists, but it would be nice to go for a`
			`more modern look there as well`
			`</ul>`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>`
new troggle discussions 2020-05-12 15:53:37 +01:00			`[The effort estimate is similarly a gross underestimate because (a) he assumes one script per page of output, forgetting all the core work to create a central consistent dataset, and (b) he is missing out most`
			`of the functionality we use without realizing it because it is built into`
			`django's SQL system, such as multi-user operations.`
New index to all troggle pages 2020-07-27 01:42:09 +01:00			`<p><span style="color:red">Eventually we will have to <a href="trogdesign.html">migrate from django</a> of course</span>, as it will eventually`
new troggle discussions 2020-05-12 15:53:37 +01:00			`fail to keep up with the rest of the world. Right now we need to get ourselves onto python3`
			`so that we can use an LTS release which has current security updates. This is`
			`<a href="https://docs.djangoproject.com/en/3.0/internals/release-process/#supported-versions-policy">more urgent for django</a> than for Linux. In Ubuntu terms we are on 18.04 LTS (Debian 10) which has no free maintenance updates from 2023. <span style="color:red">We should plan to migrate troggle from django to another framework in about 2025. See stroggle below.</span>]`
			`</em>`
			`</div>`
			`Things this [Rad's] solution doesn't solve:`
			`<ul>`
			`<li> no webpage-based 'wizzard' for creating caves and such (did people use it`
			`much? do we care?) -> maybe 'send an email to this address' is the ultimate`
			`solution to this`
			`<li> uploading photos is still difficult (in the sense that they don't get`
			`minified automatically)`
			`</ul>`
			`Rad`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`<div style="margin-left: 2em; margin-right:2em; color: blue; font-family: sanserif"><em>`
new troggle discussions 2020-05-12 15:53:37 +01:00			`[Creating a cave description for a new cave, and especially linking in images,`
			`is currently so difficult that only a couple of people can do it. Fixing this is`
			`a matter of urgency. No one should have to imagine where the path to a file will be but isn't now.`
			`We need a file uploading system to put things in the right place; and this would help photos too.]`
			`</em>`
			`</div>`

			`</div>`



			`<hr />`
More emphasis and explanation and links 2020-05-12 19:59:10 +01:00			`Return to: <a href="trogdesign.html">Troggle design and future implementations</a><br />`
New index to all troggle pages 2020-07-27 01:42:09 +01:00			`Return to: <a href="trogintro.html">Troggle intro</a><br />`
			`Troggle index:`
			`<a href="trogindex.html">Index of all troggle documents</a><br />`

new troggle discussions 2020-05-12 15:53:37 +01:00			`<hr />`
			`</body>`
			`</html>`