expoweb/handbook/troggle/trognotes.html

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle NOTES</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle - programmers' guide</h1>
<img  border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>

<h2>Programming Guide</h2>
<p>If you want to roll up your sleves and fix some horrifying annoyance that is bugging the hell out of you then you have come to the right place.
  
<h3>Issue Tracking</h3>
<p>In the firm belief that agreement on the problem must precede implementation of a solution, the Troggle git repository has been mirrored by Charlie Crossley. You may find it on their server at <a href="https://git.xle.sh/expo/troggle">git.xle.sh/expo/troggle</a>. This is an instance of Gitea, and includes an <a href="https://git.xle.sh/expo/troggle/issues">issue tracker</a> (a la GitHub but minus the Microsoft). Please browse issues and feel free to submit new ones for discussion as well as to add comments to ones already opened.</p>
  
<h3>you will already have tried all these..</h3>
<ul>
<li><a href="../computing/manual.html">the <em>Data</em> Maintenance</a> manual.
<li><a href="trogmanual.html">Troggle - Maintenance</a> - list of maintenance tasks<br>
<li><a href="trogarch.html">Troggle - Architecture</a> - diagrams, files, structure<br>
</ul>

<p>and you will have dismissed them as too trivial for your needs. You will have skipped over all that nasty confusing Django stuff:
<ul>
<a href="trogdjango.html">Troggle and Django</a> - The Django web framework we use<br>
<a href="trogdjangup.html">Troggle: updating Django</a> - Upgrading troggle to use a later Django version<br>
</ul>

and you will also also have merely skimmed as irrelevant or trivial:
<ul>
<li><a href="serverconfig.html">Troggle server configuration</a> - troggle running on a new server<br>
<li><a href="scriptsother.html">Other scripts</a>
<li><a href="scriptscurrent.html">Additional Scripts</a> - more detail<br>
<li><a href="../computing/repos.html">Version control</a> - why we have repos<br>
  <li> <a href="/handbook/troggle/troglaptop.html">Troggle development machine</a> - your laptop
</ul>

<p>You want to dive into the real stuff and add missing fields to the core data model:
<ul>
<li><a href="datamodel.html">Troggle - Data Model</a> - syntax-coloured list of classes, instance variables and foreign keys<br>
</ul>
<p>We admire your drive and enthusiasm. We were like you once.

<p>First, you need to get the "expoadmin" password from a nerd. Then you will be able to dive into the online 
admin system where you can explore and poke the live system and 
database using the <a href="/admin/">system admin control panel</a>. 
This has a data structure browser for the classes and fields of the system as 
live and in-use: all the foreign keys and relationship cardinalities. 
(This is not the same thing as the <a href="/troggle">troggle control panel</a> which is for deeper data management: bulk uploading and export of data files.)

<p>And, you may be surprised to discover, we have given a lot of thought into how to make it easier for you 
to <a href="../computing/contribute.html#programming">get involved in the programming</a> while not screwing up this multi-decade software project.

<h3>Kill it with Fire</h3>
<a href="trogkill.html">
<img  border="1" class="onright" width="150px" src='../i/dumpsterfire.jpg' alt='dumpster fire'/></a>
<p>Everyone at some point feels like giving up and wonders if we should just rewrite it all from scratch using 'modern' techniques.
Several of us have given this a lot of thought, see <a href="trogsimpler.html">'A kinder, simpler troggle'</a>, 
the <a href="trog2030.html">2030 plan</a>, <a href="trogspeculate.html">architecture constraints</a> and most importantly, <a href="trogdjango.html#4layer">the 4-layer stack</a> <em>any</em> future technology with some public URLs would need to emulate.
<p>We have also been learning from the global software community when you really do need to 'Kill it with Fire': what are 
the characteristics of code and of organisations that mean that it is a good idea to give up, and if so how to
manage the rewrite. The answers are "almost never" and "don't": as explained in <a href="trogkill.html">this summary of the book 'Kill it with 
Fire'</a>.

<p>Replacing old systems, and changing their architecture, while keeping them operating continuously, is now 
<a href="https://martinfowler.com/articles/patterns-legacy-displacement/transitional-architecture.html">a well-understood 
software engineering process</a> comprising a dozen separate techniques. With troggle, we can probably fairly easily replace the cross-referencing 
between different types of things (e.g. logbook entries, survex files, persons, expedition days)  but we will find it tricky to replace the 
sychronisation and concurrency that SQL databases give us 'for free'. We are waiting to see if Django comes up with something cunning as it steadily 
converts to a more asynchronous architecture using python <a href="https://docs.djangoproject.com/en/4.0/topics/async/">asyncio</a>.

<h3>Perhaps an external add-on?</h3>
<p>Perhaps you can program something external to troggle, in JavaScript say, using troggle data?
<ul>
<li><a href="exportjson.html">JSON export</a> - how to extract cave and expo data in JSON format.
<li><a href="exporttgz.html">Compressed data export</a> - how to extract cave and expo data in gzip/tar format.
</ul>

<h3 id="devtree">Creating <a href="/handbook/troggle/troglaptop.html">your own system development machine</a></h3>
<p>It's a simple sequence which steadily builds your expertise:
<ol>
<li>Use the online forms and reports. 
<li>Read documentation to get the <a href="trogarch.html">concepts and structures</a> into your head, so that you can see how it all fits together and works.
<li>Explore the public pages of the <a href="/repositories">4 git repositories</a> including <a href="/repositories/troggle/.git/tree/">all the source code</a>
<li>Use the online forms and reports together with 'survex' and 'therion' installed on your own laptop.
<li>Get the 'expoadmin' password and explore the live system online: watch the data change in real time
<li>Configure <a href="../computing/keyexchange.html">an ssh key</a> and <a href="../computing/fzconfig.html">Filezilla</a> or scp/sFTP.
<li>Configure  a <a href="../computing/basiclaptop.html">basic expo laptop</a>
<li>Configure <a href="../computing/surveylaptop.html">cave survey software on your laptop</a>
<li>Configure a <a href="troglaptop.html">troggle development machine</a> - a laptop where you have cloned the repos and installed django. Installing is the same as updating Django. See <a href="trogdjangup.html">Troggle: updating Django</a>.
</ol>

<h3>Fixing broken URL dispatch</h3>
<p>This <em>happens all the time</em>. If ever troggle is crashing on you very mysteriously, it is almost certainly due to a typo in <var>troggle/urls.py</var> . Odd but true.

<h3>Fixing a broken HTML template</h3>
[to be written]

<h3>Running databaseReset</h3>
<ul>
<li><a href="trogimport.html">Troggle - Data Import</a> - reset and import data<br>
</ul>
<h3>Running the test suite</h3>
<ul>
<li><a href="trogtests.html">Troggle - Page tests and smoke tests</a> <br>
</ul>

<h3>This manual is very incomplete - notes</h3>
<p>[Note that /survey_scans/ is a url published by troggle and is not the same thing as the real directory /expofiles/surveyscans/ at all.]
<p>Only a small part of troggle's original plan was fully implemented and deployed. 
Many of the things it was intended to replace are still operating as a motley collection written by many different people in
several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).

<h3>Writing python</h3>
<p>We aspire to a coding standard, and the <a href="https://github.com/google/styleguide/blob/gh-pages/pyguide.md">Google python coding standard</a> is a pretty good one. We don't use their TODO system though (para <a href="https://github.com/google/styleguide/blob/gh-pages/pyguide.md#312-todo-comments">3.12</a>) and the function (re)naming in troggle is still work in progress (para. 3.16.4).

<p>We don't have any type hints in troggle. It is not a big enough project to be worth it - probably.

<p>We have run much of the code through <a href="https://pypi.org/project/black/">black</a> in 2022 so that is our default formatting standard.

<p>We have also used ruff, isort and chill to trim and deduplicate the various pip installation packages and to fix the order of python packages in import statements.

<hr />
<h3><a id="arch">Archived updates</a></h3>
<p>Since 2008 we have been keeping detailed records of all data management system updates in the version control system. 
Before then we manually maintained <a href="../computing/update.html">a list of updates</a> which are now only of historical interest.
<p>A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article <a href="../c21bs.html">Taking Expo Bullshit into the 21st Century</a> is archived here.
	<hr />
	Go on to: <a href="trogarch.html">Troggle architecture</a><br />

    Return to: <a href="trogintro.html">Troggle intro</a><br />
Troggle index: 
<a href="trogindex.html">Index of all troggle documents</a><br />

	<hr /></body>
</html>
fix and moving a few pages into the handbook snf fixing internal links. Soft links probably not working when created with Windows. Will fix later. 2019-12-23 16:29:50 +00:00			`<!DOCTYPE html>`
Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00			`<html>`
			`<head>`
fix and moving a few pages into the handbook snf fixing internal links. Soft links probably not working when created with Windows. Will fix later. 2019-12-23 16:29:50 +00:00			`<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />`
Restructuring troggle notes and creation of troggle documentation directory in the handbook 2020-04-02 14:25:58 +01:00			`<title>Handbook Troggle NOTES</title>`
moved files to handbook, fixing out-links 2019-12-30 01:23:56 +00:00			`<link rel="stylesheet" type="text/css" href="../../css/main2.css" />`
Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00			`</head>`
add links to trog laptop page - online edit of handbook/troggle/trognotes.html 2023-04-26 20:15:08 +01:00			`<body>`
			`<style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>`
Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00			`<h2 id="tophead">CUCC Expedition Handbook</h2>`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<h1>Troggle - programmers' guide</h1>`
Tidying and extending troggle documentation 2020-07-10 02:51:53 +01:00			`<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00
			`<h2>Programming Guide</h2>`
			`<p>If you want to roll up your sleves and fix some horrifying annoyance that is bugging the hell out of you then you have come to the right place.`
Added note about issue tracking - online edit of handbook/troggle/trognotes.html 2023-08-02 15:57:23 +01:00
			`<h3>Issue Tracking</h3>`
			<p>In the firm belief that agreement on the problem must precede implementation of a solution, the Troggle git repository has been mirrored by Charlie Crossley. You may find it on their server at <a href="https://git.xle.sh/expo/troggle">git.xle.sh/expo/troggle</a>. This is an instance of Gitea, and includes an <a href="https://git.xle.sh/expo/troggle/issues">issue tracker</a> (a la GitHub but minus the Microsoft). Please browse issues and feel free to submit new ones for discussion as well as to add comments to ones already opened.</p>

Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<h3>you will already have tried all these..</h3>`
More links to historical stuff and troggle docm to do list 2019-02-26 18:19:16 +00:00			`<ul>`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<li><a href="../computing/manual.html">the <em>Data</em> Maintenance</a> manual.`
			`<li><a href="trogmanual.html">Troggle - Maintenance</a> - list of maintenance tasks<br>`
			`<li><a href="trogarch.html">Troggle - Architecture</a> - diagrams, files, structure<br>`
More links to historical stuff and troggle docm to do list 2019-02-26 18:19:16 +00:00			`</ul>`

Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<p>and you will have dismissed them as too trivial for your needs. You will have skipped over all that nasty confusing Django stuff:`
			`<ul>`
			`<a href="trogdjango.html">Troggle and Django</a> - The Django web framework we use<br>`
			`<a href="trogdjangup.html">Troggle: updating Django</a> - Upgrading troggle to use a later Django version<br>`
			`</ul>`
More links to historical stuff and troggle docm to do list 2019-02-26 18:19:16 +00:00
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`and you will also also have merely skimmed as irrelevant or trivial:`
			`<ul>`
Django migrations and troggle laptops 2021-10-24 14:31:22 +01:00			`<li><a href="serverconfig.html">Troggle server configuration</a> - troggle running on a new server<br>`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<li><a href="scriptsother.html">Other scripts</a>`
			`<li><a href="scriptscurrent.html">Additional Scripts</a> - more detail<br>`
			`<li><a href="../computing/repos.html">Version control</a> - why we have repos<br>`
add links to trog laptop page - online edit of handbook/troggle/trognotes.html 2023-04-26 20:15:08 +01:00			`<li> <a href="/handbook/troggle/troglaptop.html">Troggle development machine</a> - your laptop`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`</ul>`

			`<p>You want to dive into the real stuff and add missing fields to the core data model:`
			`<ul>`
			`<li><a href="datamodel.html">Troggle - Data Model</a> - syntax-coloured list of classes, instance variables and foreign keys<br>`
			`</ul>`
			`<p>We admire your drive and enthusiasm. We were like you once.`

			`<p>First, you need to get the "expoadmin" password from a nerd. Then you will be able to dive into the online`
			`admin system where you can explore and poke the live system and`
			`database using the <a href="/admin/">system admin control panel</a>.`
			`This has a data structure browser for the classes and fields of the system as`
			`live and in-use: all the foreign keys and relationship cardinalities.`
			`(This is not the same thing as the <a href="/troggle">troggle control panel</a> which is for deeper data management: bulk uploading and export of data files.)`

			`<p>And, you may be surprised to discover, we have given a lot of thought into how to make it easier for you`
			`to <a href="../computing/contribute.html#programming">get involved in the programming</a> while not screwing up this multi-decade software project.`

Kill it with Fire 2021-12-05 23:57:56 +00:00			`<h3>Kill it with Fire</h3>`
			`<a href="trogkill.html">`
			`<img border="1" class="onright" width="150px" src='../i/dumpsterfire.jpg' alt='dumpster fire'/></a>`
			`<p>Everyone at some point feels like giving up and wonders if we should just rewrite it all from scratch using 'modern' techniques.`
			`Several of us have given this a lot of thought, see <a href="trogsimpler.html">'A kinder, simpler troggle'</a>,`
			`the <a href="trog2030.html">2030 plan</a>, <a href="trogspeculate.html">architecture constraints</a> and most importantly, <a href="trogdjango.html#4layer">the 4-layer stack</a> <em>any</em> future technology with some public URLs would need to emulate.`
			`<p>We have also been learning from the global software community when you really do need to 'Kill it with Fire': what are`
			`the characteristics of code and of organisations that mean that it is a good idea to give up, and if so how to`
asyncio and legacy replacement patterns 2022-03-29 01:53:38 +01:00			`manage the rewrite. The answers are "almost never" and "don't": as explained in <a href="trogkill.html">this summary of the book 'Kill it with`
			`Fire'</a>.`

			`<p>Replacing old systems, and changing their architecture, while keeping them operating continuously, is now`
			`<a href="https://martinfowler.com/articles/patterns-legacy-displacement/transitional-architecture.html">a well-understood`
			`software engineering process</a> comprising a dozen separate techniques. With troggle, we can probably fairly easily replace the cross-referencing`
			`between different types of things (e.g. logbook entries, survex files, persons, expedition days) but we will find it tricky to replace the`
			`sychronisation and concurrency that SQL databases give us 'for free'. We are waiting to see if Django comes up with something cunning as it steadily`
			`converts to a more asynchronous architecture using python <a href="https://docs.djangoproject.com/en/4.0/topics/async/">asyncio</a>.`
Kill it with Fire 2021-12-05 23:57:56 +00:00
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<h3>Perhaps an external add-on?</h3>`
			`<p>Perhaps you can program something external to troggle, in JavaScript say, using troggle data?`
			`<ul>`
			`<li><a href="exportjson.html">JSON export</a> - how to extract cave and expo data in JSON format.`
			`<li><a href="exporttgz.html">Compressed data export</a> - how to extract cave and expo data in gzip/tar format.`
			`</ul>`

add links to trog laptop page - online edit of handbook/troggle/trognotes.html 2023-04-26 20:15:08 +01:00			`<h3 id="devtree">Creating <a href="/handbook/troggle/troglaptop.html">your own system development machine</a></h3>`
typos 2021-12-02 22:31:46 +00:00			`<p>It's a simple sequence which steadily builds your expertise:`
adding missing link to troggle-ish 2019-02-25 20:59:50 +00:00			`<ol>`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<li>Use the online forms and reports.`
Django migrations and troggle laptops 2021-10-24 14:31:22 +01:00			`<li>Read documentation to get the <a href="trogarch.html">concepts and structures</a> into your head, so that you can see how it all fits together and works.`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<li>Explore the public pages of the <a href="/repositories">4 git repositories</a> including <a href="/repositories/troggle/.git/tree/">all the source code</a>`
			`<li>Use the online forms and reports together with 'survex' and 'therion' installed on your own laptop.`
			`<li>Get the 'expoadmin' password and explore the live system online: watch the data change in real time`
			`<li>Configure <a href="../computing/keyexchange.html">an ssh key</a> and <a href="../computing/fzconfig.html">Filezilla</a> or scp/sFTP.`
fixing mismatches in laptop setup docm 2021-10-22 21:43:02 +01:00			`<li>Configure a <a href="../computing/basiclaptop.html">basic expo laptop</a>`
rename yourlaptop as surveylaptop 2022-11-25 20:33:16 +00:00			`<li>Configure <a href="../computing/surveylaptop.html">cave survey software on your laptop</a>`
New troggle development laptop config page 2021-10-22 14:18:09 +01:00			`<li>Configure a <a href="troglaptop.html">troggle development machine</a> - a laptop where you have cloned the repos and installed django. Installing is the same as updating Django. See <a href="trogdjangup.html">Troggle: updating Django</a>.`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`</ol>`
Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<h3>Fixing broken URL dispatch</h3>`
typos 2021-12-02 22:31:46 +00:00			`<p>This <em>happens all the time</em>. If ever troggle is crashing on you very mysteriously, it is almost certainly due to a typo in <var>troggle/urls.py</var> . Odd but true.`
Restructuring troggle notes and creation of troggle documentation directory in the handbook 2020-04-02 14:25:58 +01:00
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`<h3>Fixing a broken HTML template</h3>`
typos 2021-12-02 22:31:46 +00:00			`[to be written]`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00
			`<h3>Running databaseReset</h3>`
			`<ul>`
			`<li><a href="trogimport.html">Troggle - Data Import</a> - reset and import data<br>`
			`</ul>`
			`<h3>Running the test suite</h3>`
			`<ul>`
Correct page on testing 2021-12-01 20:24:34 +00:00			`<li><a href="trogtests.html">Troggle - Page tests and smoke tests</a> <br>`
Troggle programmers guide - beginnings 2021-04-29 17:52:28 +01:00			`</ul>`

			`<h3>This manual is very incomplete - notes</h3>`
			`<p>[Note that /survey_scans/ is a url published by troggle and is not the same thing as the real directory /expofiles/surveyscans/ at all.]`
Restructuring troggle notes and creation of troggle documentation directory in the handbook 2020-04-02 14:25:58 +01:00			`<p>Only a small part of troggle's original plan was fully implemented and deployed.`
			`Many of the things it was intended to replace are still operating as a motley collection written by many different people in`
			`several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).`
Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00
Trggle Django documentation 2023-02-22 23:36:05 +00:00			`<h3>Writing python</h3>`
			`<p>We aspire to a coding standard, and the <a href="https://github.com/google/styleguide/blob/gh-pages/pyguide.md">Google python coding standard</a> is a pretty good one. We don't use their TODO system though (para <a href="https://github.com/google/styleguide/blob/gh-pages/pyguide.md#312-todo-comments">3.12</a>) and the function (re)naming in troggle is still work in progress (para. 3.16.4).`

			`<p>We don't have any type hints in troggle. It is not a big enough project to be worth it - probably.`

			`<p>We have run much of the code through <a href="https://pypi.org/project/black/">black</a> in 2022 so that is our default formatting standard.`

			`<p>We have also used ruff, isort and chill to trim and deduplicate the various pip installation packages and to fix the order of python packages in import statements.`

Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00			`<hr />`
enlarging future troggle notes 2020-05-11 00:56:04 +01:00			`<h3><a id="arch">Archived updates</a></h3>`
			`<p>Since 2008 we have been keeping detailed records of all data management system updates in the version control system.`
			`Before then we manually maintained <a href="../computing/update.html">a list of updates</a> which are now only of historical interest.`
Sorting out troggle design docum. 2020-06-04 02:12:54 +01:00			`<p>A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article <a href="../c21bs.html">Taking Expo Bullshit into the 21st Century</a> is archived here.`
new troggle discussions 2020-05-12 15:53:37 +01:00			`<hr />`
			`Go on to: <a href="trogarch.html">Troggle architecture</a><br />`
New index to all troggle pages 2020-07-27 01:42:09 +01:00
			`Return to: <a href="trogintro.html">Troggle intro</a><br />`
			`Troggle index:`
			`<a href="trogindex.html">Index of all troggle documents</a><br />`

add links to trog laptop page - online edit of handbook/troggle/trognotes.html 2023-04-26 20:15:08 +01:00			`<hr /></body>`
Adding a page of consolidated troggle notes and hopes 2019-02-25 20:54:07 +00:00			`</html>`