expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Documenting the weird and wonderful set of scripts we have

Browse Source
This commit is contained in:
Philip Sargent 2020-04-11 23:48:36 +01:00
parent 60ce9e0f93
commit 1a7a1a1b64
4 changed files with 177 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
handbook/computing/manual.html
View File

@ -122,10 +122,18 @@ editing and keeps track of all changes so we can roll back and have branches if
<h3><a id="cavepages">Updating cave pages</a></h3>
<span style="color:red">
<p>Cave description pages are automatically generated from a set of
<p>Public cave description pages are automatically generated by troggle from a set of
cave files in noinfo/cave_data/ and noinfo/entrance_data/. These files
are named <area>-<cavenumber>.html (where area is 1623 or 1626). These
files are processed by troggle. Use <tt>python databaseReset.py
are named <area>-<cavenumber>.html (where area is 1623 or 1626).
<p>
Read the survey handbook section on <a href="../survey/caveentry.html">creating a new cave</a> in the system for instructions on how to name caves and the files you use to recoird them.
<p>Cave names do not have leading zeros
They are stored by number/ID in the dataset, not by name.
<p>Caves with a provisional number consisting of a year and a serial number
should be hyphenated, thus 2002-04 not 2002_04 or any of the various other
variants
<p>Use <tt>python databaseReset.py
caves</tt> in /expofiles/troggle/ to update the site/database after
editing these files.
Use <tt>python databaseReset.py

6
handbook/computing/todo.html
View File

@ -126,7 +126,7 @@ If a heading is in italics, then there are hidden items.
</dl>
<h3>System Documentation</h3>
<dl>
<dl><!-- 2020-04-11 psargent -->
<dt><!--2020-04-11 psargent-->Document how we generate the SMK overall lengths and stats</dt>
<dd><!--2020-04-11 psargent-->scripts and manual actions in cavern
@ -159,8 +159,8 @@ If a heading is in italics, then there are hidden items.
<dd><!--2020-04-03 psargent-->REPORT if import surveyscan file failure
<dd><!--2020-04-03 psargent-->Why are so many surveyscan foolder missing after import?
<dd><!--2020-04-03 psargent-->WHY are total lengths of surveyed cave different between expo server and WSL server for 2019 and a few other years only ?
<dt>NOEDIT flag
<dd>Descripe how this is implemented using &lt;meta name="keywords" content="NOEDIT"&gt;
<dt><!-- 2020-04-11 psargent -->NOEDIT flag
<dd><!-- 2020-04-11 psargent -->Describe how this is implemented using &lt;meta name="keywords" content="NOEDIT"&gt;
<dt>Logbook data model: unfilled fields?
<dd><!--2020-04-03 psargent-->'cave' of location field missing on nearly all logbook entroies. Missing from logbook templates. Where is this used and how can we fill it it in? Is it useful?

57
handbook/troggle/otherscripts.html
View File

@ -8,41 +8,50 @@
<body>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Other scripts</h1>
<p>Apart from these scripts, troggle in full deployment also needs a running mySQL database, a running apache webserver and cgit to display git repos.
<h3 id="inscripts">Scripts</a></h3>
<h3 id="inscripts">Current Scripts</a></h3>
<ul>
<li><a href="">make-folklist.py</a> turns the folk.csv file into a webpage.
<li><a href="">updatephotos</a> generates navigable webpages from annual directories of photo images
<li><a href="">wallets.py</a> generates statuspages and to-do list pages for survey data production.
<li><a href="">make_essentials.sh</a> generates GPS tracks and routes for surface walking
<li><a href="">make_svx.sh</a> generates surface Survex tracks
<li><a href="">svxtrace.py</a> analyses Survex tracks to help find errors
<li><a href="">chk-refs.sh</a> analyses Survex files and wallets files to find reference errors
<li><a href="">create_dplong_table.py</a> Creates and updates the lengths and depths of all the caves
<li><a href="">make-areaindices.py</a> (obsolete?)
<li><a href="">make-prospectingguide-new.py</a> and <a href="">prospecting_guide_short.py</a> - unmaintained
<li><a href="">boe</a> Bank of Expo runs the accounts preceding and during expo
<li><a href="">bierbook.tex</a> LaTeX script for generating the bierbook - a new list of names and dates each year
<li><a href="">seshbook.tex</a> LaTeX script for generating the seshbook - works from the same list of names
<li><a href="">therionpage.tex</a> LaTeX script anbd makefile for generating therion-style protractors
<li><a href="scriptscurrent.html#folk">make-folklist.py</a> turns the folk.csv file into a webpage.
<li><a href="scriptscurrent.html#gps">make_essentials.sh</a> generates GPS tracks and routes for surface walking
<li><a href="scriptscurrent.html#photos">updatephotos</a> generates navigable webpages from annual directories of photo images
<li><a href="scriptscurrent.html#dplong">create_dplong_table.py</a> Creates and updates the lengths and depths of all the caves
<li><a href="scriptscurrent.html#area">make-areaindices.py</a> (unmaintained?)
<li><a href="scriptscurrent.html#prosp">make-prospectingguide-new.py</a> and <a href="">prospecting_guide_short.py</a> - (unmaintained?)</li><br />
<li><a href="../bankofexpo.html">boe</a> Bank of Expo (perl) runs the accounts preceding and during expo
<li><a href="scriptscurrent.html#wallets">bierbook.tex</a> LaTeX script for generating the bierbook - a new list of names and dates each year
<li><a href="scriptscurrent.html#latex">seshbook.tex</a> LaTeX script for generating the seshbook - works from the same list of names
<li><a href="scriptscurrent.html#latex">therionpage.tex</a> LaTeX script anbd makefile for generating therion-style protractors</li><br />
<li><a href="scriptscurrent.html#latex">wallets.py</a> generates statuspages and to-do list pages for survey data production.
<li><a href="scriptscurrent.html#svx2qm">svx2qm.py</a> extracts QMs from the survex files
<li><a href="scriptscurrent.html#tabqms">tablize-qms.pl</a> turns the list of QMs extracted into an HTML file
<li><a href="scriptscurrent.html#surface">make_svx.sh</a> generates surface Survex tracks
<li><a href="scriptscurrent.html#svxtrace">svxtrace.py</a> analyses Survex tracks to help find errors
<li><a href="scriptscurrent.html#survex">chk-refs.sh</a> analyses Survex files and wallets files to find reference errors</li><br />
<li><a href="scriptscurrent.html#cavestats">cavestats</a> extracts overall lengths and depths from all the survex files
<li><a href="scriptscurrent.html#makefile">Makefile</a> coordinates running many of these scripts (needs maintenance)
</ul>
<p>There are also scripts running cron jobs on the server to fix file permissions and to periodically tidy repositories, and example rsync and scp scripts to help manage synchronisation of the expofiles directories which are not under version control.
<h4 id="wallets">Wallets</a></h4>
<p><a href="../survey/onlinewallet.html">Online wallets</a> are maintained using the <a href="">wallets.py</a> script, but troggle also directly imports all the expofiles/surveyscans/ directories of scanned survey notes.
<p>Apart from these scripts, troggle in full deployment also needs <br>- a running mySQL database, <br>- a running apache webserver and <br>- cgit to display git repos.
<h4 id="survex">Survex files</a></h4>
<p>Survex files contain a reference to the wallet which contains the original survey notes for that surveyed passage. These sometimes have errors and also get out of date as caves get renamed when they get a kataster number issued. Each online survey wallet also has a reference to the survex file(s) which have been typed up from that data.
<h3 id="inscripts">Old and possibly obsolete scripts</a></h3>
<ul>
<li><a href="scriptscurrent.html#glossary">make-glossary.pl</a> - used while documenting very complex caves (161 and 204)
<li><a href="scriptscurrent.html#find-dead-qms">find-dead-qms.py</a> - several QM scripts and intermediate files
<li><a href="scriptscurrent.html#alljs">make-alljs.py</a> - extracts legs and entrances
<li><a href="scriptscurrent.html#indxl4">make-indxal4.pl</a> - obsolete
<li><a href="scriptscurrent.html#logbk">logbk.pl</a> - obsolete
</ul>
<p>There are perl and python scripts scattered through the :drawings: and :loser: (svx files) repos. Not all of the functions in these scripts have been recreated in more recent scripts or programmed into troggle.
<h4 id="folk">Folk</a></h4>
<p><a href="../computing/folkupdate.html">Folk update</a> process produces a webpage listing all expo participants but it also runs some validation checks on the input file /folk/folk.csv . Troggle also directly imports folk.csv so that it knows who everyone is, but errors during the importing are not as easy to see as validation errors printed when running the <a href="">make-folklist.py</a> script.
<h4 id="photos">Photos</a></h4>
<p><a href="">updatephotos</a> uses the BINS package to generate the webpages. BINS is no longer maintained by its author so expo has taken on the responsibility for keeping it running. (Wookey is in the process of packaging it as a proper debian package).
<hr />
Go in to: <a href="scriptscurrent.html">More details about these scripts</a><br />
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
Return to: <a href="trogintro.html">Troggle intro</a><br />
<hr />

130
handbook/troggle/scriptscurrent.html Normal file
View File

@ -0,0 +1,130 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Other scripts supporting troggle</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<style>
h4 {
margin-top: 1.5em;
margin-bottom: 0;
}
p {
margin-top: 0;
margin-bottom: 0.5em;
}
</style>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Current scripts</h1>
<h2>Current scripts</h2>
<p>See index to the full list of these scripts at <a href="otherscripts.html">Other Scripts</a>. This page only documents a sub-set which is not more fully documented elsewhere.
<p>[This page should be split so that the obsolete stuff is recorded but doesn't get in the way.]
<h4 id="makefile">Makefiles</h4>
<p>In :expoweb:/noinfo/
<p>This coordinates running very essential updates but also includes redundant and obsolete stuff. It <span style="color:red">needs some serious attention</span>.
<p>It coordinates producing the 3d surveys used in the cave description pages, updates the area pages, runs the folk script, runs the QM list generation within each of the cave pages that needs it, runs svxtrace, and reports on everything usign "bigbro" which we don't have any other reference to.
<h4 id="wallets">Wallets</h4>
<p><a href="../survey/onlinewallet.html">Online wallets</a> are initially maintained using the <a href="/expofiles/surveyscans/wallets.py">wallets.py</a> script, but troggle also directly imports all the expofiles/surveyscans/ directories of scanned survey notes and produces <a href="/survey_scans/">reports</a> on then.
<h4 id="folk">Folk</a></h4>
<p><a href="../computing/folkupdate.html">Folk update</a> process produces a webpage listing all expo participants and it also runs some validation checks on the input file /folk/folk.csv . Troggle also directly imports folk.csv so that it knows who everyone is, but errors during the importing are not as easy to see as validation errors printed when running the <a href="../computing/folkupdate.html">make-folklist.py</a> script.
<h4 id="photos">Photos</a></h4>
<p><a href="">updatephotos</a> (in the :loser: repo) uses the BINS package to generate photo albums. BINS uses the EXIF data (date, location if available) in the uploaded image files to produce a page showing the information available about the picture. All image meta-data are stored in XML files.
<p>BINS is no longer maintained by its author so expo has taken on the responsibility for keeping it running. (Wookey is in the process of packaging it as a proper debian package).
<h4 id="svx2qm">svx2qm.py</a> and QMs</h4>
<p>Philip Withnall's QM extractor (in :loser:/qms/). It generates a list of all the QMs in all the svx files in either text or CSV format. This will produce a text output of all the QMs:
<pre><code>cd loser
find -name '*.svx' | xargs ./svx2qm.py --format csv
</code></pre>
<h4 id="tabqmsqms">tablize-qms.pl</h4>
<p>Takes a CSV file name as the program's argument (e.g. qm.csv as generated by svx2qm.py) and generates an HTML page listing all the QMs.
<p>In :expoweb:/1623/204/ and several other subdirectories of /1623/
<h4 id="find-dead-qms">find-dead-qms.py</h4>
<p>This finds references to completed qms in the cave descriptions.
<p>Also
<a href="#qmreader">qmreader.pl</a> which reads and parses qm.html (why?)
<p>In :expoweb:/1623/204/ - Nial Peters (2011)
<h4 id="svxtrace">svxtrace.py</h4>
<p>Traces all the svx file dependencies via the *include statements. In :expoweb:/ .
<h4 id="survex">Survex files</a> - reference checking</h4>
<p>Survex files contain a reference to the wallet which contains the original survey notes for that surveyed passage. These sometimes have errors and also get out of date as caves get renamed when they get a kataster number issued. Each online survey wallet also has a reference to the survex file(s) which have been typed up from that data.
<p>Validating the references is checked by scripts check-svx.sh, check-refs.sh, check-refs.awk in the :loser: repository to produce <a href="../../svxvalid.html">svxvalid.html</a> which lists mismatches between the svx files and the survey scan wallets.
<h4 id="draw">Drawings files</a> - reference checking</h4>
<p>Tunnel files contain references to the wallet which contained the original survey notes for that surveyed and drawn passage.
<p>The results of validation checks are in <a href="../../xmlvalid.html">xmlvalid.html</a> and generated by script check-xml.sh in the :drawings: repository.
<p>(Therion files would too, if people inserted "#Ref" comments. In which case the script would need improving.)
<h4 id="gps">make_essentials.sh GPS</h4>
<p>In :expoweb:/noinfo/
<p>Makes essentials.gpx - see <a href="../essentials.html">GPS on expo</a>.
<h4 id="dplong">create_dplong_table.py</h4>
<p>In :expoweb:/noinfo/
<p><span style="color:red">[to be documented]</span>
<h4 id="area">make-areaindices.py</h4>
<p>In :expoweb:/noinfo/
<p><span style="color:red">[to be documented]</span>
<h4 id="prosp">make-prospectingguide-new.py</h4>
<p>In :expoweb:/noinfo/prospecting_guide_scripts/
<p><span style="color:red">[to be documented]</span>
<h4 id="latex">seshbook, bierbook & protractors</h4>
<p>How these are used once produced is <a href="../bierbook.html">documented in the the handbook</a>
<p>These are LaTeX files and the instructions for how to process them are in each .tex file. The protractors do not change but the others need a configuration file for all the cavers expected to attend expo.
<p>The .tex files are in :expoweb:/documents/. There is a style file also there bierbook-style.sty which is used by both the bierbook and seshbook. Read the readme.txt file which explains which LaTeX packages you need. Build like this:
<p><pre><code>pdflatex.exe -synctex=1 -interaction=nonstopmode -shell-escape bierbook.tex
pdflatex.exe -synctex=1 -interaction=nonstopmode -shell-escape seshbook.tex
</code></pre>
Due to the way LaTeX works out table column witdths, these commands may need to be run several times until a stable output is produced.
The design of these files is intended to confine all changes year to year to the names.txt and dates.txt files, thanks to LaTeX's capability to read an external file and iterate through line by line performing the same action for each name.
<h4 id="surface">make_svx.sh</h4>
<p><span style="color:red">[to be documented]</span>
<h4 id="cavestats">cavestats</h4>
<p><span style="color:red">[to be documented]</span>
<h3 id="inscripts">Old and possibly obsolete scripts</a></h3>
<h4 id="alljs">make-alljs.py</h4>
<p>Writes out legs and entrances in json format. In :loser:/fixedpts/
<h4 id="glossary">make-glossary.pl</h4>
<p>In :expoweb:/1623/204/ and /1623/161/. It reads a cave-specific glossary.csv and produces the HTML files for caves 161 and 204:
<ul>
<li><a href="/1623/204/atoz.html">/1623/204/atoz.html</a>
<li><a href="/1623/161/a-z.htm">/1623/161/a-z.htm</a>
</ul>
<p>which are indexes to passage names and locations in the very extensive vcave descriptions for Kaninchenhohle and Steinbruckenhohle.
<h4 id="indxl4">make-indxal4.pl</h4>
<p>Obsolete.
<p>See history documents which refer to CAVESTATS.CSV
<p><span style="color:red">[to be documented - for historic interest only]</span>
<h4 id="logbk">logbk.pl</h4>
<p>Obsolete.
<p>This function is now done by the troggle input parsers.
<p><span style="color:red">[to be documented - for historic interest only]</span>
<hr />
Return to: <a href="otherscripts.html">Other scripts</a><br />
<hr />
</body>
</html>