expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-29 13:21:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Documn update for QMs

Browse Source
This commit is contained in:
Philip Sargent 2022-07-05 17:42:09 +03:00
parent cf839322c9
commit 463c861409
1 changed files with 41 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
handbook/troggle/scriptsqms.html
View File

@ -30,20 +30,46 @@ tl;dr - use <em>svx2qm.py</em>. Look at the output at:<br>
<h2>QMs - the fourfold path</h2>
<img class="onright" src ="../i/qm-image.jpg" />
<p>You will be familiar with <a href="../survey/qmentry.html">documenting newly found QMs</a> in the survex file when you type it in. But QMs are only useful if they can be easily scanned by people planning the next pushing trip. That's what we are discussing here.
<p>You will be familiar with <a href="../survey/qmentry.html">documenting newly found QMs</a> in the survex file when you type it in. But
QMs are only useful if they can be easily scanned by people planning the next pushing trip. That's what we are discussing here.
<p>There are four (and a half) ways we have used to manage QMs:
<p>There are four (roughly) ways we have used to manage QMs:
<ol>
<li><strong>Perl script</strong> - Historically QMs were not in the survex file but typed up in a separate list <var>qms.csv</var> for each cave system. A perl script turned that into an HTML file for the website. But there appear to be 3 different formats for this.
<li><strong>Perl + troggle</strong> - One of troggle's input parsers "QM parser" is specifically designed to import the three HTML files produced from <var>qms.csv</var> but doesn't do anything with that data (yet).
<li><strong>Hand-edited lists of QMS</strong> - only exist for 1623-161 <a href="/1623/161/qmtodo.htm">Kaninchenh&ouml;hle</a>
<li><strong>Perl script</strong> - Historically QMs were not in the survex file but typed up in a separate list <var>qms.csv</var> for
each cave system. A perl script turned that into an HTML file for the website. But there appear to be 3 different formats for this. Not currently used.
<li><strong>Perl + troggle</strong> - One of troggle's input parsers "QM parser" is specifically designed to import the three HTML files
produced from <var>qms.csv</var> and produces reports by cave and individually, e.g. see <a href="/cave/qms/1623-161">the 161 QMs</a>
(slow page), which is <em>old</em> compared with the hand-edited <a href="/1623/161/qmtodo.htm">1623-161</a> page which was derived from it.
<li><strong>Python script</strong> - Phil Withnall's 2019 script <em>svx2qm.py</em> scans all the QMs in a single survex file. See below for how to run it on all survex files.
<li><strong>New troggle</strong> - Sam's recent addition to troggle's "survex parser" makes it recognise and store QMs when it parses the survex files.
<li><strong>The elderly Prospecting Guide</strong> - covers some of the same sort sof information as needed by someone wanting to
chase QMs. It is a troggle-generated document at <a href="/prospecting_guide/">expo.survex.com/prospecting_guide/</a>. It is so old that "top camp" in the guide refers to the col camp and not the Stonebridge bivvy. Some updates were done in 2007.
<li><strong>The elderly Prospecting Guide</strong> - Used to cover some of the same sorts of information as needed by someone wanting to
chase QMs. It was a troggle-generated document at <a href="/prospecting_guide/">expo.survex.com/prospecting_guide/</a>.
It has been retired because the mapping software packages it used were terminally outdated.
</ol>
<p>The plan is to write new parsers to extract the QM information from the survex files and to
integrate these QMs into the same reports as those parsed from the CSV files.
<p>QMs all use <a href="../survey/qm.html">the same QM description conventions</a>.
<h4 id="qms.py">troggle/parsers/qms.py</a></h4>
<p>Troggle currently reports QMs for only three historic caves and ignores all the QMs inside survex files.
<p>The parser <em>troggle/parsers/qms.py</em> currently imports the <var>qm.csv</var> files used by
the 2004 perl script tablize-qms.pl (see below) into troggle using a mixture of csv and html parsers:
<code><pre>parseCaveQMs(cave='stein',inputFile=r"1623/204/qm.csv")
parseCaveQMs(cave='hauch',inputFile=r"1623/234/qm.csv")
parseCaveQMs(cave='kh', inputFile="1623/161/qmtodo.htm")
#parseCaveQMs(cave='balkonhoehle',inputFile=r"1623/264/qm.csv")</pre></code>
and reports these by cave and individually, e.g. see <a href="/cave/qms/1623-204">the 204 QMs</a> (slow page).
</p>
These URL recognisers work:
<ul>
<li>/cave/qms/&lt;caveslug&gt; e.g. <a href="/cave/qms/1623-161/">/cave/qms/1623-161/</a> (slow page)
<li>/cave/&lt;caveslug&gt;-&lt;year&gt;&lt;qm_id&gt; e.g. <a href="/cave/qms/1623-161/1997-1C">/cave/qms/1623-161/1997-1C</a>
</ul>
<p>Note that the hand-edited <var>qm.csv</var> for Balkonhohle was apparently abandoned unfinished as we transitioned to putting the QMs in the survex files instead. It contains QMs from 2014 and 2016:<br />
<a href="../../1623/264/qm.csv" download>/1623/264/qm.csv</a> - unused <br/>
<h4 id="QM_helper">js/QM_helper.js</h4>
<p>A relic.
<p>This is referred to in core/admin.py and appears to help with the userinterface within the
@ -107,16 +133,6 @@ with these details:
</ul> then a short description here is OK."
<h4 id="qms.py">troggle/parsers/qms.py</a></h4>
<p>The parser <em>troggle/parsers/qms.py</em> currently imports those same <var>qm.csv</var> files from the perl script into troggle using a mixture of csv and html parsers:
<code><pre>parseCaveQMs(cave='stein',inputFile=r"1623/204/qm.csv")
parseCaveQMs(cave='hauch',inputFile=r"1623/234/qm.csv")
parseCaveQMs(cave='kh', inputFile="1623/161/qmtodo.htm")
#parseCaveQMs(cave='balkonhoehle',inputFile=r"1623/264/qm.csv")</pre></code>
but does not apparently have any output webpage to display them (yet).
</p>
<p>Note that the hand-edited <var>qm.csv</var> for Balkonhohle was apparently abandoned unfinished as we transitioned to putting the QMs in the survex files instead. It contains QMs from 2014 and 2016:<br />
<a href="../../1623/264/qm.csv" download>/1623/264/qm.csv</a> - unused <br/>
<h4 id="svx2qm">svx2qm.py</a></h4>
<p>Philip Withnall's 2019 QM extractor <em>svx2qm.py</em> (in :loser:/qms/) can be used to generate a list of all the QMs in all the svx files in either text or CSV format. When run together with <em>file</em> and <em>xargs</em> it will produce a output listing all the QMs:
@ -135,27 +151,25 @@ This will work on all survex *.svx files even those which have not yet been run
<p>Phil says (13 April 2020): <em>"The generated files are not meant to be served by the webserver, it's a tool for people to run locally. Someone could modify it to create HTML output (or post-process the CSV output to do the same), but that is work still to be done."</em>
<h4>troggle/parsers/survex.py</a></h4>
<p>The QMs inside the survex files are parsed by troggle along with all the other information
inside survex files and stored in the database. But the webpages which display this data are very rudimentary and currently useless, e.g. <a href="/getQMs/1623-204">/getQMs/1623-204</a> or <a href="/cave/qms/1623-204">/cave/qms/1623-204</a>.
Looking through urls.py and core/view_caves.py we see a lot of code for providing new QM numbers, producing lists of QMs for a given cave and for downloading QM.csv files generated by the database. But none of it appears to be working today (14 May 2020), see below.
<p>Troggle <em>troggle/parsers/survex.py</em> currently parses and stores all the QMs it finds in survex files. The tables where the data
is put are listed in <a href="datamodel.html">the current data model</a> including structure for ticking them off. These QMs are stored in
the database. But the webpage which displays this data is currently broken, e.g. <a href="/getQMs/1623-204"> /getQMs/1623-204</a>.
<h4 id="samqms">Sam's parser additions</a></h4>
<p>Troggle <em>troggle/parsers/survex.py</em> currently parses and stores all the QMs it finds in survex files. The tables where the data is put are listed in <a href="datamodel.html">the current data model</a> including structure for ticking them off.
<h4>troggle archeology</a></h4>
<p>Looking through urls.py and core/view_caves.py we see a lot
of archaic code for providing new QM numbers, producing lists of QMs for a given cave and for downloading QM.csv files generated by the database.
But none of it appears to be working today (5 July 2022).
<p>There is a troggle template file :troggle:/templates/qm.html which is intended to become a useful outstanding QM report in future. Though since it was last edited in 2009, this does not seem to be on anyone's urgent task list.
<p>Troggle has archaic URL recognisers in <var>:troggle:/urls.py</var> for:
<ul>
<li>/newqmnumber/ - crashes troggle
<li>/getQMs/&lt;caveslug&gt; - crashes troggle
<li>/cave/qms/ such that <a href="/cave/qms/1623-161/">/cave/qms/1623-161/</a> doesn't actually crash
<li>/cave/&lt;caveslug&gt;-&lt;year&gt;&lt;qm_id&gt;- crashes troggle
<li>/cave/&lt;cave-id&gt;/qm.csv - to download a <var>qm.csv</var> file (NB not qms.csv) - crashes troggle
<li>/downloadqms - crashes troggle
</ul>
So someone was busy at one time.
<p>There is not yet a troggle report listing the QMs which works.
<h2>QMs - monitoring progress</h2>
<h4 id="find-dead-qms">find-dead-qms.py</h4>