You will be familiar with documenting newly found QMs in the survex file when you type it in. But QMs are only useful if they can be easily scanned by people planning the next pushing trip. That's what we are discussing here.
There are four (roughly) ways we have used to manage QMs:
The plan is to write new parsers to extract the QM information from the survex files and to integrate these QMs into the same reports as those parsed from the CSV files.
QMs all use the same QM description conventions.
Troggle currently reports QMs for only three historic caves and also imports all the QMs inside survex files. Thus a recent cave such as 1623-264 (Balkhöhle)will only show QMs imported from the survex files:
The bug in the individual QMs is probably in the importing process.
The parser troggle/parsers/qms.py currently imports the qm.csv files used by
the 2004 perl script tablize-qms.pl (see below) into troggle using a mixture of csv and html parsers:
parseCaveQMs(cave='stein',inputFile=r"1623/204/qm.csv")
parseCaveQMs(cave='hauch',inputFile=r"1623/234/qm.csv")
parseCaveQMs(cave='kh', inputFile="1623/161/qmtodo.htm")
#parseCaveQMs(cave='balkonhoehle',inputFile=r"1623/264/qm.csv")
Note that the hand-edited qm.csv for Balkonhohle was apparently abandoned unfinished as we transitioned to putting the QMs in the survex files instead. It contains QMs from 2014 and 2016:
/1623/264/qm.csv - unused
A relic.
This is referred to in core/admin.py and appears to help with the userinterface within the Django Admin control panel for manipulating QMs. It is not live as media/js/ is not plumbed in. (Live javascript lives in media/jslib/ which is routed to the URL /javascript/.)
This is a perl script dating from November 2004.
it takes a hand-edited CSV file name as the program's argument and generates an HTML page listing all the QMs.
Varient copies of it (they are all slightly different) live in the three cave file folders in :expoweb:/1623/, in 258/, 234/, and 204/ . These generated html files are live pages in the cave descriptions:
/1623/258/qm.html
/1623/234/qm.html
/1623/204/qm.html
Note that the qms.csv file file used as input by this script is an entirely different format and table structure from the qms.csv file produced by svx2qm.py.
And in fact the formats of these 3 qm.csv files are not the same (These are the
"older or artisanal QM formats" referred to by Phil Withnall at th ebottom if this page) :
Fields in 204/qm.csv are:
Number, grade, area, description, page reference, nearest station, completion description, Comment
e.g.
C1999-204-09 C Wolp Hole in floor through dangerous boulders veined.10 Filled with rocks
Cave, year, number, Grade, nearest station, description, completion description, found by, completed by
e.g.
258 2006 27 C 258.gknodel.4 Small passage to E in Germknödel Sandeep Mavadia and Dave Loeffler
Year, number, Grade, Survey folder ref#, Surveyname, Nearest Station number, Area of the cave, Description, Y if marked on drawn-up survey,
2014 7 C 2014#11 roomwithaview 4 Room With a View Room With a View: "Probably chokes" opposite stations 4 and 5 ALREADY EXPLORED PROBABLY
There are also three versions of the QM list for cave 161 (Kaninchenhohle) apparently produced by this method but hand-edited:
/1623/161/qmaven.html 1996 version
/1623/161/qmtodo.html 1998 version
/1623/161/qmdone.html 1999 (incomplete) version
In the /1623/204/ folder there is a script qmreader.pl which apparently does the inverse of tablize-qms.pl: it transforms a QMs' HTML file into a CSV file.
As Wookey says (Slack, 7 Jan. 2020): "I'm not quite sure what the best format is. Some combination of the 258 and 264 formats might be best. Including the cave number seems pointless. Including 'conclusion' info seems like a good idea. I'm not sure there what the benefit of separating the 'surveyname' and 'nearest station' fields is. Having an 'area of cave' field is somewhat useful for grouping, even though it is sort-of repeating the 'survey-station' info. If I was making a QM list I'd enter these fields: year, number, Grade, nearest station, folder reference, description, found by, completed (Year), completion description/cave description link, completed by with these details:
Philip Withnall's 2019 QM extractor svx2qm.py (in :loser:/qms/) can be used to generate a list of all the QMs in all the svx files in either text or CSV format. When run together with file and xargs it will produce a output listing all the QMs:
cd loser
find -name '*.svx' | xargs qms/svx2qm.py --format csv
The 2019 copies are online in /expofiles/: qms2019.txt and qms2019.csv.
This will work on all survex *.svx files even those which have not yet been run through the troggle import process.
Phil says (13 April 2020): "The generated files are not meant to be served by the webserver, it's a tool for people to run locally. Someone could modify it to create HTML output (or post-process the CSV output to do the same), but that is work still to be done."
Troggle troggle/parsers/survex.py currently parses and stores all the QMs it finds in survex files. The tables where the data is put are listed in the current data model including structure for ticking them off. These QMs are stored in the database. But the webpage which displays this data is currently broken, e.g. /getQMs/1623-204.
Looking through urls.py and core/view_caves.py we see a lot of archaic code for providing new QM numbers, producing lists of QMs for a given cave and for downloading QM.csv files generated by the database. But none of it appears to be working today (5 July 2022).
Troggle has archaic URL recognisers in :troggle:/urls.py for:
This stand-alone script finds references to completed qms in the qm.csv files in the cave folders (/1623/ etc.) in the :expoweb: repository. It looks to see which QMs have been completed but where there is not yet a matching text in the cave description.
Quick and dirty Python script to find references to completed qms in the cave description pages. Run this to find which bits of description need updating.
The list of qms is read from the qm.csv file and any with an entry in the "Completion description" column (column 7) are searched for in all the html files.
The script prints a list of the completed qms that it found references to and in which file.
Nial Peters - 2011
From: Philip Withnall [tecnocode] Sent: 13 April 2020 23:41 To: Philip Sargent (Gmail) Subject: Re: svx2qm Hi Philip, Hope you're well, thanks for getting in touch about this. The generated files are not meant to be served by the webserver, it's a tool for people to run locally. Someone could modify it to create HTML output (or post-process the CSV output to do the same), but that is work still to be done. I can't see any problem with moving it all to expoweb/scripts/ - so long as it is run with the loser top level directory specified - but I might be mistaken: find /home/expo/loser -name '*.svx' | xargs ./svx2qm.py --format human and it should go into the Makefile too at some point. Feel free to move it wherever; I am not planning on doing any further work on it. The script itself just expects to be passed some (relative or absolute) paths to SVX files, so can be placed wherever, as long as it's passed appropriate relative paths. I haven't written any other scripts which post-process the data or otherwise format it. I guess it all depends on what questions people are trying to answer using the QM data, as to how (and where) best to present it. I'm afraid I don't have any suggestions there. :Rob Watson wrote some documentation about QMs :http://expo.survex.com/handbook/survey/qmentry.html :is there anything subtle missing as to how they are used ? Nope, I think Rob's page covers it all. That page also documents the correct QM format which is what svx2qm.py understands. (There were some older or artisanal QM formats floating around at one point, although I think I reformatted them all so the tool would understand them, and so people would hopefully standardise on what Rob's documented from then on.) Philip