mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2024-11-29 13:21:58 +00:00
276 lines
13 KiB
HTML
276 lines
13 KiB
HTML
<html>
|
|
<head>
|
|
<title>Survey Handbook - online wallet</title>
|
|
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
|
|
</head>
|
|
<body>
|
|
<h2 id="tophead">CUCC Expedition Handbook</h2>
|
|
<h1>Maintaining the online wallets</h1>
|
|
<p>If you are a newcomer to the system, read the <a href="newcave.html#onlinew">beginners introduction to online wallets</a> first.
|
|
|
|
<h3>Why we have online wallets</h3>
|
|
<p>There are three quite different reasons:
|
|
<ol>
|
|
<li>The scans of the survey notebook pages are the ultimate original raw survey data and completely irreplaceable.
|
|
<li>The other files in the wallet are part of the process of producing a survey of the cave as a whole.
|
|
<li>Individual to-do lists are produced automatically for each caver listing what survey processing tasks they haven't finished yet.
|
|
</ol>
|
|
|
|
<h3>The scanned pages</h3>
|
|
<p>These are simply the scanned imaages (or digital photographs) of each page of the original survey notes.
|
|
They should be named <em><span style="font-family:monospace">notesXXX.jpg</span></em> where "XXX" can be
|
|
anything you like. Typically we have the scanned pages called notes1.jpg, notes2.jpg, notes3.jpg.
|
|
<p>It is important that you use use the .jpg (JPEG) file format, and definitely not PNG (very voluminous)
|
|
or PDF (very hard to re-use elsewhere). Set the scanner at 300 dpi and adjust the contrast of the image after scanning
|
|
by using photo-editing software to enhance the writing. Also please crop each image to just the area containing
|
|
the survey data.
|
|
<p>As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) upload the entire online wallet to the expo server in Cambridge
|
|
<span style="font-family:monospace">expo.survex.com</span>. This is so that these precious files are backed-up as soon as possible.
|
|
|
|
<h3>The other files and online index <em><span style="font-family:monospace">contents.json</span></em></h3>
|
|
<p>All the other files are part of the multi-step process of producing the cave survey - see <a href="newcave.html">
|
|
Creating a new cave...</a> for the full list of steps.
|
|
|
|
<p>We keep an index of how many of those steps have been completed in two places:
|
|
<ul>
|
|
<li>On paper, in the tick boxes in the contents list of this years' survey trips' lever-arch file in the potato hut.
|
|
<li>Online in the <em><span style="font-family:monospace">contents.json</span></em> file which exists in each trip online wallet.
|
|
</ul>
|
|
|
|
<p>but the <em><span style="font-family:monospace">contents.json</span></em> file has another,completely different function:
|
|
it is the <b>only online record</b> that connects the wallet number to the cave identifier. So if a future cave surveyor deperately needs
|
|
to consult the original cave survey, it can be done by, e.g. <br>
|
|
<span style="font-family:monospace">
|
|
grep -rl "2018-dm-07" expofiles/surveyscans
|
|
</span><br>
|
|
will find and list all the wallets which contain survey data for cave 2018-dm-07 (which is also known as "Homecoming Cave" and which will
|
|
have a different Austrian Kataster number issued for it in due course).
|
|
|
|
<p> The paper tick-list tracks the following steps for each online wallet:
|
|
<ul>
|
|
<li><u>Survex</u>
|
|
<ul>
|
|
<li>Data
|
|
<li>LRUD
|
|
<li>Description + QMs
|
|
</ul>
|
|
<li><u>Drawn</u>
|
|
<ul>
|
|
<li>Plan + Xsections
|
|
<li>Elevation
|
|
</ul>
|
|
<li><u>Scanned</u>
|
|
<ul>
|
|
<li>In cave notes
|
|
<li>Plan + Xsections
|
|
<li>Elevation
|
|
</ul>
|
|
|
|
<li>Tunnel
|
|
<li>Online guidebook updated
|
|
<li>json file edited
|
|
</ul>
|
|
<p>(where the "json file updated" step only refers to the initial editing of the json file to ensure
|
|
that it has the right people, date and cave identifier and name).
|
|
|
|
<p>A fully-populated and complete <em><span style="font-family:monospace">contents.json</span></em> file
|
|
looks like this:
|
|
<pre>
|
|
{
|
|
"description written": false,
|
|
"website updated": false,
|
|
"people": [
|
|
"Dickon Morris",
|
|
"Jon Arne Toft",
|
|
"Becka Lawson"],
|
|
"elev not required": false,
|
|
"cave": "2018-dm-07",
|
|
"survex not required": false,
|
|
"qms written": true,
|
|
"plan not required": false,
|
|
"electronic survey": false,
|
|
"plan drawn": true,
|
|
"date": "2018-07-13",
|
|
"elev drawn": true,
|
|
"description url": "",
|
|
"survex file": "caves-1626/2018-dm-07/2018-dm-07.svx",
|
|
"name": "Homecoming cave"
|
|
}
|
|
</pre>
|
|
Yes, this is <a href="https://en.wikipedia.org/wiki/JSON">a programming format</a>
|
|
(standardised in 2013) and every comma is critical.
|
|
|
|
<h3>"To do" lists for every caver</h3>
|
|
<p>The folder containing all the wallets for the year, e.g.
|
|
<pre>
|
|
/home/expo/expofiles/surveyscans/2018/
|
|
</pre>
|
|
will, after the appropriate magic has happened, contain a file
|
|
<pre>
|
|
index.html
|
|
</pre>
|
|
which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them.
|
|
You can see <a href="../../../expofiles/surveyscans/2016/index.html">the index.html for 2016</a>
|
|
(most of these have been done, but the index has not been updated). Also there will be a linked file for each individual
|
|
for their personal to-do list, and each online wallet contains its own
|
|
<span style="font-family:monospace">index.html</span> file which describes the survey production status for all the wallets.
|
|
|
|
<p>All this magic is created by a script <span style="font-family:monospace">make.py</span>.
|
|
|
|
<h3>Setting up the online wallets</h3>
|
|
<p>When, at the beginning of expo, you create the folder in
|
|
<span style="font-family:monospace">expofiles/surveyscans/</span> for the current year, e.g.
|
|
<span style="font-family:monospace">/2018/</span>, you will copy <span style="font-family:monospace">make.py</span>
|
|
from the previous year's folder. You will do this on your own laptop or on the expo laptop.
|
|
<p>You will also manually create a number of subfolders, e.g. 2018#01, 2018#02 etc. to be ready for the influx of
|
|
new trip surveys.
|
|
<p>Next you will test that the magic works: open a terminal in
|
|
<span style="font-family:monospace">expofiles/surveyscans/2018/</span> and run
|
|
<pre>
|
|
python make.py
|
|
</pre>
|
|
<p>which will work immediately without errors because you are using Linux and the
|
|
correct version of python is already installed.
|
|
<p>This will create a default <span style="font-family:monospace">contents.json</span> and
|
|
<span style="font-family:monospace">index.html</span> in each online wallet folder and also a
|
|
<span style="font-family:monospace">index.html</span> in the /2018/ folder.
|
|
|
|
<h3>Maintaining the online wallets</h3>
|
|
<p>Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the
|
|
<span style="font-family:monospace">contents.json</span> file in their wallet. In your dreams.
|
|
|
|
<p>The first difficulty when editing a blank <span style="font-family:monospace">contents.json</span>
|
|
for a newly-created wallet is finding out which cave the wallet describes.
|
|
The wallet may say "radaghost to blitzkriek"
|
|
(or whatever) but without the name of the cave you can't find the .svx files
|
|
as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/.
|
|
Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the
|
|
informal name of the cave,e.g. "Homecoming",instead of the identifier "2018-dm-07" you want.
|
|
<p>A regular task during expo is for a nerd to review the <span style="font-family:monospace">contents.json</span> files for
|
|
recently created wallets and to check that names, dates and cave numbers are correct.
|
|
|
|
You will run
|
|
<pre>
|
|
python make.py
|
|
</pre>
|
|
regularly, after every batch of survey data is entered or scanned.
|
|
<p>This will always overwrite all the <span style="font-family:monospace">index.html</span> files but it will never touch
|
|
the <span style="font-family:monospace">contents.json</span> files.
|
|
<p>You will also regularly synchronise your laptop
|
|
and the expo laptop with <span style="font-family:monospace">expo.survex.com/expofiles/surveyscan/2018/</span>
|
|
and this is where it gets tricky.
|
|
<p><span style="font-family:monospace">expo.survex.com/expofiles/</span> is not under version control,
|
|
so the most recent person
|
|
to upload the contents of <span style="font-family:monospace">/2018/</span> will overwrite everyone else's work.
|
|
This does not matter for the autogenerated files, but it is vital that it does not overwrite all the painfully manually edited
|
|
<span style="font-family:monospace">contents.json</span> files. Which is very easy to do.
|
|
This does mean that this is one of the cases where
|
|
it is better to use <span style="font-family:monospace">rsync</span> rather than an FTP client such as Filezilla.
|
|
|
|
<p>A copy of useful rsync scripts is kept in a file such as
|
|
<span style="font-family:monospace">expo.survex.com/expofiles/rsync2018toserver</span>. Always run it with the -n option first,
|
|
to see what overwriting you will do.
|
|
|
|
<h3>More <em><span style="font-family:monospace">make.py</span></em> magic</h3>
|
|
<p>The python script does more than just re-format the <span style="font-family:monospace">contents.json</span> data into
|
|
different formats. It also
|
|
<ul>
|
|
<li>checks whether the .svx files listed are actually present,
|
|
<li>checks for the presence of notesXXX.jpg, planXXX.jpg and elevXXX.jpg files
|
|
<li>creates a template <span style="font-family:monospace">contents.json</span> in any wallet which does not have one.
|
|
</ul>
|
|
|
|
<p>Things it might do in future (if someone gets around to it) include: <br>
|
|
- checking the cave number specified matches the folder for the .svx file,<br>
|
|
- detecting whether there is a description or a list of QMs in the survex file,<br>
|
|
- accepting a list of .svx files and not just one (a very common thing),<br>
|
|
- checking the name of the cave against the cave number,<br>
|
|
- checking whether the website page even exists for this cave,<br>
|
|
- being more intelligent about .topo files and thus the lack of scan files,<br>
|
|
- checking the date is in the recent past etc.<br><br>
|
|
|
|
<h3>How <em><span style="font-family:monospace">contents.json</span></em> fields match
|
|
<em><span style="font-family:monospace">index.html</span></em> reports</h3>
|
|
<p>
|
|
<em>to be written...</em>
|
|
<hr>
|
|
Old notes, being turned into real documentation...
|
|
<pre>
|
|
# Instructions
|
|
# 2018-08-14
|
|
# Philip Sargent
|
|
|
|
|
|
Wookey told me to sort out the contents.json files in expofiles/surveyscans/2018/
|
|
and these are my notes to remind myself what this entails.
|
|
|
|
The job is to populate the contents.json file in each folder, e.g.
|
|
|
|
expofiles/surveyscans/2018/2018#03/contents.json
|
|
|
|
using the following input materials:
|
|
- the wallet 2018#03 and the papers inside it. This is in the 2018 lever-arch file.
|
|
- the folder in repo 'loser' holding the appropriate .svx files e.g.
|
|
"caves-1623/2017-cucc-24/gshclimb.svx"
|
|
- the script expofiles/surveyscans/2018/make.py (run by "python make.py")
|
|
|
|
the "make.py" script creates index.html files in each folder /2018/2018#nn/
|
|
and creates or updates a webpage for each person listed in any of the contents.json files
|
|
in the folder/2018/ e.g. "Becka Lawson.html".
|
|
|
|
The script make.py requires that the //loser// repo is populated on the machien that you
|
|
run the script on so that it can find the.svx files. Do not run this script on
|
|
the expo server itself as the loser repo is in the wrong place and all the .svx files
|
|
will be flagged as "not found" (Paul Fox did this once).
|
|
|
|
Before doing anything else, run make.py. This will create empty template contents.json
|
|
files in each folder.
|
|
|
|
You may need to create missing folders,e.g. I just had to create /2018/2018#30 to #32.
|
|
|
|
Every time you finish entering the data in contents.json in a folder,
|
|
run make.py to update the "person" html files and to
|
|
re-generate the index.html file for the 2018 folder as
|
|
a whole (surveyscans/2018/index.html).
|
|
|
|
There are ambiguities about how the entries in the contents.json actually lead to
|
|
reminder instructions in the html files produced, and this is particularly
|
|
difficult for electronic caves where the topo files are missing
|
|
and for surface prospecting where it is not clear which of the actions
|
|
should be done and thus which products should be produced.
|
|
|
|
This needs to be documented.
|
|
|
|
For prospecting and surface surveying it is not clear whether the default folder
|
|
for the url link should be repo //loser// surface/1623/allplateau.svx
|
|
|
|
When there are more than one .svx file there seems to be no way of recording the list
|
|
in contents.json so it is impossible to tell what was done on that trip or whether
|
|
there is anything missing. This is especially true if it was electronic and the
|
|
.topo files are missing. Wookey confirms that this is the case.
|
|
|
|
NAMING
|
|
The script detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and
|
|
produces a reminder/warning if they are not,even if these have all been scanned
|
|
and given different names.
|
|
|
|
The job of the checker (perhaps on a second pass) if to rename files so that these
|
|
warnings disappear.
|
|
|
|
HINT
|
|
When there are a lot on wallets all with the same cave, make your own template
|
|
with the cave name and the right folder prefix for the svx folder
|
|
(in the loser repo) and copy it in to all those wallet folders - overwriting
|
|
the blank template produced by the make.py
|
|
|
|
|
|
|
|
|
|
</pre>
|
|
|
|
|
|
<hr />
|
|
|
|
</body>
|
|
</html>
|