2018-08-14 20:39:34 +01:00
< html >
< head >
< title > Survey Handbook - online wallet< / title >
< link rel = "stylesheet" type = "text/css" href = "../../css/main2.css" / >
< / head >
< body >
< h2 id = "tophead" > CUCC Expedition Handbook< / h2 >
< h1 > Maintaining the online wallets< / h1 >
2018-08-15 21:44:24 +01:00
< p > If you are a newcomer to the system, read the < a href = "newcave.html#onlinew" > beginners introduction to online wallets< / a > first.
< h3 > Why we have online wallets< / h3 >
< p > There are three quite different reasons:
< ol >
< li > The scans of the survey notebook pages are the ultimate original raw survey data and completely irreplaceable.
< li > The other files in the wallet are part of the process of producing a survey of the cave as a whole.
< li > Individual to-do lists are produced automatically for each caver listing what survey processing tasks they haven't finished yet.
< / ol >
< h3 > The scanned pages< / h3 >
< p > These are simply the scanned imaages (or digital photographs) of each page of the original survey notes.
They should be named < em > < span style = "font-family:monospace" > notesXXX.jpg< / span > < / em > where "XXX" can be
anything you like. Typically we have the scanned pages called notes1.jpg, notes2.jpg, notes3.jpg.
< p > It is important that you use use the .jpg (JPEG) file format, and definitely not PNG (very voluminous)
or PDF (very hard to re-use elsewhere). Set the scanner at 300 dpi and adjust the contrast of the image after scanning
by using photo-editing software to enhance the writing. Also please crop each image to just the area containing
the survey data.
< p > As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) upload the entire online wallet to the expo server in Cambridge
< span style = "font-family:monospace" > expo.survex.com< / span > . This is so that these precious files are backed-up as soon as possible.
< h3 > The other files and online index < em > < span style = "font-family:monospace" > contents.json< / span > < / em > < / h3 >
< p > All the other files are part of the multi-step process of producing the cave survey - see < a href = "newcave.html" >
Creating a new cave...< / a > for the full list of steps.
< p > We keep an index of how many of those steps have been completed in two places:
< ul >
< li > On paper, in the tick boxes in the contents list of this years' survey trips' lever-arch file in the potato hut.
< li > Online in the < em > < span style = "font-family:monospace" > contents.json< / span > < / em > file which exists in each trip online wallet.
< / ul >
< p > but the < em > < span style = "font-family:monospace" > contents.json< / span > < / em > file has another,completely different function:
it is the < b > only online record< / b > that connects the wallet number to the cave identifier. So if a future cave surveyor deperately needs
to consult the original cave survey, it can be done by, e.g. < br >
< span style = "font-family:monospace" >
grep -rl "2018-dm-07" expofiles/surveyscans
< / span > < br >
will find and list all the wallets which contain survey data for cave 2018-dm-07 (which is also known as "Homecoming Cave" and which will
have a different Austrian Kataster number issued for it in due course).
< p > The paper tick-list tracks the following steps for each online wallet:
< ul >
< li > < u > Survex< / u >
< ul >
< li > Data
< li > LRUD
< li > Description + QMs
< / ul >
< li > < u > Drawn< / u >
< ul >
< li > Plan + Xsections
< li > Elevation
< / ul >
< li > < u > Scanned< / u >
< ul >
< li > In cave notes
< li > Plan + Xsections
< li > Elevation
< / ul >
< li > Tunnel
2018-08-18 14:25:36 +01:00
< li > Online guidebook updated
2018-08-15 21:44:24 +01:00
< li > json file edited
< / ul >
< p > (where the "json file updated" step only refers to the initial editing of the json file to ensure
that it has the right people, date and cave identifier and name).
< p > A fully-populated and complete < em > < span style = "font-family:monospace" > contents.json< / span > < / em > file
looks like this:
< pre >
{
"description written": false,
"website updated": false,
"people": [
"Dickon Morris",
"Jon Arne Toft",
"Becka Lawson"],
"elev not required": false,
"cave": "2018-dm-07",
"survex not required": false,
"qms written": true,
"plan not required": false,
"electronic survey": false,
"plan drawn": true,
"date": "2018-07-13",
"elev drawn": true,
"description url": "",
"survex file": "caves-1626/2018-dm-07/2018-dm-07.svx",
"name": "Homecoming cave"
}
< / pre >
2018-08-18 14:25:36 +01:00
Yes, this is < a href = "https://en.wikipedia.org/wiki/JSON" > a programming format< / a >
(standardised in 2013) and every comma is critical.
2018-08-15 21:44:24 +01:00
< h3 > "To do" lists for every caver< / h3 >
< p > The folder containing all the wallets for the year, e.g.
< pre >
/home/expo/expofiles/surveyscans/2018/
< / pre >
will, after the appropriate magic has happened, contain a file
< pre >
index.html
< / pre >
which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them.
You can see < a href = "../../../expofiles/surveyscans/2016/index.html" > the index.html for 2016< / a >
(most of these have been done, but the index has not been updated). Also there will be a linked file for each individual
for their personal to-do list, and each online wallet contains its own
< span style = "font-family:monospace" > index.html< / span > file which describes the survey production status for all the wallets.
< p > All this magic is created by a script < span style = "font-family:monospace" > make.py< / span > .
< h3 > Setting up the online wallets< / h3 >
< p > When, at the beginning of expo, you create the folder in
< span style = "font-family:monospace" > expofiles/surveyscans/< / span > for the current year, e.g.
< span style = "font-family:monospace" > /2018/< / span > , you will copy < span style = "font-family:monospace" > make.py< / span >
from the previous year's folder. You will do this on your own laptop or on the expo laptop.
< p > You will also manually create a number of subfolders, e.g. 2018#01, 2018#02 etc. to be ready for the influx of
new trip surveys.
< p > Next you will test that the magic works: open a terminal in
< span style = "font-family:monospace" > expofiles/surveyscans/2018/< / span > and run
< pre >
python make.py
< / pre >
< p > which will work immediately without errors because you are using Linux and the
correct version of python is already installed.
< p > This will create a default < span style = "font-family:monospace" > contents.json< / span > and
< span style = "font-family:monospace" > index.html< / span > in each online wallet folder and also a
< span style = "font-family:monospace" > index.html< / span > in the /2018/ folder.
< h3 > Maintaining the online wallets< / h3 >
< p > Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the
< span style = "font-family:monospace" > contents.json< / span > file in their wallet. In your dreams.
< p > The first difficulty when editing a blank < span style = "font-family:monospace" > contents.json< / span >
for a newly-created wallet is finding out which cave the wallet describes.
The wallet may say "radaghost to blitzkriek"
(or whatever) but without the name of the cave you can't find the .svx files
as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/.
Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the
informal name of the cave,e.g. "Homecoming",instead of the identifier "2018-dm-07" you want.
< p > A regular task during expo is for a nerd to review the < span style = "font-family:monospace" > contents.json< / span > files for
recently created wallets and to check that names, dates and cave numbers are correct.
You will run
< pre >
python make.py
< / pre >
regularly, after every batch of survey data is entered or scanned.
< p > This will always overwrite all the < span style = "font-family:monospace" > index.html< / span > files but it will never touch
the < span style = "font-family:monospace" > contents.json< / span > files.
< p > You will also regularly synchronise your laptop
and the expo laptop with < span style = "font-family:monospace" > expo.survex.com/expofiles/surveyscan/2018/< / span >
and this is where it gets tricky.
< p > < span style = "font-family:monospace" > expo.survex.com/expofiles/< / span > is not under version control,
so the most recent person
to upload the contents of < span style = "font-family:monospace" > /2018/< / span > will overwrite everyone else's work.
This does not matter for the autogenerated files, but it is vital that it does not overwrite all the painfully manually edited
< span style = "font-family:monospace" > contents.json< / span > files. Which is very easy to do.
This does mean that this is one of the cases where
it is better to use < span style = "font-family:monospace" > rsync< / span > rather than an FTP client such as Filezilla.
< p > A copy of useful rsync scripts is kept in a file such as
< span style = "font-family:monospace" > expo.survex.com/expofiles/rsync2018toserver< / span > . Always run it with the -n option first,
to see what overwriting you will do.
< h3 > More < em > < span style = "font-family:monospace" > make.py< / span > < / em > magic< / h3 >
< p > The python script does more than just re-format the < span style = "font-family:monospace" > contents.json< / span > data into
different formats. It also
< ul >
< li > checks whether the .svx files listed are actually present,
< li > checks for the presence of notesXXX.jpg, planXXX.jpg and elevXXX.jpg files
< li > creates a template < span style = "font-family:monospace" > contents.json< / span > in any wallet which does not have one.
< / ul >
< p > Things it might do in future (if someone gets around to it) include: < br >
- checking the cave number specified matches the folder for the .svx file,< br >
- detecting whether there is a description or a list of QMs in the survex file,< br >
- accepting a list of .svx files and not just one (a very common thing),< br >
- checking the name of the cave against the cave number,< br >
- checking whether the website page even exists for this cave,< br >
- being more intelligent about .topo files and thus the lack of scan files,< br >
- checking the date is in the recent past etc.< br > < br >
< h3 > How < em > < span style = "font-family:monospace" > contents.json< / span > < / em > fields match
< em > < span style = "font-family:monospace" > index.html< / span > < / em > reports< / h3 >
< p >
< em > to be written...< / em >
< hr >
Old notes, being turned into real documentation...
2018-08-14 20:39:34 +01:00
< pre >
# Instructions
# 2018-08-14
# Philip Sargent
2018-08-15 21:44:24 +01:00
2018-08-14 20:39:34 +01:00
Wookey told me to sort out the contents.json files in expofiles/surveyscans/2018/
and these are my notes to remind myself what this entails.
The job is to populate the contents.json file in each folder, e.g.
expofiles/surveyscans/2018/2018#03/contents.json
using the following input materials:
- the wallet 2018#03 and the papers inside it. This is in the 2018 lever-arch file.
- the folder in repo 'loser' holding the appropriate .svx files e.g.
"caves-1623/2017-cucc-24/gshclimb.svx"
- the script expofiles/surveyscans/2018/make.py (run by "python make.py")
the "make.py" script creates index.html files in each folder /2018/2018#nn/
and creates or updates a webpage for each person listed in any of the contents.json files
in the folder/2018/ e.g. "Becka Lawson.html".
The script make.py requires that the //loser// repo is populated on the machien that you
run the script on so that it can find the.svx files. Do not run this script on
the expo server itself as the loser repo is in the wrong place and all the .svx files
will be flagged as "not found" (Paul Fox did this once).
Before doing anything else, run make.py. This will create empty template contents.json
files in each folder.
You may need to create missing folders,e.g. I just had to create /2018/2018#30 to #32.
Every time you finish entering the data in contents.json in a folder,
run make.py to update the "person" html files and to
re-generate the index.html file for the 2018 folder as
a whole (surveyscans/2018/index.html).
There are ambiguities about how the entries in the contents.json actually lead to
reminder instructions in the html files produced, and this is particularly
difficult for electronic caves where the topo files are missing
and for surface prospecting where it is not clear which of the actions
should be done and thus which products should be produced.
This needs to be documented.
For prospecting and surface surveying it is not clear whether the default folder
for the url link should be repo //loser// surface/1623/allplateau.svx
When there are more than one .svx file there seems to be no way of recording the list
in contents.json so it is impossible to tell what was done on that trip or whether
there is anything missing. This is especially true if it was electronic and the
2018-08-15 21:44:24 +01:00
.topo files are missing. Wookey confirms that this is the case.
2018-08-14 20:39:34 +01:00
NAMING
The script detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and
produces a reminder/warning if they are not,even if these have all been scanned
and given different names.
The job of the checker (perhaps on a second pass) if to rename files so that these
warnings disappear.
HINT
When there are a lot on wallets all with the same cave, make your own template
with the cave name and the right folder prefix for the svx folder
(in the loser repo) and copy it in to all those wallet folders - overwriting
the blank template produced by the make.py
< / pre >
< hr / >
< / body >
< / html >
