mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2025-12-08 14:54:28 +00:00
documenting online wallets and make.py
This commit is contained in:
@@ -96,7 +96,7 @@ survey books are divided into "Kaninchenhöhle" (usually referred to as
|
||||
"notKH survey book"). There should be an index page at the front, which you
|
||||
should also fill in so that people can find your survey again.</p>-->
|
||||
|
||||
<h3>Scan the notes into the online wallet</h3>
|
||||
<h3 id="onlinew">Scan the notes into the online wallet</h3>
|
||||
|
||||
<p>Each wallet has a corresponding folder in the online system where a record is kept
|
||||
of what information is in the wallet and where the corresponding survey data is filed:
|
||||
@@ -126,7 +126,8 @@ and tell someone nerdy when you have finished and they will
|
||||
ensure that it is copied to the expo server.
|
||||
[If you want to do this yourself, or are using your own laptop, then learn how to use Filezilla
|
||||
- as <a href="../uploading.html">documented for uploading your expo photographs</a>. The correct folder
|
||||
on the expo server is the same as that on the expo laptop- because we set up the expo laptop to be like that.]
|
||||
on the expo server is the same as that on the expo laptop- because we set up the expo laptop to be like that.
|
||||
But <em>only copy files to the server that you created yourself and which live in your own wallet folder</em>]
|
||||
|
||||
<h3 id="runsurvex">Storing your electronic survey files</h3>
|
||||
<p>If you used a PDA instead of making notes on paper, you need to store your .topo files in the right place.
|
||||
@@ -195,7 +196,8 @@ as the scanned notes, i.e. (for wallet #19) you would put them in:
|
||||
|
||||
<p>Tunnel only produces plan surveys, but they are very pretty.
|
||||
|
||||
<p>The tunnel (or therion) files should be stored in the same folder as the scanned notes.
|
||||
<p>The tunnel (or therion) files should NOT stored in the same folder as the scanned notes. They should
|
||||
be uploaded to the version control repository //tunneldata//.
|
||||
|
||||
<h3>The cave description and rigging guide</h3>
|
||||
|
||||
|
||||
@@ -6,12 +6,200 @@
|
||||
<body>
|
||||
<h2 id="tophead">CUCC Expedition Handbook</h2>
|
||||
<h1>Maintaining the online wallets</h1>
|
||||
<p>This is not the page you are looking for.
|
||||
<p>If you are a newcomer to the system, read the <a href="newcave.html#onlinew">beginners introduction to online wallets</a> first.
|
||||
|
||||
<h3>Why we have online wallets</h3>
|
||||
<p>There are three quite different reasons:
|
||||
<ol>
|
||||
<li>The scans of the survey notebook pages are the ultimate original raw survey data and completely irreplaceable.
|
||||
<li>The other files in the wallet are part of the process of producing a survey of the cave as a whole.
|
||||
<li>Individual to-do lists are produced automatically for each caver listing what survey processing tasks they haven't finished yet.
|
||||
</ol>
|
||||
|
||||
<h3>The scanned pages</h3>
|
||||
<p>These are simply the scanned imaages (or digital photographs) of each page of the original survey notes.
|
||||
They should be named <em><span style="font-family:monospace">notesXXX.jpg</span></em> where "XXX" can be
|
||||
anything you like. Typically we have the scanned pages called notes1.jpg, notes2.jpg, notes3.jpg.
|
||||
<p>It is important that you use use the .jpg (JPEG) file format, and definitely not PNG (very voluminous)
|
||||
or PDF (very hard to re-use elsewhere). Set the scanner at 300 dpi and adjust the contrast of the image after scanning
|
||||
by using photo-editing software to enhance the writing. Also please crop each image to just the area containing
|
||||
the survey data.
|
||||
<p>As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) upload the entire online wallet to the expo server in Cambridge
|
||||
<span style="font-family:monospace">expo.survex.com</span>. This is so that these precious files are backed-up as soon as possible.
|
||||
|
||||
<h3>The other files and online index <em><span style="font-family:monospace">contents.json</span></em></h3>
|
||||
<p>All the other files are part of the multi-step process of producing the cave survey - see <a href="newcave.html">
|
||||
Creating a new cave...</a> for the full list of steps.
|
||||
|
||||
<p>We keep an index of how many of those steps have been completed in two places:
|
||||
<ul>
|
||||
<li>On paper, in the tick boxes in the contents list of this years' survey trips' lever-arch file in the potato hut.
|
||||
<li>Online in the <em><span style="font-family:monospace">contents.json</span></em> file which exists in each trip online wallet.
|
||||
</ul>
|
||||
|
||||
<p>but the <em><span style="font-family:monospace">contents.json</span></em> file has another,completely different function:
|
||||
it is the <b>only online record</b> that connects the wallet number to the cave identifier. So if a future cave surveyor deperately needs
|
||||
to consult the original cave survey, it can be done by, e.g. <br>
|
||||
<span style="font-family:monospace">
|
||||
grep -rl "2018-dm-07" expofiles/surveyscans
|
||||
</span><br>
|
||||
will find and list all the wallets which contain survey data for cave 2018-dm-07 (which is also known as "Homecoming Cave" and which will
|
||||
have a different Austrian Kataster number issued for it in due course).
|
||||
|
||||
<p> The paper tick-list tracks the following steps for each online wallet:
|
||||
<ul>
|
||||
<li><u>Survex</u>
|
||||
<ul>
|
||||
<li>Data
|
||||
<li>LRUD
|
||||
<li>Description + QMs
|
||||
</ul>
|
||||
<li><u>Drawn</u>
|
||||
<ul>
|
||||
<li>Plan + Xsections
|
||||
<li>Elevation
|
||||
</ul>
|
||||
<li><u>Scanned</u>
|
||||
<ul>
|
||||
<li>In cave notes
|
||||
<li>Plan + Xsections
|
||||
<li>Elevation
|
||||
</ul>
|
||||
|
||||
<li>Tunnel
|
||||
<li>Website updated
|
||||
<li>json file edited
|
||||
</ul>
|
||||
<p>(where the "json file updated" step only refers to the initial editing of the json file to ensure
|
||||
that it has the right people, date and cave identifier and name).
|
||||
|
||||
<p>A fully-populated and complete <em><span style="font-family:monospace">contents.json</span></em> file
|
||||
looks like this:
|
||||
<pre>
|
||||
{
|
||||
"description written": false,
|
||||
"website updated": false,
|
||||
"people": [
|
||||
"Dickon Morris",
|
||||
"Jon Arne Toft",
|
||||
"Becka Lawson"],
|
||||
"elev not required": false,
|
||||
"cave": "2018-dm-07",
|
||||
"survex not required": false,
|
||||
"qms written": true,
|
||||
"plan not required": false,
|
||||
"electronic survey": false,
|
||||
"plan drawn": true,
|
||||
"date": "2018-07-13",
|
||||
"elev drawn": true,
|
||||
"description url": "",
|
||||
"survex file": "caves-1626/2018-dm-07/2018-dm-07.svx",
|
||||
"name": "Homecoming cave"
|
||||
}
|
||||
</pre>
|
||||
Yes, this is a programming format and every comma is critical.
|
||||
|
||||
<h3>"To do" lists for every caver</h3>
|
||||
<p>The folder containing all the wallets for the year, e.g.
|
||||
<pre>
|
||||
/home/expo/expofiles/surveyscans/2018/
|
||||
</pre>
|
||||
will, after the appropriate magic has happened, contain a file
|
||||
<pre>
|
||||
index.html
|
||||
</pre>
|
||||
which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them.
|
||||
You can see <a href="../../../expofiles/surveyscans/2016/index.html">the index.html for 2016</a>
|
||||
(most of these have been done, but the index has not been updated). Also there will be a linked file for each individual
|
||||
for their personal to-do list, and each online wallet contains its own
|
||||
<span style="font-family:monospace">index.html</span> file which describes the survey production status for all the wallets.
|
||||
|
||||
<p>All this magic is created by a script <span style="font-family:monospace">make.py</span>.
|
||||
|
||||
<h3>Setting up the online wallets</h3>
|
||||
<p>When, at the beginning of expo, you create the folder in
|
||||
<span style="font-family:monospace">expofiles/surveyscans/</span> for the current year, e.g.
|
||||
<span style="font-family:monospace">/2018/</span>, you will copy <span style="font-family:monospace">make.py</span>
|
||||
from the previous year's folder. You will do this on your own laptop or on the expo laptop.
|
||||
<p>You will also manually create a number of subfolders, e.g. 2018#01, 2018#02 etc. to be ready for the influx of
|
||||
new trip surveys.
|
||||
<p>Next you will test that the magic works: open a terminal in
|
||||
<span style="font-family:monospace">expofiles/surveyscans/2018/</span> and run
|
||||
<pre>
|
||||
python make.py
|
||||
</pre>
|
||||
<p>which will work immediately without errors because you are using Linux and the
|
||||
correct version of python is already installed.
|
||||
<p>This will create a default <span style="font-family:monospace">contents.json</span> and
|
||||
<span style="font-family:monospace">index.html</span> in each online wallet folder and also a
|
||||
<span style="font-family:monospace">index.html</span> in the /2018/ folder.
|
||||
|
||||
<h3>Maintaining the online wallets</h3>
|
||||
<p>Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the
|
||||
<span style="font-family:monospace">contents.json</span> file in their wallet. In your dreams.
|
||||
|
||||
<p>The first difficulty when editing a blank <span style="font-family:monospace">contents.json</span>
|
||||
for a newly-created wallet is finding out which cave the wallet describes.
|
||||
The wallet may say "radaghost to blitzkriek"
|
||||
(or whatever) but without the name of the cave you can't find the .svx files
|
||||
as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/.
|
||||
Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the
|
||||
informal name of the cave,e.g. "Homecoming",instead of the identifier "2018-dm-07" you want.
|
||||
<p>A regular task during expo is for a nerd to review the <span style="font-family:monospace">contents.json</span> files for
|
||||
recently created wallets and to check that names, dates and cave numbers are correct.
|
||||
|
||||
You will run
|
||||
<pre>
|
||||
python make.py
|
||||
</pre>
|
||||
regularly, after every batch of survey data is entered or scanned.
|
||||
<p>This will always overwrite all the <span style="font-family:monospace">index.html</span> files but it will never touch
|
||||
the <span style="font-family:monospace">contents.json</span> files.
|
||||
<p>You will also regularly synchronise your laptop
|
||||
and the expo laptop with <span style="font-family:monospace">expo.survex.com/expofiles/surveyscan/2018/</span>
|
||||
and this is where it gets tricky.
|
||||
<p><span style="font-family:monospace">expo.survex.com/expofiles/</span> is not under version control,
|
||||
so the most recent person
|
||||
to upload the contents of <span style="font-family:monospace">/2018/</span> will overwrite everyone else's work.
|
||||
This does not matter for the autogenerated files, but it is vital that it does not overwrite all the painfully manually edited
|
||||
<span style="font-family:monospace">contents.json</span> files. Which is very easy to do.
|
||||
This does mean that this is one of the cases where
|
||||
it is better to use <span style="font-family:monospace">rsync</span> rather than an FTP client such as Filezilla.
|
||||
|
||||
<p>A copy of useful rsync scripts is kept in a file such as
|
||||
<span style="font-family:monospace">expo.survex.com/expofiles/rsync2018toserver</span>. Always run it with the -n option first,
|
||||
to see what overwriting you will do.
|
||||
|
||||
<h3>More <em><span style="font-family:monospace">make.py</span></em> magic</h3>
|
||||
<p>The python script does more than just re-format the <span style="font-family:monospace">contents.json</span> data into
|
||||
different formats. It also
|
||||
<ul>
|
||||
<li>checks whether the .svx files listed are actually present,
|
||||
<li>checks for the presence of notesXXX.jpg, planXXX.jpg and elevXXX.jpg files
|
||||
<li>creates a template <span style="font-family:monospace">contents.json</span> in any wallet which does not have one.
|
||||
</ul>
|
||||
|
||||
<p>Things it might do in future (if someone gets around to it) include: <br>
|
||||
- checking the cave number specified matches the folder for the .svx file,<br>
|
||||
- detecting whether there is a description or a list of QMs in the survex file,<br>
|
||||
- accepting a list of .svx files and not just one (a very common thing),<br>
|
||||
- checking the name of the cave against the cave number,<br>
|
||||
- checking whether the website page even exists for this cave,<br>
|
||||
- being more intelligent about .topo files and thus the lack of scan files,<br>
|
||||
- checking the date is in the recent past etc.<br><br>
|
||||
|
||||
<h3>How <em><span style="font-family:monospace">contents.json</span></em> fields match
|
||||
<em><span style="font-family:monospace">index.html</span></em> reports</h3>
|
||||
<p>
|
||||
<em>to be written...</em>
|
||||
<hr>
|
||||
Old notes, being turned into real documentation...
|
||||
<pre>
|
||||
# Instructions
|
||||
# 2018-08-14
|
||||
# Philip Sargent
|
||||
|
||||
|
||||
Wookey told me to sort out the contents.json files in expofiles/surveyscans/2018/
|
||||
and these are my notes to remind myself what this entails.
|
||||
|
||||
@@ -44,16 +232,6 @@ run make.py to update the "person" html files and to
|
||||
re-generate the index.html file for the 2018 folder as
|
||||
a whole (surveyscans/2018/index.html).
|
||||
|
||||
But make.py does not delete old files which are not regenerated.
|
||||
So it is a good idea to delete all the .html files
|
||||
before running it.
|
||||
|
||||
The first difficulty is finding out which cave a wallet describes..
|
||||
The wallet may say "radaghost to blitzkriek"
|
||||
(or whatever) but without the name of the cave you can't find the .svx files
|
||||
as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/
|
||||
Usually this is written by hand on the label of the wallet.
|
||||
|
||||
There are ambiguities about how the entries in the contents.json actually lead to
|
||||
reminder instructions in the html files produced, and this is particularly
|
||||
difficult for electronic caves where the topo files are missing
|
||||
@@ -68,14 +246,7 @@ for the url link should be repo //loser// surface/1623/allplateau.svx
|
||||
When there are more than one .svx file there seems to be no way of recording the list
|
||||
in contents.json so it is impossible to tell what was done on that trip or whether
|
||||
there is anything missing. This is especially true if it was electronic and the
|
||||
.topo files are missing. Wookey confirms that this ios the case.
|
||||
|
||||
make.py will not overwrite any contents.json file, it only creates one where one
|
||||
doesn't exist.
|
||||
|
||||
make.py also creates an index.html file inside each wallet folder dewcribing the status of this
|
||||
folder. This is linked from the index.html file created in /2018/ by the script which
|
||||
summarises everything for this year.
|
||||
.topo files are missing. Wookey confirms that this is the case.
|
||||
|
||||
NAMING
|
||||
The script detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and
|
||||
@@ -95,7 +266,6 @@ the blank template produced by the make.py
|
||||
|
||||
|
||||
</pre>
|
||||
<p>This will be replaced with the information you want as soon as someone gets around to writing it. Why not find out how to do this yourself ?
|
||||
|
||||
|
||||
<hr />
|
||||
|
||||
Reference in New Issue
Block a user