mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2024-11-25 16:52:00 +00:00
update wallets documn
This commit is contained in:
Philip Sargent
parent
40ef76048e
commit
2d4f61169e
@ -122,12 +122,18 @@ should also fill in so that people can find your survey again.</p>-->
|
||||
<p>As well as looking through the other plastic wallets in this year's lever-arch binder, have a look online at
|
||||
the <a href="/survey_scans/">list of all online wallets</a> and have a look at all the folders referred to
|
||||
in your cave in, e.g. here is the data for <a href="/survexfile/204">cave 204</a>: the wallet identifier is
|
||||
in the right-most column titles "Scans". If you are doing an entirely new cave, then this won't exist yet but it
|
||||
in the right-most column titles "Scans". (But this is not all the wallets, only those from which a survex file has been created).
|
||||
If you are doing an entirely new cave, then this page won't exist yet but it
|
||||
will be where your new discovery is eventually published.
|
||||
|
||||
<p>There are also lists of the wallets per year, per cave and per person (surveyor),
|
||||
e.g. see <a href="/wallets/year/2018">all the wallets for 2018</a>.
|
||||
|
||||
<p>As well as following through the links on those pages, have a look at the
|
||||
<a href="/dwgfiles">drawings files page</a> which lists all the Tunnel and Therion files, and associated with each, the wallets and
|
||||
their the scanned notes, plan and elevation scans ("Scan files" column).
|
||||
|
||||
|
||||
<h3 id="onlinew">Scan the notes into the online wallet</h3>
|
||||
|
||||
<style>figure {font-weight: bold; font-size: small; font-family: sans-serif;font-variant-caps: small-caps;}</style>
|
||||
|
||||
@ -31,7 +31,7 @@
|
||||
</ol>
|
||||
|
||||
<h2>You need to know this bit</h2>
|
||||
<p>All this information is in <span style="font-family:monospace">expo.survex.com/expofiles/</span> which is <font color=red><b>not under
|
||||
<p>All this scanned information is in <span style="font-family:monospace">expo.survex.com/expofiles/</span> which is <font color=red><b>not under
|
||||
version control</b></font>,
|
||||
so the most recent person
|
||||
to change anything <font color=red><b>can overwrite everyone else's work</b></font>.
|
||||
@ -96,17 +96,20 @@ Creating a new cave...</a> for the full list of steps.
|
||||
<p>We keep an index of how many of those steps have been completed in two places:
|
||||
<ul>
|
||||
<li>On paper, in the tick boxes in the contents list of this years' survey trips' lever-arch file in the potato hut.
|
||||
<li>Online in the <em><span style="font-family:monospace">contents.json</span></em> file which exists in each trip online wallet.
|
||||
<li>Online in the <em><span style="font-family:monospace">contents.json</span></em> file, a copy of which exists in each trip online wallet.
|
||||
</ul>
|
||||
|
||||
<p>but the <em><span style="font-family:monospace">contents.json</span></em> file has another,completely different function:
|
||||
it may be the <b>only online record</b> that connects the wallet number to the cave identifier. So if a future cave surveyor desperately needs
|
||||
to consult the original cave survey, it can be done by, e.g. <br>
|
||||
<span style="font-family:monospace">
|
||||
grep -rl "2018-dm-07" expofiles/surveyscans
|
||||
grep -rl "2001-12" drawings/walletjson
|
||||
</span><br>
|
||||
will find and list all the wallets which contain survey data for cave 2018-dm-07 (which is also known as "Homecoming Cave" and which will
|
||||
have a different Austrian Kataster number issued for it in due course).
|
||||
or<br>
|
||||
<span style="font-family:monospace">
|
||||
grep -rl "234" expofiles/surveyscans
|
||||
</span><br>
|
||||
will find and list all the wallets which contain survey data for cave 1623-234 .
|
||||
|
||||
<p>The link between a .svx file and the wallet should also be recorded in the .svx file itself using the "*ref:" field, e.g.
|
||||
<pre>
|
||||
@ -123,7 +126,9 @@ and so this information was not properly recorded. This will be fixed by hand-ed
|
||||
<li><a href="http://expo.survex.com/survey_scans/">List of all wallets</a> [SLOW to load] and the survex files produced from them (incomplete due to poor data entry especially since 2015)
|
||||
<li><a href="http://expo.survex.com/expedition/2018">List of all trips and survex files</a> (scroll down to the bottom of the page) in any one year - includes a link to the logbook fragment for the relevant day
|
||||
<li><a href="http://expo.survex.com/personexpedition/MichaelSargent/2018">List of trips and surveys</a> by a particular person in a particular year (nothing to do with wallets but added here for completeness).
|
||||
</ul>
|
||||
<li> <a href="/wallets/year/2019">Report on wallets status for 2019</a>
|
||||
<li><a href="/cave/scans/1623-264">Report on wallets status for cave 1623-264</a>
|
||||
</ul></ul>
|
||||
These troggle reports are invaluable for finding data entry errors or other mistakes.
|
||||
|
||||
<p> The paper tick-list tracks the following steps for each online wallet:
|
||||
@ -180,138 +185,88 @@ looks like this:
|
||||
Yes, this is <a href="https://en.wikipedia.org/wiki/JSON">a programming format</a>
|
||||
(standardised in 2013) and every comma is critical.
|
||||
|
||||
<p>When entering people's names it is important not to use any funny characters (such as "?") because
|
||||
<p>If you feel an urge to edit the file directly, instead of using the online form and checkboxes,
|
||||
you should take great care with the format and also with also updating the version-controlled copy of
|
||||
<em><span style="font-family:monospace">contents.json</span></em>
|
||||
which is committed in the drawings repository. [The drawings repo copy will beocme the master copy
|
||||
sometime during 2022 and the <em><span style="font-family:monospace">contents.json</span></em> in the same
|
||||
folder as all the scans will be removed.]
|
||||
<p>
|
||||
When entering people's names, using the form or with a text editor, it is important not to use any funny characters (such as "?") because
|
||||
peoples names here are used by the software to construct filenames for the surveying to-do lists. And "?" (for instance) is illegal
|
||||
in filenames on Windows computers.
|
||||
|
||||
<h3>"To do" lists for every caver</h3>
|
||||
<p>The folder containing all the wallets for the year, e.g.
|
||||
<pre>
|
||||
/home/expo/expofiles/surveyscans/2018/
|
||||
</pre>
|
||||
will, after the appropriate magic has happened, contain a file
|
||||
<p>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">walletindex.html</span>
|
||||
<p>
|
||||
which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them.
|
||||
You can see <a href="http://expo.survex.com/expofiles/surveyscans/2018/walletindex.html">the walletindex.html for 2018</a> . Also there will be a linked file for each individual
|
||||
for their personal to-do list, and each online wallet contains its own
|
||||
<span style="font-family:monospace">index.html</span> file which describes the survey production status for all the wallets.
|
||||
<p>You can get a summary of the outstanding 'tick list' for each individual caver at, e.g.
|
||||
<ul>
|
||||
<li><a href="/wallets/person/CorinDonne">/wallets/person/CorinDonne</a>
|
||||
</ul>
|
||||
|
||||
<p>The magic also creates walletindex.html files in each folder /2018/2018#nn/
|
||||
and creates or updates a webpage for each person listed in any of the contents.json files
|
||||
in the folder/2018/ e.g. <a href="http://expo.survex.com/expofiles/surveyscans/2018/Becka%20Lawson.html">Becka Lawson.html</a>.
|
||||
<p>
|
||||
which lists all the wallets which have uncompleted tasks in a table of coloured boxes showing the status as "red" or "green".
|
||||
|
||||
<p>If you want to see what the files are in each online wallet then look at <br>
|
||||
<a href="http://expo.survex.com/expofiles/surveyscans/2018/">expo.survex.com/expofiles/surveyscans/2018/</a> <br>
|
||||
and you can navigate to the individual wallet folders too and look directly at the files there.
|
||||
|
||||
<p>All this magic is created by a script <span style="font-family:monospace; size=x-small; background-color: lightgray">wallets.py</span>.
|
||||
and you can navigate to the individual wallet folders too and look directly at the files there. Or you can navigate using
|
||||
the wallet upload form, which similarly lists the files in each wallet but also reports
|
||||
on the progress data for the wallet.
|
||||
|
||||
<h3>Setting up the online wallets</h3>
|
||||
<p>When, at the beginning of expo, you create the folder in
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">expofiles/surveyscans/</span> for the current year, e.g.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">/2019/</span>, you will copy <span style="font-family:monospace">wallets.py</span>
|
||||
from the previous year's folder. You will do this on your own laptop or on the expo laptop.
|
||||
<p>You will also manually create a number of subfolders, e.g. 2019#01, 2019#02 etc. to be ready for the influx of
|
||||
new trip surveys.
|
||||
<p>Next you will test that the magic works: open a terminal in
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">expofiles/surveyscans/2018/</span> and run
|
||||
<p>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">python wallets.py</span>
|
||||
<br>
|
||||
<p>This will create a default <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> and
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">walletindex.html</span> in each online wallet subfolder and also a
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">walletindex.html</span> in the <span style="font-family:monospace; size=x-small; background-color: lightgray">/2018/</span> folder.
|
||||
|
||||
<p>This python3 script works fine on Linux (Debian, Xubuntu, etc.) and also now works fine in the <a href="https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/">Windows 10 bash system</a>.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">/2019/</span>. You will do this on your own laptop or on the expo laptop.
|
||||
<p>You may also manually create a number of subfolders, e.g. 2019#01, 2019#02 etc. to be ready for the influx of
|
||||
new trip surveys, though this is absolutely not necessary as the folders are created as-needed by the upload form.
|
||||
|
||||
<h3>Maintaining the online wallets</h3>
|
||||
<p>Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> file in their wallet. In your dreams.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> data in their wallet using the tick-boxes on the upload form. In your dreams.
|
||||
|
||||
<p>The first difficulty when editing a blank <span style="font-family:monospace">contents.json</span>
|
||||
<p>The first difficulty when editing the data
|
||||
for a newly-created wallet is finding out which cave the wallet describes.
|
||||
The lable on the plastic wallet may say "radaghost to blitzkriek"
|
||||
The label on the plastic wallet may say "radaghost to blitzkriek"
|
||||
(or whatever) but without the name of the cave you can't find the .svx files
|
||||
as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/.
|
||||
as you don't know that you need to look in e.g. loser/caves-1626/2022-W-04/.
|
||||
Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the
|
||||
informal name of the cave,e.g. "Homecoming",instead of the identifier "2018-dm-07" you want.
|
||||
<p>A regular task during expo is for a nerd to review the <span style="font-family:monospace">contents.json</span> files for
|
||||
informal name of the cave,e.g. "BrokenNeckHole",instead of the identifier "1623/2022-W-04" you want.
|
||||
<p>A regular task during expo is for a nerd to review the files and tick-list data for
|
||||
recently created wallets and to check that names, dates and cave numbers are correct.
|
||||
|
||||
You will run
|
||||
<p>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">python wallets.py</span>
|
||||
<p>
|
||||
regularly, after every batch of survey data is entered or scanned.
|
||||
<p>This will always overwrite all the <span style="font-family:monospace">walletindex.html</span> files but it will never touch
|
||||
the <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> files.
|
||||
<p>You will also regularly synchronise your laptop
|
||||
and the expo laptop with <br />
|
||||
<span style="font-family:monospace">expo.survex.com/expofiles/surveyscan/2018/</span>
|
||||
<br />and this is where it gets tricky.
|
||||
<p><span style="font-family:monospace">expo.survex.com/expofiles/</span> is <font color=red><b>not under version control</b></font>,
|
||||
so the most recent person
|
||||
to upload the contents of <span style="font-family:monospace">/2018/</span> <font color=red><b>will overwrite everyone else's work</b></font>.
|
||||
This does not matter for the autogenerated files, but it is vital that it does not overwrite all the painfully manually edited
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> files. Which is very easy to do.
|
||||
This does mean that this is one of the cases where
|
||||
it may be better to use <span style="font-family:monospace; size=x-small; background-color: lightgray">rsync</span> rather than an FTP client such as Filezilla.
|
||||
|
||||
<h4>Naming the included files</h4>
|
||||
<p>The script detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and
|
||||
<p>Troggle detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and
|
||||
produces a reminder/warning if they are not,even if these have all been scanned
|
||||
and given different names.
|
||||
<p>
|
||||
The job of the checker (perhaps on a second pass) if to rename files so that these
|
||||
warnings disappear. But if tunnel or therion files have already been produce don't rename anything.
|
||||
warnings disappear. But if tunnel or therion files have already been produced, then don't rename anything.
|
||||
|
||||
|
||||
<h4>Not under version control</h4>
|
||||
<p>
|
||||
As all this is not under version control the timestamps of the files are really quite important in figuring things out when someone makes an update mistake.
|
||||
<p>
|
||||
So script <span style="font-family:monospace; size=x-small; background-color: lightgray">wallets.py</span> has been fixed so that
|
||||
<ul>
|
||||
<li>the generated <span style="font-family:monospace; size=x-small; background-color: lightgray">walletindex.html</span> file in each wallet folder is given the same timestamp as the <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> file there, <em>not the time of when the script is run</em>. This is unusual but intentional and in practice very helpful.
|
||||
<li>the script no longer overwrites the <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> files every time it runs. It only changes that file's timestamp if it actually changes anything in the contents.json file.
|
||||
</ul>
|
||||
|
||||
<h4>Useful rsync scripts</h4>
|
||||
<p>A copy of useful rsync scripts is kept in a file such as
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">expo.survex.com/expofiles/rsync2018toserver</span>. Always run it with the -n option first,
|
||||
to see what overwriting you will do.
|
||||
to see what overwriting you will do. These snippets update the server with your recent changes on your own laptop and vice versa, but these days (2022) it is probably easier
|
||||
to do everything online to the server, and occasionally make a copy to your local machine to keep it synchronised - if you have a laptop you want to use for this sort of thing.
|
||||
|
||||
<h3>More <em><span style="font-family:monospace; size=x-small; background-color: lightgray">wallets.py</span></em> magic</h3>
|
||||
<p>The python script does more than just re-format the <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> data into
|
||||
different formats. It also
|
||||
<h3>More Troggle magic</h3>
|
||||
<p> Troggle also
|
||||
<ul>
|
||||
<li>checks whether the .svx files listed are actually present <br>in the <a href="../computing/repos.html">:loser: repository</a>
|
||||
<li>checks for the presence of notesXXX.jpg, planXXX.jpg and elevXXX.jpg files
|
||||
<li>checks for the presence of XXXnotes.jpg, XXXplan.jpg and XXXelev.jpg files
|
||||
<li>creates a template <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> in any wallet which does not have one.
|
||||
<li>checks for the presence of XXXnotes.jpg, XXXplan.jpg, XXXelevation.jpg and XXXelev.jpg files
|
||||
<li>creates helpful URL links to the existing online survey documentation for the cave being surveyed
|
||||
<li>creates helpful URL links to the working files you are managing on your own laptop
|
||||
<li>Inserts the date & time into each generated file so that anyone can see when it was last updated
|
||||
</ul>
|
||||
|
||||
<p>Things it might do in future (if someone gets around to it) include: <br>
|
||||
- checking the cave number specified matches the folder for the .svx file,<br>
|
||||
- checking that the *ref: filed in the survex file is the same as the wallet name<br>
|
||||
- detecting whether there is a description or a list of QMs in the survex file,<br>
|
||||
- accepting a list of .svx files and not just one (a very common thing),<br>
|
||||
- checking the name of the cave against the cave number,<br>
|
||||
- checking whether the website page even exists for this cave,<br>
|
||||
- being more intelligent about .topo files and thus the lack of scan files,<br>
|
||||
- checking the date is in the recent past etc.<br><br>
|
||||
|
||||
<h3>How <em><span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span></em> fields match
|
||||
<em><span style="font-family:monospace; size=x-small; background-color: lightgray">walletindex.html</span></em> reports</h3>
|
||||
<p>
|
||||
<em>to be written...</em>
|
||||
<hr>
|
||||
Old notes, being turned into real documentation...
|
||||
<br>Much of this just needs deleting..
|
||||
<pre>
|
||||
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user