expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-25 00:32:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
fbb5da96b9
expoweb/handbook/survey/onlinewallet.html
Philip Sargent af7e727dc5 handbook edits to better describe new wallets system
2022-10-11 00:30:47 +03:00

299 lines
15 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Survey Handbook - online wallet</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Maintaining the online wallets</h1>
<p>If you are a newcomer to the system, read the <a href="newwallet.html#onlinew">beginner's introduction to online wallets</a> first.
<style>figure {font-weight: bold; font-size: small; font-family: sans-serif;font-variant-caps: small-caps;}</style>
<div class="onright">
<figure>
<a href="../i/qm-image.jpg">
<img src="../i/qm-image.jpg"></a>
<figcaption style="font-variant-caps: small-caps;">
<em>Original notes</em>
</figcaption>
</figure>
</div>
<h2>Why we have online wallets</h2>
<p>There are three quite different reasons:
<ol>
<li>The scans of the survey notebook pages are the ultimate original raw survey data and completely irreplaceable.
<li>The other files in the wallet are part of the process of producing a survey of the cave as a whole.
<li>Individual online to-do lists are produced automatically for each caver, each cave, and each year. These
have a colour-coded table showing what survey processing tasks they haven't finished yet.
</ol>
<h2>You need to know this bit</h2>
<p>All this scanned information is in <span style="font-family:monospace">expo.survex.com/expofiles/</span> which is <font color=red><b>not under
version control</b></font>,
so the most recent person
to change anything <font color=red><b>can overwrite everyone else's work</b></font>.
<h3>What's in the online wallet</h3>
<h4>scans in the expofiles/surveyscans/</h4>
<ul>
<li>The <a href="#scans">scanned images</a> of survey notes
</ul>
<h4>metadata in the version control system (drawings/walletjson/)</h4>
<ul>
<li><a href="#status">Progress status</a> of drawing up the survey
<li>Links to <a href="#derived">derived data</a>: survex files, survey drawings
</ul>
<h3 id="scans">The scanned images</h3>
<p>These are simply the scanned images (or digital photographs) of each page of the original survey notes.
They should be named <em><span style="font-family:monospace">notesXXX.jpg</span></em> where "XXX" can be
anything you like. In the past we have had scanned pages called notes1.jpg, notes2.jpg, notes3.jpg; but <em>don't
do this</em>, instead give descriptive names, e.g. <span style="font-family:monospace">notes-twentyseconddropdronesurvey.jpg</span> or whatever,
but do put "notes-.." at the beginning or "..-notes" at the end of the filename (or "plan-.." or "elev-.." as appropriate).
<p>It is important that you use use the .jpg (JPEG) file format, and definitely not PNG (very voluminous)
or PDF (very hard to re-use elsewhere). Set the scanner at 300 dpi and adjust the contrast of the image after scanning
by using photo-editing software to enhance the writing. Also please crop each image to just the area containing
the survey data.
<p>As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) check that the entire online wallet is on the expo server at
<span style="font-family:monospace">expo.survex.com</span> using the <a href="/scanupload/2019:01">Upload Scans</a> form.
This is so that these precious files are backed-up as soon as possible.
<p>(If you want to move things around, then assuming you have done the
<a href="../computing/keyexchange.html">key-pair procedure</a>, do the upload by <a href="../uploading.html">sFTP</a>
as you do for photos, but to the directory described in the <a href="newwallet.html">introduction to online wallets</a>.)
<p>The <em><span style="font-family:monospace">notesXXX.jpg</span></em> files need to be at moderately high resolution but the plan and
elevation files are usually fine at 200 dpi. So if the caver has scanned these at high resolution you can reduce the size of these files
without damange.
<div class="onright">
<figure>
<a href="../i/scan-upload.jpg">
<img src="../t/scan-upload.jpg"></a>
<figcaption style="font-variant-caps: small-caps;">
<em>Wallet data on Scan Upload (click to enlarge)</em>
</figcaption>
</figure>
</div>
<h3 id="status">Progress status</h3>
<p>The illustration shows the lower half of the <a href="/scanupload/2019:01">Upload Scans</a> form.
This is the information about the wallet itself (this is called 'metadata').
<p>Once you are logged on as user 'expo', you can edit all the data. If you use the right runes for the survex files (as shown in the image),
a clickable-list of the survex files will appear on the upper-left which will take you directly to the survex files - which you can edit
there online too.
<p>There is only one absolutely required field: the date. Unless you specify a date the form will not save your data and the "Submit changes"
button won't work.
<h3 id="derived">Links to derived data</h3>
<p>These links are not actually in the wallet, they are in the survex files, Tunnel files and Therion files.
The links are simply <b>the name of the wallet</b>, e.g. <var>2019#01</var>. So once a wallet has been used, i.e. once you have typed up the first
survex file, you should not rename it or move the contents (unless you can find and change all the files referring to it).
<h2>How this all actually works</h2>
<h3>The other files and online index <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span></h3>
<p>All the other files are part of the multi-step process of producing the cave survey - see <a href="newcave.html">
Creating a new cave...</a> for the full list of steps.
<p>We keep an index of how many of those steps have been completed in two places:
<ul>
<li>On paper, in the tick boxes in the contents list of this years' survey trips' lever-arch file in the potato hut.
<li>Online in the <em><span style="font-family:monospace">contents.json</span></em> file, a copy of which exists in the version control system.
</ul>
<p>but the <em><span style="font-family:monospace">contents.json</span></em> file has another,completely different function:
it may be the <b>only online record</b> that connects the wallet number to the cave identifier. So if a future cave surveyor desperately needs
to consult the original cave survey, it can be done by, e.g. <br>
<a href="/cave/scans/1623-264">/cave/scans/1623-264</a><br>
will find and list all the wallets which contain survey data for cave 1623-264.
<p>The link between a .svx file and the wallet should also be recorded within the .svx file itself using the "*ref:" field, e.g.
<pre>
*ref 2018#06
; the #number is on the clear pocket containing the original notes
</pre>
But sometime in mid-Expo 2015 everyone stopped using the survex template file and copied from a friend instead,
and so this information was not properly recorded. This will be fixed by hand-editing in due course.
(Note that many old .svx files were processed with an older version of survex which did not support this feature and so a comment was used instead.)
<h2>Much more detail - more than you want</h2>
<p>Troggle produces very useful auto-generated reports of the status of the wallets and the survex files
<ul>
<li><a href="http://expo.survex.com/survey_scans/">List of all wallets</a> [SLOW to load] and the survex files produced from them (incomplete due to poor data entry especially since 2015)
<li><a href="http://expo.survex.com/expedition/2018">List of all trips and survex files</a> (scroll down to the bottom of the page) in any one year - includes a link to the logbook fragment for the relevant day
<li><a href="http://expo.survex.com/personexpedition/MichaelSargent/2018">List of trips and surveys</a> by a particular person in a particular year (nothing to do with wallets but added here for completeness).
<li> <a href="/wallets/year/2022">Report on wallets status for 2022</a>
<li><a href="/cave/scans/1623-264">Report on wallets status for cave 1623-264</a>
</ul></ul>
These troggle reports are invaluable for finding data entry errors or other mistakes.
<p> The paper tick-list tracks the following steps for each online wallet:
<ul>
<li><u>Survex</u>
<ul>
<li>Data
<li>LRUD
<li>Description + QMs
</ul>
<li><u>Drawn</u>
<ul>
<li>Plan + Xsections
<li>Elevation
</ul>
<li><u>Scanned</u>
<ul>
<li>In cave notes
<li>Plan + Xsections
<li>Elevation
</ul>
<li>Tunnel
<li>Online guidebook updated
<li>json file edited
</ul>
<p>(where the "json file updated" step only refers to the initial editing of the json file to ensure
that it has the right people, date and cave identifier and name).
<p>A fully-populated and complete <em><span style="font-family:monospace">contents.json</span></em> file (this is the 'metadata')
looks like this:
<pre>
{
"description written": false,
"website updated": false,
"people": [
"Dickon Morris",
"Jon Arne Toft",
"Becka Lawson"],
"elev not required": false,
"cave": "1626-359",
"survex not required": false,
"qms written": true,
"plan not required": false,
"electronic survey": false,
"plan drawn": true,
"date": "2018-07-13",
"elev drawn": true,
"description url": "",
"survex file": "['caves-1626/2018-dm-07/2018-dm-07.svx', 'caves-1626/359/359.svx'] ",
"name": "Homecoming cave"
}
</pre>
Yes, this is <a href="https://en.wikipedia.org/wiki/JSON">a programming format</a>
(standardised in 2013) and every comma is critical.
(Though you can miss off the ".svx" of the survex filenames in the list and it will cope.)
<p>If you feel an urge to edit the file directly, instead of using the online form and checkboxes,
you should take great care with the format and also with also updating the version-controlled copy of
<em><span style="font-family:monospace">contents.json</span></em>
which is committed in the drawings repository. [The drawings repo copy is now, August 2022, the master and only copy.]
<p>
When entering people's names, using the form or with a text editor, it is important not to use any funny characters (such as "?") because
peoples names here are used by the software to construct filenames for the surveying to-do lists. And "?" (for instance) is illegal
in filenames on Windows computers.
<h3>"To do" lists for every caver</h3>
<p>You can get a summary of the outstanding 'tick list' for each individual caver at, e.g.
<ul>
<li><a href="/wallets/person/CorinDonne">/wallets/person/CorinDonne</a>
</ul>
<p>
which lists all the wallets which have uncompleted tasks in a table of coloured boxes showing the status as "red" or "green".
<p>If you want to see what the files are in each online wallet then look at <br>
<a href="http://expo.survex.com/expofiles/surveyscans/2018/">expo.survex.com/expofiles/surveyscans/2018/</a> <br>
and you can navigate to the individual wallet folders too and look directly at the files there. Or you can navigate using
the wallet upload form, which similarly lists the files in each wallet but also reports
on the progress data for the wallet.
<h3>Setting up the online wallets</h3>
At the beginning of a new expo you do not need to do any setup for new wallets. The necessary folders etc. will be created automatically
as soon as you upload a new scan for a new year.
<h3>Maintaining the online wallets</h3>
<p>Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the 'metadata' (in the
<span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> file) using the tick-boxes on the upload form. In your dreams.
<p>The first difficulty when editing the data
for a newly-created wallet is finding out which cave the wallet describes.
The label on the plastic wallet may say "radaghost to blitzkriek"
(or whatever) but without the name of the cave you can't find the .svx files
as you don't know that you need to look in e.g. loser/caves-1626/2022-W-04/.
Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the
informal name of the cave,e.g. "BrokenNeckHole",instead of the identifier "1623/2022-W-04" you want.
<p>A regular task during expo is for a nerd to review the files and tick-list data for
recently created wallets and to check that names, dates and cave numbers are correct.
<h4>Naming the included files</h4>
<p>Troggle detects if there are notesX.jpg planX.jpg and elevX.jpg files present, and
produces a reminder/warning if they are not,even if these have all been scanned
and given different names.
<p>
The job of the checker (perhaps on a second pass) if to rename files so that these
warnings disappear. But if tunnel or therion files have already been produced, then don't rename anything.
<h4>Useful rsync scripts</h4>
<p>A copy of useful rsync scripts is kept in a file such as
<span style="font-family:monospace; size=x-small; background-color: lightgray">expo.survex.com/expofiles/rsync2018toserver</span>. Always run it with the -n option first,
to see what overwriting you will do. These snippets update the server with your recent changes on your own laptop and vice versa, but these days (2022) it is probably easier
to do everything online to the server, and occasionally make a copy to your local machine to keep it synchronised - if you have a laptop you want to use for this sort of thing.
<h4>Surface trips</h4>
<p>For prospecting and surface surveying it is not clear whether the default folder
for the url link to a survex file should be repo :loser: surface/1623/allplateau.svx
<h3>More Troggle magic</h3>
<p> Troggle also
<ul>
<li>checks whether the .svx files listed are actually present in the <a href="../computing/repos.html">:loser: repository</a>
<li>checks for the presence of notesXXX.jpg, planXXX.jpg and elevXXX.jpg files
<li>checks for the presence of XXXnotes.jpg, XXXplan.jpg, XXXelevation.jpg and XXXelev.jpg files
<li>creates helpful URL links to the existing online survey documentation for the cave being surveyed
<li>creates helpful URL links to the working files you are managing on your own laptop
</ul>
<p>Things it might do in future (if someone gets around to it) include: <br>
- checking the cave number specified matches the folder for the .svx file,<br>
- checking that the *ref: filed in the survex file is the same as the wallet name<br>
- being more intelligent about .topo files and thus the lack of scan files,<br>
- checking the date is in the recent past etc.<br><br>
<hr>
The old wallets.py script runs without errors on earliers years before 1999 when there was not the same wallet system, but the results are less useful, e.g. see
http://expo.survex.com/expofiles/surveyscans/1999/
and compare with
http://expo.survex.com/wallets/year/1999
<p>
Old notes, being turned into real documentation...
<br>Much of this just needs deleting..
<pre>
# Update March 2019
For 2014 and earlier one needs to do a lot more data entry. The contents.json files for 2014 and earlier do not say who the
people were on the trip. So we would need to work from the svx files (where they contain the *ref: wallet ID),
original plastic wallets (and the scanned drawings and notes - which are incomplete) to enter that data.
This is made much easier by the troggle reports
http://expo.survex.com/survey_scans/
http://expo.survex.com/expedition/2014
http://expo.survex.com/survey/2018%2330
This is probably not worth doing except maybe for specific critical connections.
</pre>
<hr />
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink