mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2024-11-21 23:01:55 +00:00
Updatin the wallets.py documentaiton
This commit is contained in:
parent
3a327e3572
commit
c39d4da002
@ -46,7 +46,7 @@ actually do these things.
|
||||
(This will mean copying the passage descriptions from the survex files.)
|
||||
<li>Update the index tick boxes on paper: as your wallet progresses through this process
|
||||
<li>Update the online record of those tick boxes
|
||||
<li>Regenerate the <a href="/expofiles/surveyscans/2018/">list of outstanding survey tasks</a> for everyone
|
||||
<li>Regenerate the <a href="http://expo.survex.com/expofiles/surveyscans/2018/">list of outstanding survey tasks</a> for everyone
|
||||
</ul>
|
||||
<p>This documentation assumes that you have recorded your survey data in
|
||||
a waterproof paper notebook. If instead you are using a PDA to record the survey readings
|
||||
|
||||
@ -27,7 +27,7 @@ the survey data.
|
||||
<p>As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) upload the entire online wallet to the expo server in Cambridge
|
||||
<span style="font-family:monospace">expo.survex.com</span>. This is so that these precious files are backed-up as soon as possible.
|
||||
|
||||
<h3>The other files and online index <em><span style="font-family:monospace">contents.json</span></em></h3>
|
||||
<h3>The other files and online index <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span></h3>
|
||||
<p>All the other files are part of the multi-step process of producing the cave survey - see <a href="newcave.html">
|
||||
Creating a new cave...</a> for the full list of steps.
|
||||
|
||||
@ -76,26 +76,26 @@ that it has the right people, date and cave identifier and name).
|
||||
<p>A fully-populated and complete <em><span style="font-family:monospace">contents.json</span></em> file
|
||||
looks like this:
|
||||
<pre>
|
||||
{
|
||||
"description written": false,
|
||||
"website updated": false,
|
||||
"people": [
|
||||
"Dickon Morris",
|
||||
"Jon Arne Toft",
|
||||
"Becka Lawson"],
|
||||
"elev not required": false,
|
||||
"cave": "2018-dm-07",
|
||||
"survex not required": false,
|
||||
"qms written": true,
|
||||
"plan not required": false,
|
||||
"electronic survey": false,
|
||||
"plan drawn": true,
|
||||
"date": "2018-07-13",
|
||||
"elev drawn": true,
|
||||
"description url": "",
|
||||
"survex file": "caves-1626/2018-dm-07/2018-dm-07.svx",
|
||||
"name": "Homecoming cave"
|
||||
}
|
||||
{
|
||||
"description written": false,
|
||||
"website updated": false,
|
||||
"people": [
|
||||
"Dickon Morris",
|
||||
"Jon Arne Toft",
|
||||
"Becka Lawson"],
|
||||
"elev not required": false,
|
||||
"cave": "2018-dm-07",
|
||||
"survex not required": false,
|
||||
"qms written": true,
|
||||
"plan not required": false,
|
||||
"electronic survey": false,
|
||||
"plan drawn": true,
|
||||
"date": "2018-07-13",
|
||||
"elev drawn": true,
|
||||
"description url": "",
|
||||
"survex file": "caves-1626/2018-dm-07/2018-dm-07.svx",
|
||||
"name": "Homecoming cave"
|
||||
}
|
||||
</pre>
|
||||
Yes, this is <a href="https://en.wikipedia.org/wiki/JSON">a programming format</a>
|
||||
(standardised in 2013) and every comma is critical.
|
||||
@ -106,42 +106,45 @@ Yes, this is <a href="https://en.wikipedia.org/wiki/JSON">a programming format</
|
||||
/home/expo/expofiles/surveyscans/2018/
|
||||
</pre>
|
||||
will, after the appropriate magic has happened, contain a file
|
||||
<pre>
|
||||
index.html
|
||||
</pre>
|
||||
<p>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">index.html</span>
|
||||
<p>
|
||||
which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them.
|
||||
You can see <a href="../../../expofiles/surveyscans/2016/index.html">the index.html for 2016</a>
|
||||
(most of these have been done, but the index has not been updated). Also there will be a linked file for each individual
|
||||
You can see <a href="http://expo.survex.com/expofiles/surveyscans/2018/">the index.html for 2018</a> . Also there will be a linked file for each individual
|
||||
for their personal to-do list, and each online wallet contains its own
|
||||
<span style="font-family:monospace">index.html</span> file which describes the survey production status for all the wallets.
|
||||
|
||||
<p>All this magic is created by a script <span style="font-family:monospace">make.py</span>.
|
||||
<p>The magic creates index.html files in each folder /2018/2018#nn/
|
||||
and creates or updates a webpage for each person listed in any of the contents.json files
|
||||
in the folder/2018/ e.g. <a href="http://expo.survex.com/expofiles/surveyscans/2018/Becka%20Lawson.html">Becka Lawson.html</a>.
|
||||
|
||||
<p>All this magic is created by a script <span style="font-family:monospace; size=x-small; background-color: lightgray">wallets.py</span>.
|
||||
|
||||
<h3>Setting up the online wallets</h3>
|
||||
<p>When, at the beginning of expo, you create the folder in
|
||||
<span style="font-family:monospace">expofiles/surveyscans/</span> for the current year, e.g.
|
||||
<span style="font-family:monospace">/2018/</span>, you will copy <span style="font-family:monospace">make.py</span>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">expofiles/surveyscans/</span> for the current year, e.g.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">/2019/</span>, you will copy <span style="font-family:monospace">wallets.py</span>
|
||||
from the previous year's folder. You will do this on your own laptop or on the expo laptop.
|
||||
<p>You will also manually create a number of subfolders, e.g. 2018#01, 2018#02 etc. to be ready for the influx of
|
||||
<p>You will also manually create a number of subfolders, e.g. 2019#01, 2019#02 etc. to be ready for the influx of
|
||||
new trip surveys.
|
||||
<p>Next you will test that the magic works: open a terminal in
|
||||
<span style="font-family:monospace">expofiles/surveyscans/2018/</span> and run
|
||||
<pre>
|
||||
python make.py
|
||||
</pre>
|
||||
<p>which will work immediately without errors because you are using Linux and the
|
||||
correct version of python is already installed.
|
||||
<p>This will create a default <span style="font-family:monospace">contents.json</span> and
|
||||
<span style="font-family:monospace">index.html</span> in each online wallet folder and also a
|
||||
<span style="font-family:monospace">index.html</span> in the /2018/ folder.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">expofiles/surveyscans/2018/</span> and run
|
||||
<p>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">python wallets.py</span>
|
||||
<br>
|
||||
<p>This will create a default <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> and
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">index.html</span> in each online wallet subfolder and also a
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">index.html</span> in the <span style="font-family:monospace; size=x-small; background-color: lightgray">/2018/</span> folder.
|
||||
|
||||
<p>This script works fine on Linux (Debian, Xubuntu, etc.) and also now works fine in the <a href="https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/">Windows 10 bash system</a>.
|
||||
|
||||
<h3>Maintaining the online wallets</h3>
|
||||
<p>Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the
|
||||
<span style="font-family:monospace">contents.json</span> file in their wallet. In your dreams.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> file in their wallet. In your dreams.
|
||||
|
||||
<p>The first difficulty when editing a blank <span style="font-family:monospace">contents.json</span>
|
||||
for a newly-created wallet is finding out which cave the wallet describes.
|
||||
The wallet may say "radaghost to blitzkriek"
|
||||
The lable on the plastic wallet may say "radaghost to blitzkriek"
|
||||
(or whatever) but without the name of the cave you can't find the .svx files
|
||||
as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/.
|
||||
Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the
|
||||
@ -150,34 +153,36 @@ informal name of the cave,e.g. "Homecoming",instead of the identifier "2018-dm-0
|
||||
recently created wallets and to check that names, dates and cave numbers are correct.
|
||||
|
||||
You will run
|
||||
<pre>
|
||||
python make.py
|
||||
</pre>
|
||||
<p>
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">python wallets.py</span>
|
||||
<p>
|
||||
regularly, after every batch of survey data is entered or scanned.
|
||||
<p>This will always overwrite all the <span style="font-family:monospace">index.html</span> files but it will never touch
|
||||
the <span style="font-family:monospace">contents.json</span> files.
|
||||
the <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> files.
|
||||
<p>You will also regularly synchronise your laptop
|
||||
and the expo laptop with <span style="font-family:monospace">expo.survex.com/expofiles/surveyscan/2018/</span>
|
||||
and this is where it gets tricky.
|
||||
<p><span style="font-family:monospace">expo.survex.com/expofiles/</span> is not under version control,
|
||||
<p><span style="font-family:monospace">expo.survex.com/expofiles/</span> is <font color=red><b>not under version control</b></font>,
|
||||
so the most recent person
|
||||
to upload the contents of <span style="font-family:monospace">/2018/</span> will overwrite everyone else's work.
|
||||
to upload the contents of <span style="font-family:monospace">/2018/</span> <font color=red><b>will overwrite everyone else's work</b></font>.
|
||||
This does not matter for the autogenerated files, but it is vital that it does not overwrite all the painfully manually edited
|
||||
<span style="font-family:monospace">contents.json</span> files. Which is very easy to do.
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> files. Which is very easy to do.
|
||||
This does mean that this is one of the cases where
|
||||
it is better to use <span style="font-family:monospace">rsync</span> rather than an FTP client such as Filezilla.
|
||||
it is better to use <span style="font-family:monospace; size=x-small; background-color: lightgray">rsync</span> rather than an FTP client such as Filezilla.
|
||||
|
||||
<p>A copy of useful rsync scripts is kept in a file such as
|
||||
<span style="font-family:monospace">expo.survex.com/expofiles/rsync2018toserver</span>. Always run it with the -n option first,
|
||||
<span style="font-family:monospace; size=x-small; background-color: lightgray">expo.survex.com/expofiles/rsync2018toserver</span>. Always run it with the -n option first,
|
||||
to see what overwriting you will do.
|
||||
|
||||
<h3>More <em><span style="font-family:monospace">make.py</span></em> magic</h3>
|
||||
<p>The python script does more than just re-format the <span style="font-family:monospace">contents.json</span> data into
|
||||
<h3>More <em><span style="font-family:monospace; size=x-small; background-color: lightgray">wallets.py</span></em> magic</h3>
|
||||
<p>The python script does more than just re-format the <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> data into
|
||||
different formats. It also
|
||||
<ul>
|
||||
<li>checks whether the .svx files listed are actually present,
|
||||
<li>checks whether the .svx files listed are actually present <br>in the <a href="http://expo.survex.com/repositories/home/expo">::loser::</a> repository
|
||||
<li>checks for the presence of notesXXX.jpg, planXXX.jpg and elevXXX.jpg files
|
||||
<li>creates a template <span style="font-family:monospace">contents.json</span> in any wallet which does not have one.
|
||||
<li>creates a template <span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span> in any wallet which does not have one.
|
||||
<li>creates helpful URL links to the existing online survey documentation for the cave being surveyed
|
||||
<li>creates helpful URL links to the working files you are managing on your own laptop
|
||||
</ul>
|
||||
|
||||
<p>Things it might do in future (if someone gets around to it) include: <br>
|
||||
@ -189,8 +194,8 @@ different formats. It also
|
||||
- being more intelligent about .topo files and thus the lack of scan files,<br>
|
||||
- checking the date is in the recent past etc.<br><br>
|
||||
|
||||
<h3>How <em><span style="font-family:monospace">contents.json</span></em> fields match
|
||||
<em><span style="font-family:monospace">index.html</span></em> reports</h3>
|
||||
<h3>How <em><span style="font-family:monospace; size=x-small; background-color: lightgray">contents.json</span></em> fields match
|
||||
<em><span style="font-family:monospace; size=x-small; background-color: lightgray">index.html</span></em> reports</h3>
|
||||
<p>
|
||||
<em>to be written...</em>
|
||||
<hr>
|
||||
@ -212,24 +217,27 @@ using the following input materials:
|
||||
- the wallet 2018#03 and the papers inside it. This is in the 2018 lever-arch file.
|
||||
- the folder in repo 'loser' holding the appropriate .svx files e.g.
|
||||
"caves-1623/2017-cucc-24/gshclimb.svx"
|
||||
- the script expofiles/surveyscans/2018/make.py (run by "python make.py")
|
||||
- the script expofiles/surveyscans/2018/wallets.py (run by "python wallets.py")
|
||||
|
||||
the "make.py" script creates index.html files in each folder /2018/2018#nn/
|
||||
the "wallets.py" script creates index.html files in each folder /2018/2018#nn/
|
||||
and creates or updates a webpage for each person listed in any of the contents.json files
|
||||
in the folder/2018/ e.g. "Becka Lawson.html".
|
||||
in the folder/2018/.
|
||||
|
||||
The script make.py requires that the //loser// repo is populated on the machien that you
|
||||
run the script on so that it can find the.svx files. Do not run this script on
|
||||
the expo server itself as the loser repo is in the wrong place and all the .svx files
|
||||
will be flagged as "not found" (Paul Fox did this once).
|
||||
The script wallets.py requires that the //loser// repo is populated on the machine that you
|
||||
run the script on so that it can find the.svx files.
|
||||
|
||||
Before doing anything else, run make.py. This will create empty template contents.json
|
||||
If your machine has the ::loser:: repo in a different place from that expected by the script, you can just
|
||||
put the path on the command line:
|
||||
|
||||
python wallets.py "/mnt/d/CUCC-Expo/loser/"
|
||||
|
||||
Before doing anything else, run wallets.py. This will create empty template contents.json
|
||||
files in each folder.
|
||||
|
||||
You may need to create missing folders,e.g. I just had to create /2018/2018#30 to #32.
|
||||
|
||||
Every time you finish entering the data in contents.json in a folder,
|
||||
run make.py to update the "person" html files and to
|
||||
run wallets.py to update the "person" html files and to
|
||||
re-generate the index.html file for the 2018 folder as
|
||||
a whole (surveyscans/2018/index.html).
|
||||
|
||||
@ -242,7 +250,7 @@ should be done and thus which products should be produced.
|
||||
This needs to be documented.
|
||||
|
||||
For prospecting and surface surveying it is not clear whether the default folder
|
||||
for the url link should be repo //loser// surface/1623/allplateau.svx
|
||||
for the url link should be repo ::loser:: surface/1623/allplateau.svx
|
||||
|
||||
When there are more than one .svx file there seems to be no way of recording the list
|
||||
in contents.json so it is impossible to tell what was done on that trip or whether
|
||||
@ -261,10 +269,39 @@ HINT
|
||||
When there are a lot on wallets all with the same cave, make your own template
|
||||
with the cave name and the right folder prefix for the svx folder
|
||||
(in the loser repo) and copy it in to all those wallet folders - overwriting
|
||||
the blank template produced by the make.py
|
||||
the blank template produced by the wallets.py
|
||||
|
||||
# Update March 2019
|
||||
|
||||
a consolidated to-do list of the last 3 years on the server:
|
||||
http://expo.survex.com/expofiles/surveyscans/2016-18/index.html
|
||||
This is a hand-done kludge and only the first level of links works - which is to the individual person's page.
|
||||
|
||||
the lists for the last 3 years individually and all the links are working for each wallet page: both local links to your PC and to the right location of the .svx files on the troggle server.
|
||||
http://expo.survex.com/expofiles/surveyscans/2016/index.html
|
||||
http://expo.survex.com/expofiles/surveyscans/2017/index.html
|
||||
http://expo.survex.com/expofiles/surveyscans/2018/index.html
|
||||
and all the names of people have been hand-edited in the .json files to be consistent and identical.
|
||||
2015 will be done soon.
|
||||
|
||||
Changes to script wallets.py
|
||||
----------------------------
|
||||
As all this is not under version control the timestamps of the files were probably quite important in figuring things out when someone makes an update mistake.
|
||||
|
||||
So script wallets.py is modified so that
|
||||
- the generated index.html file in each wallet folder is given
|
||||
the same timestamp as the contents.json file there,
|
||||
|
||||
- the script no longer overwrites the contents.json files every
|
||||
time it runs. It only changes the timestamp if it actually changes anything.
|
||||
|
||||
|
||||
For 2014 and earlier one needs to do a lot more data entry. The contents.json files for 2014 and earlier do not say who the people were on the trip. So we would need to work from the original plastic wallets (and the scanned drawings and notes – which are incomplete) to enter that data. This is probably not worth doing except maybe for specific critical connections.
|
||||
|
||||
The script runs without errors on each of the years 1999-2014, but the results are less useful, e.g. see
|
||||
http://expo.survex.com/expofiles/surveyscans/1999/
|
||||
or
|
||||
http://expo.survex.com/expofiles/surveyscans/2014/
|
||||
|
||||
</pre>
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user