Each wallet has a corresponding folder in the online system where a record is kept of what information is in the wallet and where the corresponding survey data is filed: @@ -126,7 +126,8 @@ and tell someone nerdy when you have finished and they will ensure that it is copied to the expo server. [If you want to do this yourself, or are using your own laptop, then learn how to use Filezilla - as documented for uploading your expo photographs. The correct folder -on the expo server is the same as that on the expo laptop- because we set up the expo laptop to be like that.] +on the expo server is the same as that on the expo laptop- because we set up the expo laptop to be like that. +But only copy files to the server that you created yourself and which live in your own wallet folder]
If you used a PDA instead of making notes on paper, you need to store your .topo files in the right place. @@ -195,7 +196,8 @@ as the scanned notes, i.e. (for wallet #19) you would put them in:
Tunnel only produces plan surveys, but they are very pretty. -
The tunnel (or therion) files should be stored in the same folder as the scanned notes. +
The tunnel (or therion) files should NOT stored in the same folder as the scanned notes. They should +be uploaded to the version control repository //tunneldata//.
This is not the page you are looking for. +
If you are a newcomer to the system, read the beginners introduction to online wallets first. + +
There are three quite different reasons: +
These are simply the scanned imaages (or digital photographs) of each page of the original survey notes. +They should be named notesXXX.jpg where "XXX" can be +anything you like. Typically we have the scanned pages called notes1.jpg, notes2.jpg, notes3.jpg. +
It is important that you use use the .jpg (JPEG) file format, and definitely not PNG (very voluminous) +or PDF (very hard to re-use elsewhere). Set the scanner at 300 dpi and adjust the contrast of the image after scanning +by using photo-editing software to enhance the writing. Also please crop each image to just the area containing +the survey data. +
As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) upload the entire online wallet to the expo server in Cambridge +expo.survex.com. This is so that these precious files are backed-up as soon as possible. + +
All the other files are part of the multi-step process of producing the cave survey - see +Creating a new cave... for the full list of steps. + +
We keep an index of how many of those steps have been completed in two places: +
but the contents.json file has another,completely different function:
+it is the only online record that connects the wallet number to the cave identifier. So if a future cave surveyor deperately needs
+to consult the original cave survey, it can be done by, e.g.
+
+grep -rl "2018-dm-07" expofiles/surveyscans
+
+will find and list all the wallets which contain survey data for cave 2018-dm-07 (which is also known as "Homecoming Cave" and which will
+have a different Austrian Kataster number issued for it in due course).
+
+
The paper tick-list tracks the following steps for each online wallet: +
(where the "json file updated" step only refers to the initial editing of the json file to ensure +that it has the right people, date and cave identifier and name). + +
A fully-populated and complete contents.json file +looks like this: +
+{
+ "description written": false,
+ "website updated": false,
+ "people": [
+ "Dickon Morris",
+ "Jon Arne Toft",
+ "Becka Lawson"],
+ "elev not required": false,
+ "cave": "2018-dm-07",
+ "survex not required": false,
+ "qms written": true,
+ "plan not required": false,
+ "electronic survey": false,
+ "plan drawn": true,
+ "date": "2018-07-13",
+ "elev drawn": true,
+ "description url": "",
+ "survex file": "caves-1626/2018-dm-07/2018-dm-07.svx",
+ "name": "Homecoming cave"
+}
+
+Yes, this is a programming format and every comma is critical.
+
+The folder containing all the wallets for the year, e.g. +
+/home/expo/expofiles/surveyscans/2018/ ++will, after the appropriate magic has happened, contain a file +
+index.html ++which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them. +You can see the index.html for 2016 +(most of these have been done, but the index has not been updated). Also there will be a linked file for each individual +for their personal to-do list, and each online wallet contains its own +index.html file which describes the survey production status for all the wallets. + +
All this magic is created by a script make.py. + +
When, at the beginning of expo, you create the folder in +expofiles/surveyscans/ for the current year, e.g. +/2018/, you will copy make.py +from the previous year's folder. You will do this on your own laptop or on the expo laptop. +
You will also manually create a number of subfolders, e.g. 2018#01, 2018#02 etc. to be ready for the influx of +new trip surveys. +
Next you will test that the magic works: open a terminal in +expofiles/surveyscans/2018/ and run +
+python make.py ++
which will work immediately without errors because you are using Linux and the +correct version of python is already installed. +
This will create a default contents.json and +index.html in each online wallet folder and also a +index.html in the /2018/ folder. + +
Ideally the cavers who are scanning their notes and typing in the survey data will also be updating the +contents.json file in their wallet. In your dreams. + +
The first difficulty when editing a blank contents.json +for a newly-created wallet is finding out which cave the wallet describes. +The wallet may say "radaghost to blitzkriek" +(or whatever) but without the name of the cave you can't find the .svx files +as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/. +Usually the cave number is written by hand on the label of the wallet. Sometimes it will just give the +informal name of the cave,e.g. "Homecoming",instead of the identifier "2018-dm-07" you want. +
A regular task during expo is for a nerd to review the contents.json files for +recently created wallets and to check that names, dates and cave numbers are correct. + +You will run +
+python make.py ++regularly, after every batch of survey data is entered or scanned. +
This will always overwrite all the index.html files but it will never touch +the contents.json files. +
You will also regularly synchronise your laptop +and the expo laptop with expo.survex.com/expofiles/surveyscan/2018/ +and this is where it gets tricky. +
expo.survex.com/expofiles/ is not under version control, +so the most recent person +to upload the contents of /2018/ will overwrite everyone else's work. +This does not matter for the autogenerated files, but it is vital that it does not overwrite all the painfully manually edited +contents.json files. Which is very easy to do. +This does mean that this is one of the cases where +it is better to use rsync rather than an FTP client such as Filezilla. + +
A copy of useful rsync scripts is kept in a file such as +expo.survex.com/expofiles/rsync2018toserver. Always run it with the -n option first, +to see what overwriting you will do. + +
The python script does more than just re-format the contents.json data into +different formats. It also +
Things it might do in future (if someone gets around to it) include:
+- checking the cave number specified matches the folder for the .svx file,
+- detecting whether there is a description or a list of QMs in the survex file,
+- accepting a list of .svx files and not just one (a very common thing),
+- checking the name of the cave against the cave number,
+- checking whether the website page even exists for this cave,
+- being more intelligent about .topo files and thus the lack of scan files,
+- checking the date is in the recent past etc.
+
+
+to be written... +
# Instructions # 2018-08-14 # Philip Sargent + Wookey told me to sort out the contents.json files in expofiles/surveyscans/2018/ and these are my notes to remind myself what this entails. @@ -44,16 +232,6 @@ run make.py to update the "person" html files and to re-generate the index.html file for the 2018 folder as a whole (surveyscans/2018/index.html). -But make.py does not delete old files which are not regenerated. -So it is a good idea to delete all the .html files -before running it. - -The first difficulty is finding out which cave a wallet describes.. -The wallet may say "radaghost to blitzkriek" -(or whatever) but without the name of the cave you can't find the .svx files -as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/ -Usually this is written by hand on the label of the wallet. - There are ambiguities about how the entries in the contents.json actually lead to reminder instructions in the html files produced, and this is particularly difficult for electronic caves where the topo files are missing @@ -68,14 +246,7 @@ for the url link should be repo //loser// surface/1623/allplateau.svx When there are more than one .svx file there seems to be no way of recording the list in contents.json so it is impossible to tell what was done on that trip or whether there is anything missing. This is especially true if it was electronic and the -.topo files are missing. Wookey confirms that this ios the case. - -make.py will not overwrite any contents.json file, it only creates one where one -doesn't exist. - -make.py also creates an index.html file inside each wallet folder dewcribing the status of this -folder. This is linked from the index.html file created in /2018/ by the script which -summarises everything for this year. +.topo files are missing. Wookey confirms that this is the case. NAMING The script detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and @@ -95,7 +266,6 @@ the blank template produced by the make.py-
This will be replaced with the information you want as soon as someone gets around to writing it. Why not find out how to do this yourself ?