diff --git a/handbook/survey/README b/handbook/survey/README new file mode 100644 index 000000000..a1ced9ffd --- /dev/null +++ b/handbook/survey/README @@ -0,0 +1,10 @@ +Add new folders of the form +2017#01 (where 01 is a serial number) for surveys with physical notes +or +2017#X01 (where 01 is a serial number) for surveys with only digital data + + +Dummy text to stop mistaken messages about broken links +from the linkchecker at +brokenlinkcheck.com + diff --git a/handbook/survey/how-to-do-it b/handbook/survey/how-to-do-it new file mode 100644 index 000000000..3a6808a15 --- /dev/null +++ b/handbook/survey/how-to-do-it @@ -0,0 +1,89 @@ +# Instructions +# 2019-03-02 +# Philip Sargent + +Wookey told me to sort out the contents.json files in expofiles/surveyscans/2018/ +and these are my notes to remind myself what this entails. +This note was written during August 2018. + +The job is to populate the contents.json file in each folder, e.g. + +expofiles/surveyscans/2018/2018#03/contents.json + +using the following input materials: +- the wallet 2018#03 and the papers inside it. This is in the 2018 lever-arch file. +- the folder in repo 'loser' holding the appropriate .svx files e.g. + "caves-1623/2017-cucc-24/gshclimb.svx" +- the script expofiles/surveyscans/2018/wallets.py (run by "python wallets.py") + +the "wallets.py" script creates index.html files in each folder /2018/2018#nn/ +and creates or updates a webpage for each person listed in any of the contents.json files +in the folder/2018/ e.g. "Becka Lawson.html". + +The script wallets.py requires that the //loser// repo is populated on the machine that you +run the script on so that it can find the.svx files. Do not run this script on +the expo server itself as the loser repo is in the wrong place and all the .svx files +will be flagged as "not found" (Pauul Fox did this once). + +wallets.py now (Feb.2019) takes a command-line argument for the location of the loser repo, e.g. +python wallets.py "/mnt/d/CUCC-Expo/loser/" + +Before doing anything else, run wallets.py. This will create empty template contents.json +files in each folder. + +You may need to create missing folders,e.g. I just had to create /2018/2018#30 to #32. + +Every time you finish entering the data in contents.json in a folder, +run wallets.py to update the "person" html files and to +re-generate the index.html file for the 2018 folder as +a whole (surveyscans/2018/index.html). + +But wallets.py does not delete old files which are not regenerated. +So it is a good idea to delete all the .html files +before running it. + +The first difficulty is finding out which cave a wallet describes.. +The wallet may say "radaghost to blitzkriek" +(or whatever) but without the name of the cave you can't find the .svx files +as you don't know that you need to look in e.g. loser/caves-1626/2018-dm-07/ +Usually this is written by hand on the label of the wallet. + +There are ambiguities about how the entries in the contents.json actually lead to +reminder instructions in the html files produced, and this is particularly +difficult for electronic caves where the topo files are missing +and for surface prospecting where it is not clear which of the actions +should be done and thus which products should be produced. + +This needs to be documented. + +For prospecting and surface surveying it is not clear whether the default folder +for the url link should be repo //loser// surface/1623/allplateau.svx + +When there are more than one .svx file there seems to be no way of recording the list +in contents.json so it is impossible to tell what was done on that trip or whether +there is anything missing. This is especially true if it was electronic and the +.topo files are missing. Wookey confirms that this ios the case. + +wallets.py will not overwrite any contents.json file, it only creates one where one +doesn't exist. + +wallets.py also creates an index.html file inside each wallet folder dewcribing the status of this +folder. This is linked from the index.html file created in /2018/ by the script which +summarises everything for this year. + +NAMING +The script detects if there are notesX.jpg planX.jpg and elevX.jpgfiles present, and +produces a reminder/warning if they are not,even if these have all been scanned +and given different names. + +The job of the checker (perhaps on a second pass) if to rename files so that these +warnings disappear. + +HINT +When there are a lot on wallets all with the same cave, make your own template +with the cave name and the right folder prefix for the svx folder +(in the loser repo) and copy it in to all those wallet folders - overwriting +the blank template produced by the wallets.py + + + diff --git a/handbook/survey/onlinewallet.html b/handbook/survey/onlinewallet.html index 144aca87e..6508379ac 100644 --- a/handbook/survey/onlinewallet.html +++ b/handbook/survey/onlinewallet.html @@ -17,7 +17,7 @@

The scanned pages

-

These are simply the scanned imaages (or digital photographs) of each page of the original survey notes. +

These are simply the scanned images (or digital photographs) of each page of the original survey notes. They should be named notesXXX.jpg where "XXX" can be anything you like. Typically we have the scanned pages called notes1.jpg, notes2.jpg, notes3.jpg.

It is important that you use use the .jpg (JPEG) file format, and definitely not PNG (very voluminous) @@ -25,7 +25,7 @@ or PDF (very hard to re-use elsewhere). Set the scanner at 300 dpi and adjust th by using photo-editing software to enhance the writing. Also please crop each image to just the area containing the survey data.

As soon as the notes have been scanned you should (a) copy them to a USB stick or email them to someone, (b) upload the entire online wallet to the expo server in Cambridge -expo.survex.com. This is so that these precious files are backed-up as soon as possible. +expo.survex.com. This is so that these precious files are backed-up as soon as possible. (Assuming you have done the key-pair procedure, do the upload by sFTP as you do for photos, but to the directory described in the introduction to online wallets.)

The other files and online index contents.json

All the other files are part of the multi-step process of producing the cave survey - see @@ -51,7 +51,7 @@ have a different Austrian Kataster number issued for it in due course). *ref 2018#06 ; the #number is on the clear pocket containing the original notes -But sometime in mid-Expo 2015 everyone stopped using the survex template file and so this information was not recorded since then. This will be fixed by hand-editing indue course. +But sometime in mid-Expo 2015 everyone stopped using the survex template file and so this information was not recorded since then. This will be fixed by hand-editing in due course. (Note that many old .svx files were processed with an older version of survex which did not suppport this feature and so a comment was used instead.)

Troggle produces very useful auto-generated reports of the status of the wallets and the survex files @@ -127,17 +127,21 @@ in filenames on Windows computers. will, after the appropriate magic has happened, contain a file

-index.html +walletindex.html

which lists all the wallets which have uncompleted tasks, and lists all the people responsible for completing them. -You can see the index.html for 2018 . Also there will be a linked file for each individual +You can see the walletindex.html for 2018 . Also there will be a linked file for each individual for their personal to-do list, and each online wallet contains its own index.html file which describes the survey production status for all the wallets. -

The magic creates index.html files in each folder /2018/2018#nn/ +

The magic also creates walletindex.html files in each folder /2018/2018#nn/ and creates or updates a webpage for each person listed in any of the contents.json files in the folder/2018/ e.g. Becka Lawson.html. +

If you want to see what the files are in each online wallet then look at
+expo.survex.com/expofiles/surveyscans/2018/
+and you can navigate to the individual wallet folders too and look directly at the files there. +

All this magic is created by a script wallets.py.

Setting up the online wallets

@@ -153,8 +157,8 @@ new trip surveys. python wallets.py

This will create a default contents.json and -index.html in each online wallet subfolder and also a -index.html in the /2018/ folder. +walletindex.html in each online wallet subfolder and also a +walletindex.html in the /2018/ folder.

This script works fine on Linux (Debian, Xubuntu, etc.) and also now works fine in the Windows 10 bash system. @@ -177,7 +181,7 @@ You will run python wallets.py

regularly, after every batch of survey data is entered or scanned. -

This will always overwrite all the index.html files but it will never touch +

This will always overwrite all the walletindex.html files but it will never touch the contents.json files.

You will also regularly synchronise your laptop and the expo laptop with
@@ -206,7 +210,7 @@ As all this is not under version control the timestamps of the files are really

So script wallets.py has been fixed so that

@@ -221,9 +225,11 @@ different formats. It also

Things it might do in future (if someone gets around to it) include:
@@ -237,60 +243,14 @@ different formats. It also - checking the date is in the recent past etc.

How contents.json fields match -index.html reports

+walletindex.html reports

to be written...


Old notes, being turned into real documentation...
-# Instructions
-# 2018-08-14
-# Philip Sargent
 
 
-Wookey told me to sort out the contents.json files in expofiles/surveyscans/2018/
-and these are my notes to remind myself what this entails.
-
-The job is to populate the contents.json file in each folder, e.g.
-
-expofiles/surveyscans/2018/2018#03/contents.json
-
-using the following input materials:
-- the wallet 2018#03 and the papers inside it. This is in the 2018 lever-arch file.
-- the folder in repo 'loser' holding the appropriate .svx files e.g.
-  "caves-1623/2017-cucc-24/gshclimb.svx"
-- the script expofiles/surveyscans/2018/wallets.py (run by "python wallets.py")
-
-the "wallets.py" script creates index.html files in each folder /2018/2018#nn/
-and creates or updates a webpage for each person listed in any of the contents.json files
-in the folder/2018/.
-
-The script wallets.py requires that the //loser// repo is populated on the machine that you
-run the script on so that it can find the.svx files. 
-
-If your machine has the ::loser:: repo in a different place from that expected by the script, you can just 
-put the path on the command line:
-
-python wallets.py "/mnt/d/CUCC-Expo/loser/"
-
-Before doing anything else, run wallets.py. This will create empty template contents.json
-files in each folder.
-
-You may need to create missing folders,e.g. I just had to create /2018/2018#30 to #32.
-
-Every time you finish entering the data in contents.json in a folder, 
-run wallets.py to update the "person" html files and to 
-re-generate the index.html file for the 2018 folder as 
-a whole (surveyscans/2018/index.html).
-
-There are ambiguities about how the entries in the contents.json actually lead to
-reminder instructions in the html files produced, and this is particularly
-difficult for electronic caves where the topo files are missing
-and for surface prospecting where it is not clear which of the actions
-should be done and thus which products should be produced.
-
-This needs to be documented.
-
 For prospecting and surface surveying it is not clear whether the default folder
 for the url link should be repo ::loser:: surface/1623/allplateau.svx
 
@@ -309,14 +269,14 @@ the blank template produced by the wallets.py
 # Update March 2019
 
 a consolidated to-do list of the last 3 years on the server:
-http://expo.survex.com/expofiles/surveyscans/2016-18/index.html
+http://expo.survex.com/expofiles/surveyscans/2016-18/walletindex.html
 This is a hand-done kludge and only the first level of links works - which is to the individual person's page.
 
 the lists for the last 3 years individually and all the links are working for each wallet page: 
 both local links to your PC and to the right location of the .svx files on the troggle server.
-http://expo.survex.com/expofiles/surveyscans/2016/index.html
-http://expo.survex.com/expofiles/surveyscans/2017/index.html
-http://expo.survex.com/expofiles/surveyscans/2018/index.html
+http://expo.survex.com/expofiles/surveyscans/2016/walletindex.html
+http://expo.survex.com/expofiles/surveyscans/2017/walletindex.html
+http://expo.survex.com/expofiles/surveyscans/2018/walletindex.html
 and all the names of people have been hand-edited in the .json files to be consistent and identical.
 
 2015 has now been done stand-alone but there is no consolidated report for 2015-18 yet.