From 67020817793cd1ec604b04cbc072beed08a50077 Mon Sep 17 00:00:00 2001
From: Philip Sargent We interact with it using:
This is part of the "new expo year" preparations.
-The folk.csv file is stored on the server under version control in the :expoweb: repository in
+The folk.csv file is stored on the server under version control in the :expoweb: repository in
Before expo starts the folk.csv file is updated.
diff --git a/handbook/computing/hbmanual1.html b/handbook/computing/hbmanual1.html
index a66037d66..cfe58a1fb 100644
--- a/handbook/computing/hbmanual1.html
+++ b/handbook/computing/hbmanual1.html
@@ -93,7 +93,7 @@ t/ - a thumbnail image which is usually used inside an <a> tag to link to
'Edit This Page' edits the file served on the web but it does not update the file in the version control
-repository. To properly finish the job you need to get a nerd to
+repository. To properly finish the job you need to get a nerd to
These pages are for cavers wanting to:
See the Expo data management manual for a fuller description of the version control software
-repositories and how to install and use the software.
+ See the repositories page.
All the scans, photos, presentations, fat documents and videos are
diff --git a/handbook/computing/manual.html b/handbook/computing/manual.html
index cb4486077..ddc2fe490 100644
--- a/handbook/computing/manual.html
+++ b/handbook/computing/manual.html
@@ -50,16 +50,7 @@ processes that a maintainer would want to do. Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the troggle intro.
- We use a distributed revision control system (git) for all the important data. (Note that we just use git: not GitHub, not GitLab, just git.)
-This means that many people can edit and merge their changes with the expo
-server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut
-and those who have returned to the UK. Also anyone who is up
-to date can take their laptop somewhere and enter data even if they have no internet access,
-and the updates will be merged when they get back to civilization.
- In principle, survey notes can be typed into a laptop up on the plateau which would
-then get synchronised when it next gets internet access.
+
A version control system is inefficient for scanned survey notes which are large files that
do not get modified, so they are kept as a plain directory of files 'expofiles'.
@@ -69,7 +60,7 @@ The same goes for holiday photographs and GPS logs. This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then
you will also need to get a login (register a key with the server). See key-pair setup for details.
- Pushing cave data to the :loser: and :drawings: repositories also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are
+ Pushing cave data to the :loser: and :drawings: repositories also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are
the expo laptop and the laptop 'aziraphale' which live in the potato hut during expo. If you want to use your own laptop then
see below.
-
- All the expo data is contained in 4 "repositories" at
-expo.survex.com. This is currently hosted on a free virtual server we have blagged on a server farm.
-We use a distributed version control system (DVCS) to manage these repositories because this allows simultaneous collaborative
-editing and keeps track of all changes so we can roll back and have branches if needed. The site has been split into four parts: Public cave description pages are automatically generated by troggle from a set of
@@ -152,7 +125,7 @@ superseded in 2012). which contains a number of files used to manage and record that year's expo. Have a look at
expoweb/years/2018/ for a recent well-documented expo (the weather was good).
-Files are added and edited using the version control system for the expoweb repository.
diff --git a/handbook/computing/exposerver.html b/handbook/computing/exposerver.html
index 6849565fe..93353c041 100644
--- a/handbook/computing/exposerver.html
+++ b/handbook/computing/exposerver.html
@@ -15,7 +15,7 @@
diff --git a/handbook/computing/folkupdate.html b/handbook/computing/folkupdate.html
index 4ea67ffd9..f43f20a6a 100644
--- a/handbook/computing/folkupdate.html
+++ b/handbook/computing/folkupdate.html
@@ -13,7 +13,7 @@
The folk list
expoweb/folk/folk.csv
Tidying up and commiting the edits
@@ -28,7 +28,7 @@ using a text editor.
What you can't do
Troggle - what it is
Version control system
-
The repositories
-
-
-
-
-
Updating cave pages
To create a new 'year' for next year's expo see adding a new year. diff --git a/handbook/computing/newyear.html b/handbook/computing/newyear.html index 0e151fede..92f689e10 100644 --- a/handbook/computing/newyear.html +++ b/handbook/computing/newyear.html @@ -18,7 +18,7 @@
There is no script to do this (yet). In the list below everything in expofiles is done by directly overwriting what is on the server. Changes to expoweb and troggle must be done using the version control system as these are repositories. +
There is no script to do this (yet). In the list below everything in expofiles is done by directly overwriting what is on the server. Changes to expoweb and troggle must be done using the version control system as these are repositories.
All the expo data is contained in 4 repositories or repos at +expo.survex.com: + + +
You can read or clone these without any control, but to write ("push") to them you will need to use So cavers wanting to upload their cave survey data have to use a machine on which they key is already set up. The expo laptop lives in the potato hut during expo and it is sometimes joined by the laptop 'aziraphale'. These are both loaned machines running Linux. If you want to use your own laptop then
+see the minimal laptop setup.
+
+
+ We use a distributed revision control system (git) for all the important data.
+ Note that we just use git: not GitHub, not GitLab, just git.
+
+This allows many people to edit and merge their changes at the same time. This includes people still on expo and those who have returned to the UK. Also anyone who is up
+to date can take their laptop somewhere (e.g. topcamp) and enter data even if they have no Internet access,
+and the updates will be merged when they get back to civilization.
+
+We use a version control system because this keeps track of all changes so we can roll back mistakes. We also have multiple copies of the data on multiple machines for safety.
+
+We use git because it is now the industry standard. In the past we have used cvs, subversion and mercurial.
+
+ A version control system is inefficient for photos, GPS tracks and scanned survey notes which are large files that do not get modified. These are kept as a plain directory of files /expofiles/.
+ Linux people like to use links. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks.
The links you are most likely to come across are that what looks like
-/home/expo/expoweb is really just a link to the folder /home/expo/repositories/hg/expoweb, For a quicker, shorter guide to only the most basic setup, see the
To regenerate the most recent version which contains the cave entrances discovered during expo
you will need to ask someone who is competent in logging into the server and running scripts.
- The data is in the version control system repository :loser: in
+ The data is in the version control system repository :loser: in
GPS tracks are voluminous and we also get a lot of repetition
@@ -112,7 +112,7 @@ This is for consistency with the naming for
If you have edited GPS tracks and waypoints with no extraneous data
then, after agreeing this with other people as to its qualityand appropriateness,
-it will go into the Loser DVCS repository
+it will go into the Loser git repository
in folder /gpx/<year>/ e.g. Note the naming convention for this file created by Anthony Day on July 12th 2018.
diff --git a/handbook/survey/caveentry.html b/handbook/survey/caveentry.html
index 426171d18..78b0bf00f 100644
--- a/handbook/survey/caveentry.html
+++ b/handbook/survey/caveentry.html
@@ -39,7 +39,7 @@ This page outlines step 6 of the survey production process. Each step is documen
For experienced people creating a batch of small caves the file upload & import process is quicker. Most people won't use this method
for their first cave so it is documented separately in new cave file method.
In other words you can either add caves (or entrances) using the web-interface
- or you can check out the 'expoweb' repository using the version control system and just edit the files.Version control system
+
+Return to Expo Data Maintenance manual
+Return to Expo Systems Overview
+
+
+
diff --git a/handbook/computing/winlaptop.html b/handbook/computing/winlaptop.html
index 0e1caa68d..88eb5575e 100644
--- a/handbook/computing/winlaptop.html
+++ b/handbook/computing/winlaptop.html
@@ -51,7 +51,7 @@
+/home/expo/expoweb is really just a link to the folder /home/expo/repositories/git/expoweb,
and that
expoweb/essentials.gpx
is a link to the file /home/expo/expofiles/gpslogs/essentials/essentials2019.gpx
diff --git a/handbook/computing/wookey-slides.html b/handbook/computing/wookey-slides.html
index 92e279c8e..4bb13a939 100644
--- a/handbook/computing/wookey-slides.html
+++ b/handbook/computing/wookey-slides.html
@@ -25,7 +25,7 @@ lang="de-at">Salzburg). Recent large projects are:
Data Storage
-
+
diff --git a/handbook/computing/yourlaptop.html b/handbook/computing/yourlaptop.html
index e70a3f07e..dbffe43b8 100644
--- a/handbook/computing/yourlaptop.html
+++ b/handbook/computing/yourlaptop.html
@@ -16,7 +16,7 @@
@@ -164,7 +164,7 @@ to Introduction to using Tunnel - article by Dave Loeffler. This is really good and should be read first.
diff --git a/handbook/essentials.html b/handbook/essentials.html
index 9afb015f1..609c38c8b 100644
--- a/handbook/essentials.html
+++ b/handbook/essentials.html
@@ -114,7 +114,7 @@ devise new routes to reach them.
loser/gpx/
diff --git a/handbook/gpxupload.html b/handbook/gpxupload.html
index 322e72fa0..1c4010030 100644
--- a/handbook/gpxupload.html
+++ b/handbook/gpxupload.html
@@ -86,7 +86,7 @@ to something recognisable such as 'top-camp-to-toilet-grike.gpx' (all lower case
GPX data is stored in two places.
:loser:/2018/stone-bridge-to-fischgesicht_aday-2018-07-12.gpx
Nevertheless you will find it useful when filling out the form for the first time to have some idea what is going on behind the scenes. So you are recommended to have a quick look diff --git a/handbook/survey/caveentryfields.html b/handbook/survey/caveentryfields.html index d0956bc54..18de3403b 100644 --- a/handbook/survey/caveentryfields.html +++ b/handbook/survey/caveentryfields.html @@ -109,7 +109,7 @@
The tunnel (or therion) files should NOT stored in the same folder as the scanned notes. They will eventually be uploaded to the version control repository - - drawings +drawings but for a first attempt store them on the expo laptop in /home/expo/drawings/{cavenumber}. Look at what is in there already and ask someone whcih directory to put them in. It will probably be a folder like this: /home/expo/drawings/264-and-258/toimport/ . diff --git a/handbook/survey/getin.htm b/handbook/survey/getin.htm index b1473c13c..4d0ada717 100644 --- a/handbook/survey/getin.htm +++ b/handbook/survey/getin.htm @@ -35,7 +35,7 @@ on the expo laptop. These instructions assume that you are using the expo laptop that you are typing it in on the expo laptop. (You can do it fom your own laptop if you have been initiated into the deep magic of the ":loser:" repository of the distributed version control system - see the -list of repos. +list of repositories.
Now you have been typing this on the expo laptop, and before you let someone else use the laptop you need to find a nerd to do the deep magic to add your -file to the version control repository "loser". Watching the nerd do this is +file to the version control repository "loser". Watching the nerd do this is usually your first initiation into learning how to do it yourself. diff --git a/handbook/survey/newsurvex.html b/handbook/survey/newsurvex.html index 403a9dd80..349e737c3 100644 --- a/handbook/survey/newsurvex.html +++ b/handbook/survey/newsurvex.html @@ -69,7 +69,7 @@ scan it again and then use that scanned image to digitise passage layout into tu
[Nerds: survex cave data belongs in the repository :loser: so e.g. +
[Nerds: survex cave data belongs in the repository :loser: so e.g. :loser:/caves-1623/264/mongolrally.svx". We are assuming that normal users have never worked with an distributed version control system at this point which is why we are only telling them to use the expo laptop.] diff --git a/handbook/survey/onlinewallet.html b/handbook/survey/onlinewallet.html index 67388e1aa..3fbba5eb6 100644 --- a/handbook/survey/onlinewallet.html +++ b/handbook/survey/onlinewallet.html @@ -223,7 +223,7 @@ to see what overwriting you will do.
The python script does more than just re-format the contents.json data into different formats. It also
There are also scripts running cron jobs on the server to fix file permissions and to periodically tidy repositories, and example rsync and scp scripts to help manage synchronisation of the expofiles directories which are not under version control. +
There are also scripts running cron jobs on the server to fix file permissions and to periodically tidy repositories, and example rsync and scp scripts to help manage synchronisation of the expofiles directories which are not under version control.
Apart from these scripts, troggle in full deployment also needs
- a running mySQL database,
- a running apache webserver and
- cgit to display git repos.
@@ -50,7 +50,7 @@ We may need this again for Tunnocks/Balkon.
There were perl and python scripts scattered through the :drawings: and :loser: (svx files) repos. Not all of the functions in these scripts have been recreated in more recent scripts or programmed into troggle. - The old scripts have been collected into /scripts or /noinfo in the :expoweb: repository. + The old scripts have been collected into /scripts or /noinfo in the :expoweb: repository.
Survex files contain a reference to the wallet which contains the original survey notes for that surveyed passage. These sometimes have errors and also get out of date as caves get renamed when they get a kataster number issued. Each online survey wallet also has a reference to the survex file(s) which have been typed up from that data. -
Validating the references is checked by scripts check-svx.sh, check-refs.sh, check-refs.awk in the :loser: repository to produce svxvalid.html which lists mismatches between the svx files and the survey scan wallets. +
Validating the references is checked by scripts check-svx.sh, check-refs.sh, check-refs.awk in the :loser: repository to produce svxvalid.html which lists mismatches between the svx files and the survey scan wallets.
This is a horrible proof-of-concept hack that needs replacing with a proper python script instead of an assemblage of awk, bash and sed.
Tunnel files contain references to the wallet which contained the original survey notes for that surveyed and drawn passage. -
The results of validation checks are in xmlvalid.html and generated by script check-xml.sh in the :drawings: repository. +
The results of validation checks are in xmlvalid.html and generated by script check-xml.sh in the :drawings: repository.
(Therion files would too, if people inserted "#Ref" comments. In which case the script would need improving.)
Currently the intermediate data it works from has to be hand-generated so a proper parsing script needs to be written. diff --git a/handbook/troggle/scriptsqms.html b/handbook/troggle/scriptsqms.html index 201c5b255..031720d4e 100644 --- a/handbook/troggle/scriptsqms.html +++ b/handbook/troggle/scriptsqms.html @@ -95,7 +95,7 @@ This will work on all survex *.svx files even those which have not yet been run
This finds references to completed qms in the qm.csv files in the cave folders (/1623/ etc.) in the :expoweb: repository. It looks to see which QMs have been completed but where there is not yet a matching text in the cave description. +
This finds references to completed qms in the qm.csv files in the cave folders (/1623/ etc.) in the :expoweb: repository. It looks to see which QMs have been completed but where there is not yet a matching text in the cave description.
Quick and dirty Python script to find references to completed qms in the cave description pages. Run this to find which bits of description need updating. diff --git a/handbook/troggle/trogmanual.html b/handbook/troggle/trogmanual.html index 256954883..494266b42 100644 --- a/handbook/troggle/trogmanual.html +++ b/handbook/troggle/trogmanual.html @@ -19,7 +19,7 @@- Other scripts - photos, folk, wallets - not integral parts of troggle.
- Troggle and database architecture: how it all fits together
- Design decisions: e.g. new systems for website menus -
- Maintain troggle itself. The code is public on repository :troggle: +
- Maintain troggle itself. The code is public on repository :troggle:
- Uncategorised notes and past speculations
This page is mostly an index to other records of what troggle is and what plans have been made - but never implemented - to improve it. diff --git a/handbook/website-history.html b/handbook/website-history.html index 26dfb4e98..bf15f75e2 100644 --- a/handbook/website-history.html +++ b/handbook/website-history.html @@ -61,9 +61,9 @@ of all caves in a large .CSV file with a cave on each row made the storage of ne more straightforward.
Another important element of this system was version control. The entire data structure was +
Another important element of this system was version control. The entire data structure was stored initially in a Concurrent Version System repository, and later migrated to -Subversion [now using a DVCS in 2019]. +Subversion [now using git in 2020]. Any edits to the spreadsheets which caused the scripts to fail, breaking the website, could be easily reversed.