Editing the expo data management system is an adventure. Learning it by trial and error is non-trivial. There are lots of things we could improve about the system, and anyone with some computer nous is very welcome to muck in. It is slowly getting better organised.
This manual is organized in a how-to sort of style. The categories, rather than referring to specific elements of the data management system, refer to processes that a maintainer would want to do.
Note that to display the survey data you will need a copy of the survex software.
Follow these links if you have reached this page by accident and this is what you want to know:
Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the troggle intro.
Some key sections of the online webpages are autogenerated by scripts or by troggle, and are not static files, so you have to edit the base data, not the generated file (e.g cave pages, QM (question mark) lists, expo members list, prospecting pages). All autogenerated files say 'This file is autogenerated - do not edit' at the top - so check for that before wasting time on changes that will just be overwritten
The primary and recommended way of editing this handbook (and the website generally) is to use a laptop which has the version control software installed. The person editing needs to know how to use this software, and also needs to know how to edit raw HTML files using a text editor. The public expo server is on a machine far, far away that we only access remotely.
The Expo laptop has the software installed, so it is best to learn how to do this when sitting at that laptop.
It's important to understand that the pages you can edit by this method are stored in a version control system (see below). This stops us losing data and makes it very hard for you to screw anything up permanently, so don't worry about making changes - they can always be reverted if there is a problem. It also means that several people can work on the site on different computers at once and normally merge their changes easily.
The recommended editing workflow is to (a) use the version control software to synchronise your local laptop copy of the website files with that on the server; (b) edit a set of .html files on your laptop so that all links between them are consistent, save the files locally, and "commit" them locally; (c) "push" the collection of changes to the expo online server as a single action.
See the Expo data management systems manual for a fuller description of the version control software repositories and how to install and use the software.
You can update a single webpage online via a browser. This is best used for urgent edits to a single page, e.g. if the emergency phone at top-camp has to use a new SIM with a different phone number. If you are a logged-on user you will see "Edit this page" on the menu on the left of this page. It appears on nearly all pages in this website. If you click on it you will be able to edit the raw HTML of the page - so you need to know how to do that.
After doing the page editing and saving your work, you need to ask a nerd to finish the process fairly soon as the "Edit this page" mechanism does not tidy-up after itself properly. See these instructions for this tidy-up
We use a distributed revision control system (git, and formerly mercurial) for all the important data. (Note that we just use git: not GitHub, not GitLab, just git.) This means that many people can edit and merge their changes with the expo server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut and those who have returned to the UK. Also anyone who is up to date can take their laptop somewhere and enter data even if they have no internet access, and the updates will be merged when they get back to civilization.
In principle, survey notes can be typed into a laptop up on the plateau which would then get synchronised when it next gets internet access.
A version control system is inefficient for scanned survey notes which are large files that do not get modified, so they are kept as a plain directory of files 'expofiles'. The same goes for holiday photographs and GPS logs.
In 2019 we had half our version-controlled repositories under mercurial and half under git. The intention is to move entirely to git before the 2020 expo.
You don't need a password to view most things, but you will need one to change them.
Use these credentials for access to the troggle site. The user is 'expo', with a cavey:beery password. Ask someone if this isn't enough clue for you. This password is important for security. The whole site will get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not unencrypted email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.
This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then you will also need to get a login (register a key with the server). See key-pair setup for details.
Pushing cave data to the ::loser:: and ::drawings:: repositories also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are the expo laptop and the laptop 'aziraphale' which live in the potato hut during expo. If you want to use your own laptop then see below.
All the expo data is contained in 4 "repositories" at expo.survex.com. This is currently hosted on a free virtual server we have blagged on a server farm. We use a distributed version control system (DVCS) to manage these repositories because this allows simultaneous collaborative editing and keeps track of all changes so we can roll back and have branches if needed.
The site has been split into four parts:
We have migrated two of these to git but the other two still use mercurial.
Currently (December 2019) after commiting and pushing your changes to expoweb to the mercurial server, you will need to login to expo.survex.com using ssh, cd to /expoweb/ and issue a "hg update" command to make your changes noticed by the webserver. This problem will go away before Expo 2020 - we hope - when we finish migrating from mercurial to git.
All the scans, photos, presentations, fat documents and videos are stored just as files (not in version control) in 'expofiles'. See below for details on that.
Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the troggle intro.
Anything you check in which affects cave data or descriptions won't appear on the site until the data management system update scripts are run. This should happen automatically every 30 mins (not since 2017), but you can also kick off a manual update. See 'The expoweb-update script' below for details.
Also note that the ::expoweb:: web pages and cave data reports you see on the visible website are not the same as the version-controlled "master" expoweb repo. So in order that your committed and pushed changes become visible on the website, they have to be 'pulled' from the repo (on teh server machine) onto the webserver (another place on the same server machine) before your changes are reflected.
Setting your own laptop so that it can do everything the expo laptop can do is quite a complicated process. At a minimum you will be an experienced software nerd already and will have git, mercurial and a text editor installed and you will know how to use them. You will have done the key-pair setup process - which you can only do entirely on your own if you have access to the expo laptop.
See setting up a minimal laptop for a short list of software. This assumes you know how to use it all.
See setting up your own laptop for the full list of software we use and where to get it.
Note that the instructions are primarily for people using Linux with some help for those using Windows. If you are a Mac user then you are on your own.
This can be used to edit web pages without installing any software or doing any key-pair setup. It even works if your laptop is a Mac.
This is the capability that you can see in the top-left-hand menu on any website page if you log in to troggle using the cavey:beery password.
'Edit This Page' is a troggle capability edits the file served by the webserver but it does not update the copy of the file in the repository (the invese of the problem described above as 'Mercurial Website Hack'). To properly finish the job you need to
Again, we hope that this issue will go away when we migrate the expoweb repo from mercurial to git before the 2020 Expo.
To edit the data management system fully, you need to use the version control system software which is currently git and mercurial. Some (static text) pages can be edited directly on-line using the 'edit this page link' which you'll see if you are logged into troggle. In general the dynamically-generated pages, such as those describing caves which are generated from the cave survey data, can not be edited in this way, but forms are provided for some types of these like 'caves'.
If you know what you are doing here is the basic info on what's where:
(if you don't know what you're doing, skip to Editing the data management system below.)
Simple changes to static HTML files will take effect immediately (or as soon as the hg update hack is done, but this will disappear when we move entirely to git), but changes to dynamically-generated files - cave descriptions, QM lists etc. - will not take effect, until a nerd runs the expoweb-update script on the server.
The import scripts for the cave data are currently (Feb.2020) run manually by a nerd. So if you enter cave data,logbooks or survey scans you won't see the result until a nerd has been placated.
Cave description pages are automatically generated from a set of
cave files in noinfo/cave_data/ and noinfo/entrance_data/. These files
are named -
Clicking on 'New cave' (at the bottom of the cave index) lets you enter a new cave. Info on how to enter new caves has been split into its own page.
(If you remember something about CAVETAB2.CSV for editing caves, that was superseded in 2012).
This may be a useful reminder of what is in a survex file how to create a survex file.
Each year's expo is recorded in the folder
/expoweb/years/which contains a number of files used to manage and record that year's expo. Have a look at expoweb/years/2018/ for a recent well-documented expo (the weather was good). Files are added and edited using the version control system for the expoweb repository.
To create a new 'year' for next year's expo see adding a new year.
See the documentation on updating the online surveyscans folders using the lever-arch file of plastic wallets.
See the menu design history and proposals page on where we are and what we might do to improveand fix menus.