CUCC Expedition: Data Maintenance Manual

Expo Data Maintenance Manual

[This page currently being restructured and edited to make it more appropriate to the task. Many fragments have been moved here but not yet properly edited together. Also there are no links to other pages which will be needed.]

Expo data maintenance manual

Editing the expo data management system is an adventure. Learning it by trial and error is non-trivial. There are lots of things we could improve about the system, and anyone with some computer nous is very welcome to muck in. It is slowly getting better organised.

This manual is organized in a how-to sort of style. The categories, rather than referring to specific elements of the data management system, refer to processes that a maintainer would want to do.

Note that to display the survey data you will need a copy of the survex software.

Follow these links if you have reached this page by accident and this is what you want to know:

Updating the online systems - overview

Troggle - what it is

Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the troggle intro.

Version control system

We use a distributed revision control system (git, and formerly mercurial) for all the important data. (Note that we just use git: not GitHub, not GitLab, just git.) This means that many people can edit and merge their changes with the expo server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut and those who have returned to the UK. Also anyone who is up to date can take their laptop somewhere and enter data even if they have no internet access, and the updates will be merged when they get back to civilization.

In principle, survey notes can be typed into a laptop up on the plateau which would then get synchronised when it next gets internet access.

A version control system is inefficient for scanned survey notes which are large files that do not get modified, so they are kept as a plain directory of files 'expofiles'. The same goes for holiday photographs and GPS logs.

In 2019 we had half our version-controlled repositories under mercurial and half under git. The intention is to move entirely to git before the 2020 expo.

Contents

  1. Getting a username, password and key
  2. The repositories
  3. Quick start
  4. Updating cave pages
  5. Updating expo year pages
  6. Maintaining the survey status table
  7. Cave pages and handbook menu system
Appendices:

Getting a username, password and key

You don't need a password to view most things, but you will need one to change them.

Use these credentials for access to the troggle site. The user is 'expo', with a cavey:beery password. Ask someone if this isn't enough clue for you. This password is important for security. The whole site will get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not unencrypted email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.

This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then you will also need to get a login (register a key with the server). See key-pair setup for details.

Pushing cave data to the :loser: and :drawings: repositories also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are the expo laptop and the laptop 'aziraphale' which live in the potato hut during expo. If you want to use your own laptop then see below.

The repositories

All the expo data is contained in 4 "repositories" at expo.survex.com. This is currently hosted on a free virtual server we have blagged on a server farm. We use a distributed version control system (DVCS) to manage these repositories because this allows simultaneous collaborative editing and keeps track of all changes so we can roll back and have branches if needed.

The site has been split into four parts:

We have migrated two of these to git but the other two still use mercurial.

Updating cave pages

Cave description pages are automatically generated from a set of cave files in noinfo/cave_data/ and noinfo/entrance_data/. These files are named -.html (where area is 1623 or 1626). These files are processed by troggle. Use python databaseReset.py caves in /expofiles/troggle/ to update the site/database after editing these files. Use python databaseReset.py on its own to get a list of command options and brief explanations.

Clicking on 'New cave' (at the bottom of the cave index) lets you enter a new cave. Info on how to enter new caves has been split into its own page.

(If you remember something about CAVETAB2.CSV for editing caves, that was superseded in 2012).

This may be a useful reminder of what is in a survex file how to create a survex file.

Updating expo year pages

Each year's expo is recorded in the folder

/expoweb/years/

which contains a number of files used to manage and record that year's expo. Have a look at expoweb/years/2018/ for a recent well-documented expo (the weather was good). Files are added and edited using the version control system for the expoweb repository.

To create a new 'year' for next year's expo see adding a new year.

Maintaining the survey status table

See the documentation on updating the online surveyscans folders using the lever-arch file of plastic wallets.

Cave pages and handbook menu design

See the menu design history and proposals page on where we are and what we might do to improveand fix menus.