expoweb/handbook/computing/manual.html

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>CUCC Expedition: Data Maintenance Manual</title>
<link rel="stylesheet" type="text/css" href="../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition: Data Maintenance Manual</h2>
<h1>Expo Data Maintenance Manual</h1>

<p><span style="color:red">[This page currently being restructured and edited to make it more appropriate to the task. Many fragments have been moved here but not yet properly edited together. Also there are no links to other pages which will be needed.]
</span>

<h2><a id="manual">Expo data maintenance manual</a></h2>

<ul>
<li>This page is for cavers wanting to edit any expo data on their own machine. So that's the website, survey data, or drawn up surveys</li>
<li>This page is <i>not</i> for cavers wanting to know how to type in logbooks or upload photographs or edit data on the expo laptop.

<li>There is another page with more info on <a href="#yourownlaptop">the software expo uses and setting up your own laptop</a>.
</ul>

<p>Editing the expo data management system is an adventure.  Learning
it by trial and error is non-trivial. There are lots of things we
could improve about the system, and anyone with some computer nous is
very welcome to muck in. It is slowly getting better organised.</p>

<p>This manual is organized in a how-to sort of style. The categories,
rather than referring to specific elements of the data management system, refer to
processes that a maintainer would want to do.</p>
<p>Note that to display the survey data you will need a copy of the <a href="../getsurvex.html">survex</a> software.

<p>Follow these links if you have reached this page by accident and this is what you want to know:
<ul>
<li><a href="../uploading.html">How to upload photos</a></li>
<li><a href="../logbooks.html">Typing in logbook entries</a></li>
<li><a href="../gpxupload.html">Recording the GPS location of a cave</a></li>
<li><a href="../survey/index.htm">How to do cave surveying</a></li>
<li><a href="../index.htm">List of "How to" pages for everything else</a></li>
</ul>

<ul>
<li><a href="newyear.html">Manual: Creating a new 'year' in the system</a></li>
<li><a href="">Manual: Expo survey data maintenance manual</a></li>
<li><a href="manual.html#update">Manual: Updating the cave guidebook descriptions</a></li>
</ul>
<h2><a id="update">Updating the online systems - overview</a></h2>

<h3 id="troggle">Troggle - what it is</a></h3>
<p>Troggle runs  the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the <a href="../troggle/trogintro.html">troggle intro</a>. 

<h3 id="git">Version control system</a></h3>
<p>We use a distributed revision control system (<a href="https://git-scm.com/about/distributed">git</a>, and formerly mercurial) for all the important data. (Note that we just use git: not GitHub, not GitLab, just <a href="https://git-scm.com/about/distributed">git</a>.)
This means that many people can edit and merge their changes with the expo 
server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut 
and those who have returned to the UK.  Also anyone who is up 
to date can take their laptop somewhere and enter data even if they have no internet access, 
and the updates will be merged when they get back to civilization.
</p>
<p>In principle, survey notes can be typed into a laptop up on the plateau which would 
then get synchronised when it next gets internet access.
</p>
<p>A version control system is inefficient for scanned survey notes which are large files that 
do not get modified, so they are kept as a plain directory of files 'expofiles'. 
The same goes for holiday photographs and GPS logs.</p>

<p>In 2019 we had half our <a href="manual.html#repositories">version-controlled repositories</a> under mercurial and half under git. 
The intention is to move entirely to git before the 2020 expo.
<h3>Contents </h3>

<ol>
<li><a href="#usernamepassword">Getting a username, password and key</a></li>
<li><a href="#repositories">The repositories</a></li>
<li><a href="#quickstart">Quick start</a></li>
<li><a href="#cavepages">Updating cave pages</a></li>
<li><a href="#updatingyears">Updating expo year pages</a></li>
<li><a href="#surveystatus">Maintaining the survey status table</a></li>
<li><a href="#menu">Cave pages and handbook menu system</a></li>

</ol>
Appendices:
<ul>
<li><a href="../website-history.html">Website history</a> - a history of the data management system up to 2019</li>
<li><a href="../c21bs.html">Taking Expo Bullshit into the 21st Century</a> - initial report from 1996</li>
<li><a href="useany.html">Platform portability</a> - making the website work widely [2006]</li>
</ul>

<h3><a id="usernamepassword">Getting a username, password and key</a></h3>

<p>You don't need a password to view most things, but you will need one to change them.</p>

<p>Use these credentials for access to the troggle site. The user is 'expo',
  with a cavey:beery password. Ask someone if this isn't enough clue for you.
  <b>This password is important for security</b>. The whole site <strong>will</strong> get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not unencrypted email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.
</p>

<p>This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then 
you will also need to get a login (register a key with the server). See <a href="keyexchange.html">key-pair setup</a> for details.

<p>Pushing cave data to the :loser: and :drawings: repositories also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are 
the <i>expo laptop</i> and the laptop '<i>aziraphale</i>' which live in the potato hut during expo. If you want to use your own laptop then 
see <a href="#yourownlaptop">below</a>.


<h3><a id="repositories">The repositories</a></h3>

<p>All the expo data is contained in 4 "repositories" at
expo.survex.com. This is currently hosted on a free virtual server we have blagged on a server farm. 
We use a distributed version control system (DVCS) to manage these repositories because this allows simultaneous collaborative 
editing and keeps track of all changes so we can roll back and have branches if needed.</p>

<p>The site has been split into four parts:</p>

<ul>
 <li><a href="/repositories/home/expo/loser/graph/">loser</a> - the survex cave survey data (hg)</li>
 <li><a href="/cgit/drawings/.git/log">drawings</a> - the tunnel and therion cave data and drawings (git)</li>
 <li><a href="/repositories/home/expo/expoweb/graph">expoweb</a> - the website pages, handbook, generation scripts (hg)</li>
 <li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="../troggle/trogintro.html">notes on troggle</a> for further explanation (git)</li>
</ul>
<p>We have migrated two of these to git but the other two still use mercurial.


<h3><a id="cavepages">Updating cave pages</a></h3>
<span style="color:red">
<p>Public cave description pages are automatically generated by troggle from a set of
cave files in noinfo/cave_data/ and noinfo/entrance_data/. These files
are named <area>-<cavenumber>.html (where area is 1623 or 1626). 
<p>
Read the survey handbook section on <a href="../survey/caveentry.html">creating a new cave</a> in the system for instructions on how to name caves and the files you use to recoird them.
<p>Cave names do not have leading zeros
They are stored by number/ID in the dataset, not by name.
<p>Caves with a provisional number consisting of a year and a serial number
  should be hyphenated, thus 2002-04 not 2002_04 or any of the various other
  variants
  
<p>Use <tt>python databaseReset.py
caves</tt> in /expofiles/troggle/ to update the site/database after
editing these files.
Use <tt>python databaseReset.py
</tt> on its own to get a list of command options and brief explanations.</p>

<p>Clicking on 'New cave' (at the bottom of the cave index) lets you enter a new cave. <a href="caveentry.html">Info on how to enter new caves has been split into its own page</a>.</p>

<p>(If you remember something about CAVETAB2.CSV for editing caves, that was 
superseded in 2012).</p>
<p>This may be a useful reminder of what is in a survex file <a href="../survey/how_to_make_a_survex_file.pdf">how to create a survex file</a>.


<h3><a id="updatingyears">Updating expo year pages</a></h3>

<p>Each year's expo is recorded in the folder</p>

<tt>/expoweb/years/</tt>

<p>which contains a number of files used to manage and record that year's expo. Have a look at 
expoweb/years/2018/ for a recent well-documented expo (the weather was good). 
Files are added and edited using the version control system for the expoweb repository.</p>

<p>To create a new 'year' for next year's expo see <a href="newyear.html">adding a new year</a>.


<h3><a id="surveystatus">Maintaining the survey status table</a></h3>
<p>See the <a href="../survey/onlinewallet.html">documentation</a> on updating the online surveyscans folders using the lever-arch file of plastic wallets.

<h3><a id="menu">Cave pages and handbook menu design</a></h3>
<p>See the <a href="menudesign.html">menu design history and proposals</a> 
page on where we are and what we might do to improveand fix menus.
</span>

<hr />
</body>
</html>
missed one 2020-04-09 19:14:09 +01:00			`<!DOCTYPE html>`
			`<html>`
			`<head>`
			`<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />`
handbook editing manual 2020-04-10 13:09:12 +01:00			`<title>CUCC Expedition: Data Maintenance Manual</title>`
missed one 2020-04-09 19:14:09 +01:00			`<link rel="stylesheet" type="text/css" href="../css/main2.css" />`
			`</head>`
			`<body>`
handbook editing manual 2020-04-10 13:09:12 +01:00			`<h2 id="tophead">CUCC Expedition: Data Maintenance Manual</h2>`
missed one 2020-04-09 19:14:09 +01:00			`<h1>Expo Data Maintenance Manual</h1>`

handbook editing manual 2020-04-10 13:09:12 +01:00			`<p><span style="color:red">[This page currently being restructured and edited to make it more appropriate to the task. Many fragments have been moved here but not yet properly edited together. Also there are no links to other pages which will be needed.]`
			`</span>`

			`<h2><a id="manual">Expo data maintenance manual</a></h2>`
missed one 2020-04-09 19:14:09 +01:00
			`<ul>`
			`<li>This page is for cavers wanting to edit any expo data on their own machine. So that's the website, survey data, or drawn up surveys</li>`
			`<li>This page is <i>not</i> for cavers wanting to know how to type in logbooks or upload photographs or edit data on the expo laptop.`

			`<li>There is another page with more info on <a href="#yourownlaptop">the software expo uses and setting up your own laptop</a>.`
			`</ul>`

			`<p>Editing the expo data management system is an adventure. Learning`
			`it by trial and error is non-trivial. There are lots of things we`
			`could improve about the system, and anyone with some computer nous is`
			`very welcome to muck in. It is slowly getting better organised.</p>`

			`<p>This manual is organized in a how-to sort of style. The categories,`
			`rather than referring to specific elements of the data management system, refer to`
			`processes that a maintainer would want to do.</p>`
			`<p>Note that to display the survey data you will need a copy of the <a href="../getsurvex.html">survex</a> software.`

			`<p>Follow these links if you have reached this page by accident and this is what you want to know:`
			`<ul>`
			`<li><a href="../uploading.html">How to upload photos</a></li>`
			`<li><a href="../logbooks.html">Typing in logbook entries</a></li>`
			`<li><a href="../gpxupload.html">Recording the GPS location of a cave</a></li>`
			`<li><a href="../survey/index.htm">How to do cave surveying</a></li>`
			`<li><a href="../index.htm">List of "How to" pages for everything else</a></li>`
			`</ul>`

Simplifying the systems overview, restructing long documents more sensibly 2020-04-09 21:58:40 +01:00			`<ul>`
			`<li><a href="newyear.html">Manual: Creating a new 'year' in the system</a></li>`
			`<li><a href="">Manual: Expo survey data maintenance manual</a></li>`
			`<li><a href="manual.html#update">Manual: Updating the cave guidebook descriptions</a></li>`
			`</ul>`
			`<h2><a id="update">Updating the online systems - overview</a></h2>`

			`<h3 id="troggle">Troggle - what it is</a></h3>`
			`<p>Troggle runs the expo cave survey data management, presents the data on the website and manages the Expo Handbook. See the <a href="../troggle/trogintro.html">troggle intro</a>.`

			`<h3 id="git">Version control system</a></h3>`
			`<p>We use a distributed revision control system (<a href="https://git-scm.com/about/distributed">git</a>, and formerly mercurial) for all the important data. (Note that we just use git: not GitHub, not GitLab, just <a href="https://git-scm.com/about/distributed">git</a>.)`
			`This means that many people can edit and merge their changes with the expo`
grey banner and lots of to-do fixes, to do items all updated from scribbled notes on printout 2020-04-22 19:37:10 +01:00			`server in Cambridge at the same time: inlcuding people still on expo in the Tatty Hut`
Simplifying the systems overview, restructing long documents more sensibly 2020-04-09 21:58:40 +01:00			`and those who have returned to the UK. Also anyone who is up`
			`to date can take their laptop somewhere and enter data even if they have no internet access,`
grey banner and lots of to-do fixes, to do items all updated from scribbled notes on printout 2020-04-22 19:37:10 +01:00			`and the updates will be merged when they get back to civilization.`
Simplifying the systems overview, restructing long documents more sensibly 2020-04-09 21:58:40 +01:00			`</p>`
			`<p>In principle, survey notes can be typed into a laptop up on the plateau which would`
			`then get synchronised when it next gets internet access.`
			`</p>`
			`<p>A version control system is inefficient for scanned survey notes which are large files that`
			`do not get modified, so they are kept as a plain directory of files 'expofiles'.`
			`The same goes for holiday photographs and GPS logs.</p>`

grey banner and lots of to-do fixes, to do items all updated from scribbled notes on printout 2020-04-22 19:37:10 +01:00			`<p>In 2019 we had half our <a href="manual.html#repositories">version-controlled repositories</a> under mercurial and half under git.`
			`The intention is to move entirely to git before the 2020 expo.`
handbook editing manual 2020-04-10 13:09:12 +01:00			`<h3>Contents </h3>`
missed one 2020-04-09 19:14:09 +01:00
			`<ol>`
			`<li><a href="#usernamepassword">Getting a username, password and key</a></li>`
			`<li><a href="#repositories">The repositories</a></li>`
			`<li><a href="#quickstart">Quick start</a></li>`
			`<li><a href="#cavepages">Updating cave pages</a></li>`
			`<li><a href="#updatingyears">Updating expo year pages</a></li>`
			`<li><a href="#surveystatus">Maintaining the survey status table</a></li>`
			`<li><a href="#menu">Cave pages and handbook menu system</a></li>`

			`</ol>`
			`Appendices:`
			`<ul>`
			`<li><a href="../website-history.html">Website history</a> - a history of the data management system up to 2019</li>`
			`<li><a href="../c21bs.html">Taking Expo Bullshit into the 21st Century</a> - initial report from 1996</li>`
grey banner and lots of to-do fixes, to do items all updated from scribbled notes on printout 2020-04-22 19:37:10 +01:00			`<li><a href="useany.html">Platform portability</a> - making the website work widely [2006]</li>`
missed one 2020-04-09 19:14:09 +01:00			`</ul>`

			`<h3><a id="usernamepassword">Getting a username, password and key</a></h3>`

			`<p>You don't need a password to view most things, but you will need one to change them.</p>`

			`<p>Use these credentials for access to the troggle site. The user is 'expo',`
			`with a cavey:beery password. Ask someone if this isn't enough clue for you.`
			`<b>This password is important for security</b>. The whole site <strong>will</strong> get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not unencrypted email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.`
			`</p>`

			`<p>This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then`
			`you will also need to get a login (register a key with the server). See <a href="keyexchange.html">key-pair setup</a> for details.`

::expoweb:: replaced by :expoweb: to indicate a repo 2020-04-11 17:46:18 +01:00			`<p>Pushing cave data to the :loser: and :drawings: repositories also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are`
missed one 2020-04-09 19:14:09 +01:00			`the <i>expo laptop</i> and the laptop '<i>aziraphale</i>' which live in the potato hut during expo. If you want to use your own laptop then`
			`see <a href="#yourownlaptop">below</a>.`


			`<h3><a id="repositories">The repositories</a></h3>`

			`<p>All the expo data is contained in 4 "repositories" at`
			`expo.survex.com. This is currently hosted on a free virtual server we have blagged on a server farm.`
			`We use a distributed version control system (DVCS) to manage these repositories because this allows simultaneous collaborative`
			`editing and keeps track of all changes so we can roll back and have branches if needed.</p>`

			`<p>The site has been split into four parts:</p>`

			`<ul>`
grey banner and lots of to-do fixes, to do items all updated from scribbled notes on printout 2020-04-22 19:37:10 +01:00			`<li><a href="/repositories/home/expo/loser/graph/">loser</a> - the survex cave survey data (hg)</li>`
			`<li><a href="/cgit/drawings/.git/log">drawings</a> - the tunnel and therion cave data and drawings (git)</li>`
			`<li><a href="/repositories/home/expo/expoweb/graph">expoweb</a> - the website pages, handbook, generation scripts (hg)</li>`
			`<li><a href="/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="../troggle/trogintro.html">notes on troggle</a> for further explanation (git)</li>`
missed one 2020-04-09 19:14:09 +01:00			`</ul>`
grey banner and lots of to-do fixes, to do items all updated from scribbled notes on printout 2020-04-22 19:37:10 +01:00			`<p>We have migrated two of these to git but the other two still use mercurial.`
missed one 2020-04-09 19:14:09 +01:00

			`<h3><a id="cavepages">Updating cave pages</a></h3>`
handbook editing manual 2020-04-10 13:09:12 +01:00			`<span style="color:red">`
Documenting the weird and wonderful set of scripts we have 2020-04-11 23:48:36 +01:00			`<p>Public cave description pages are automatically generated by troggle from a set of`
missed one 2020-04-09 19:14:09 +01:00			`cave files in noinfo/cave_data/ and noinfo/entrance_data/. These files`
Documenting the weird and wonderful set of scripts we have 2020-04-11 23:48:36 +01:00			`are named <area>-<cavenumber>.html (where area is 1623 or 1626).`
			`<p>`
			`Read the survey handbook section on <a href="../survey/caveentry.html">creating a new cave</a> in the system for instructions on how to name caves and the files you use to recoird them.`
			`<p>Cave names do not have leading zeros`
			`They are stored by number/ID in the dataset, not by name.`
			`<p>Caves with a provisional number consisting of a year and a serial number`
			`should be hyphenated, thus 2002-04 not 2002_04 or any of the various other`
			`variants`

			`<p>Use <tt>python databaseReset.py`
missed one 2020-04-09 19:14:09 +01:00			`caves</tt> in /expofiles/troggle/ to update the site/database after`
			`editing these files.`
			`Use <tt>python databaseReset.py`
			`</tt> on its own to get a list of command options and brief explanations.</p>`

			`<p>Clicking on 'New cave' (at the bottom of the cave index) lets you enter a new cave. <a href="caveentry.html">Info on how to enter new caves has been split into its own page</a>.</p>`

			`<p>(If you remember something about CAVETAB2.CSV for editing caves, that was`
			`superseded in 2012).</p>`
			`<p>This may be a useful reminder of what is in a survex file <a href="../survey/how_to_make_a_survex_file.pdf">how to create a survex file</a>.`


			`<h3><a id="updatingyears">Updating expo year pages</a></h3>`

			`<p>Each year's expo is recorded in the folder</p>`

			`<tt>/expoweb/years/</tt>`

			`<p>which contains a number of files used to manage and record that year's expo. Have a look at`
			`expoweb/years/2018/ for a recent well-documented expo (the weather was good).`
			`Files are added and edited using the version control system for the expoweb repository.</p>`

			`<p>To create a new 'year' for next year's expo see <a href="newyear.html">adding a new year</a>.`



			`<h3><a id="surveystatus">Maintaining the survey status table</a></h3>`
			`<p>See the <a href="../survey/onlinewallet.html">documentation</a> on updating the online surveyscans folders using the lever-arch file of plastic wallets.`

			`<h3><a id="menu">Cave pages and handbook menu design</a></h3>`
			`<p>See the <a href="menudesign.html">menu design history and proposals</a>`
			`page on where we are and what we might do to improveand fix menus.`
handbook editing manual 2020-04-10 13:09:12 +01:00			`</span>`
missed one 2020-04-09 19:14:09 +01:00
			`<hr />`
			`</body>`
			`</html>`