expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Setting up your own laptop with all the software on the expo laptop

Browse Source
This commit is contained in:
Philip Sargent 2019-12-04 23:39:41 +00:00
parent 5cf3070c1a
commit 8f76a6176c
6 changed files with 186 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

117
handbook/computing/yourlaptop.html Normal file
View File

@ -0,0 +1,117 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>CUCC Expedition Handbook: Programmers manual</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook - Your laptop</h2>
<h1>Setting up an Expo laptop</h1>
<h3>Operating Systems</h3>
<ul>
<li>The quickest way to get a complete setup is to use a <b>Debian Linux laptop</b>.
<br>
The <i>expo laptop</i> uses <a href="https://www.debian.org/intro/about">Debian</a> with the <a href="https://computingforgeeks.com/how-to-install-cinnamon-desktop-environment-on-debian/">Cinnamon</a> interface, but pretty much any Linux system works fine. This handbook does assume that you are using apt - the Debian package manager - which is good for all Debian-derived Linuxes such as <a href="https://ubuntu.com/">Ubuntu</a>. Old, slow machines without much memory can be very effective with <a href="https://xubuntu.org/">Xubuntu/xfce</a>.
<li>Windows machines can do everything needed, but some useful software has no Windows version and you will need to find your own equivalents. There are also dangerous 'gotchas' to look out for because the file-naming system is different. Do use Linux instead if you can.
<li>WSL: the Windows Subsystem for Linux. The first release of this didn't do the ssh key exchange process easily. The <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">2019 WSL2 release</a> includes a complete Linux kernel. If you want to use this, then please do - and write the handbook documentation too. But beware, it can work in <a href="https://docs.microsoft.com/en-us/windows/wsl/wsl2-index">two different modes</a> which behave differently.
<li>Mac users will need to use the Linux documentation as a guide and work it all out for themselves.
<li>Android phone apps can be very handy too.
</ul>
<h3>Software</h3>
<p>Long-standing Expo policy is to retain absolute control of all software and all data. So we use <a href="https://en.wikipedia.org/wiki/Free_and_open-source_software">FOSS software</a> and we store data in text files not in a database. You can use other software on your own machine if it is format-compatible and exports data in the formats we want, but all the recommended software here is open source and please don't install proprietary software on the '<em>expo laptop</em>'.
<p>The list of software:
<ul>
<li>mercurial
<li>git
<li>TortoiseHg
<li>scp and ftp GUI software, such as <a href="../fzconfig.html">Filezilla</a> - with a configuration file to get to the expo server
<li><a href="https://tortoisegit.org/support/faq/#prerequisites">TortoiseGit</a> - GUI interface to git. Optional.
<li>Survex, including the Aven visualisation tool.
<li><a href="https://github.com/CaveSurveying/tunnelx">Tunnel</a>: a Java 2.5D cave drawing program surveys based on Survex-compatible data which can also read PocketTopo files. (Generally called 'tunnel' even though the project and executable is actually 'tunnelx'.)
<li><a href="https://therion.speleo.sk/">therion.speleo.sk/</a>Therion - Therion processes survey data and generates maps or 3D models of caves.
<li><a href="https://www.qgis.org/en/site/">QGIS</a> - A Free and Open Source Geographic Information System
<li><a href="https://activityworkshop.net/software/gpsprune/development.html">GPSprune</a> - to edit GPS tracks in GPX files. Just download the .jar file.
<li><a href="https://www.java.com/en/">java</a> - needed to run GPSprune
<li><a href="https://sourceforge.net/projects/viking/">Viking</a> - an alternative to GPSprune. The <a href="https://github.com/viking-gps/viking">code</a> is on github.
<li>python, if you are extending troggle or the scripts in expoweb/noinfo/ - already installed on Linux
<li>python packages you will want to install: [<em>not yet documented</em>]
</ul>
<p>Nearly all the Austrian survey has beeen produced using Tunnel but we are moving to Therion for new caves because Therion does elevations properly and Tunnel never will.
<p>For Windows users only:
<ul>
<li><a href="https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html">PuTTY</a> including pagent. Version 0.73 was released on 2019-09-29. You need this to generate and to use ssh keys on Windows.
<li><a href="https://gitforwindows.org/">Git for Windows</a>
<li><a href="https://www.python.org/downloads/windows/">python for windows</a>. We are using python 2.7 not 3.8.
<li>WinScp can be used instead of Filezilla, see
</ul>
<p>For Android phones:
<ul>
<li><a href="ftpusage.html">FTP using Cx_File_Explorer</a> (this is probably out of date)
<li><a href="https://www.termius.com/">Terminus</a> - command line ssh to login to the expo server.
<li>OsmAnd - GPS app. See separate pages on using GPS software
<li><a href="../uploading.html#android">andftp</a> app
<li><a href="https://sites.google.com/site/speleoapps/home">TopoDroid</a> makes cave surveys with the DistoX.
<li><a href="https://sites.google.com/site/speleoapps/home">Cave3D</a> is a Therion 3D viewer.
<li><a href="https://sites.google.com/site/speleoapps/home">ThManager</a> organizes single surveys, exported by TopoDroid, into Therion projects encoded by Therion "thconfig" files.
</ul>
<h3>Logins to external systems</h3>
<ul>
<li><a href="https://github.com/join">Github</a> - create your own github account if you don't have one already.
<li><a href="https://bitbucket.org/product/">Bitbucket</a> - create an account to help develop tunnel.
<li><a href="https://trac.survex.com/wiki">Trac issue tracker and wiki</a> for survex
<li><a href="https://launchpad.net/">Launchpad</a> - create an account to help develop survex.
</ul>
<p>Bug lists and open issues are discussed on the github <a href="https://github.com/CaveSurveying/CUCCexposurveyissues/issues">CaveSurveying/CUCCexposurveyissues/issues</a> issue list so you will need to subscribe to the Cave Surveying Group on github to participate.
<p><a href="https://github.com/CaveSurveying/CUCCexposurveyissues/wiki/Expo-tunnel-workflows">github.com/CaveSurveying/CUCCexposurveyissues/wiki/Expo-tunnel-workflows</a> is a wiki on github discussing workflows to generate centerlines, GEOTIFF and QGIS integration.
<p><a href="https://launchpad.net/survex">launchpad.net/survex</a> - the main Survex development system.
<h3>Configuration</h3>
You need to do the
<a href="computing/keyexchange.html">key exchange</a> process - which you can only do entirely on your own if
you have access to the <i>expo laptop</i> to upload and install the public key generated by your laptop.
<p>On a Windows machine you will need to configure pageant (the putty authentication agent)
to <a href="https://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html">run at startup to load your key</a>. Note that you are loading your <em>private</em> key, the .ppk file, into pageant and that this key never leaves your laptop.
<p>When using Windows please be <a href="http://expo.survex.com/handbook/survey/getin.htm#filenames">excessively careful when naming files and survex names</a>.
<p>The handbook has extensive documentation on when and where it is necessary to use scp and ftp to manage large files in 'expofiles'. See <a href="../upload-expert.html">Experts: Uploading files</a>, a href="../uploading.html">Uploading files</a> and <a href="../gpxupload.html">Uploading GPS tracks</a>. Only machines which have done the key exchange process can do scp, ftp or rsync.
<h3>Learning how to use this software</h3>
<img src="/expofiles/tunnelwiki/wiki/images/6/65/Banner204.png" align="right" width=100>
<p>For Survex, Tunnel and Therion, see the <a href="http://expo.survex.com/handbook/survey/">Expo Surveying Handbook</a>.
<p><em>For installing Survex, Tunnel etc. see <a href="../getsurvex.html">this page</a> which will be merged in here eventually.
<p><a href="http://expo.survex.com/expofiles/tunnelwiki/wiki/pages/Tunnel.html">The Tunnel tutorial</a> - installation notes and a wiki of examples and tutorials
<p><a href="https://bitbucket.org/goatchurch/tunnelx/src/default/">bitbucket.org/goatchurch/tunnelx</a> - documentation and source code in the bitbucket repository system.
<p>Quick <a href="qstart-hg.html">reminders for using mercurial</a> at the command line.
<p>Full instructions for <a href="../tortoise/tortoise-win.htm">installing TortoiseHg and PuTTy on Windows</a>
<hr />
<div id="menu">
<ul id="links">
<li><a href="../index.htm">Handbook</a>
<li><a href="../../infodx.htm">Main index</a></li>
</ul>
</div>
</body>
</html>

64
handbook/manual.html
View File

@ -13,11 +13,10 @@
<li>This page is <i>not</i> for cavers wanting to know how to record their cave survey data.
<li>This page is <i>not</i> for cavers wanting to know how to type in logbooks or upload photographs.
<li>This page <i>is for programmers</i> who are modifying the software or helping cavers do their thing.
<li>This page <i>is for programmers</i> who are helping cavers do their thing and <a href="#yourownlaptop">setting up their own laptop</a>.
</ul>
<p>Editing the expo data management system is an adventure. Until 2007, there was no
guide which explained the whole thing as a functioning system. Learning
<p>Editing the expo data management system is an adventure. Learning
it by trial and error is non-trivial. There are lots of things we
could improve about the system, and anyone with some computer nous is
very welcome to muck in. It is slowly getting better organised.</p>
@ -36,9 +35,10 @@ processes that a maintainer would want to do.</p>
<h3>Contents of this manual</h3>
<ol>
<li><a href="#usernamepassword">Getting a username and password</a></li>
<li><a href="#usernamepassword">Getting a username, password and key</a></li>
<li><a href="#repositories">The repositories</a></li>
<li><a href="#howitworks">How the data management system works</a></li>
<li><a href="#yourownlaptop">Your own laptop</a></li>
<li><a href="#quickstart">Quick start</a></li>
<li><a href="#editingthedata management system">Modifying the data management system</a></li>
<li><a href="#expowebupdate">The expoweb-update script</a></li>
@ -55,20 +55,22 @@ Appendices:
<li><a href="c21bs.html">Taking Expo Bullshit into the 21st Century</a> - initial report from 1996</li>
</ul>
<h3><a id="usernamepassword">Getting a username and password</a></h3>
<h3><a id="usernamepassword">Getting a username, password and key</a></h3>
<p>Use these credentials for access to the site. The user is 'expo',
<p>You don't need a password to view most things, but you will need one to change them.</p>
<p>Use these credentials for access to the troggle site. The user is 'expo',
with a cavey:beery password. Ask someone if this isn't enough clue for you.
<b>This password is important for security</b>. The whole site <strong>will</strong> get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not unencrypted email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.
</p>
<p>Note that you don't need a password to view most things, but you will need one to change them</p>
<p>This password is all you need to log in to troggle and to use the control panel. But if you want to edit the software itself, or update webpages, then
<p>This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then
you will also need to get a cryptographic key and register it with the server. See <a href="computing/keyexchange.html">key exchange</a> for details.
<p>Unfortunately, pushing cave data to ::loser:: and ::drawings:: also needs a key. So cavers entering their cave survey data
<p>Unfortunately, pushing cave data to the ::loser:: and ::drawings:: repositories also needs a key. So cavers entering their cave survey data
currently have to use a machine on which this already set up. These machines are
the <i>expo laptop</i> and the laptop '<i>aziaphale</i>' which live in the potato hut during expo.
the <i>expo laptop</i> and the laptop '<i>aziraphale</i>' which live in the potato hut during expo. If you want to use your own laptop then
see <a href="#yourownlaptop">below</a>.
<h3><a id="repositories">The repositories</a></h3>
@ -86,8 +88,11 @@ editing and keeps track of all changes so we can roll back and have branches if
<li><a href="http://expo.survex.com/repositories/home/expo/expoweb/graph">expoweb</a> - the website pages, handbook, generation scripts</li>
<li><a href="http://expo.survex.com/cgit/troggle/.git/log">troggle</a> - the database/software part of the survey data management system - see <a href="troggle-ish.html">notes on troggle</a> for further explanation</li>
</ul>
<p>We have migrated two of these to git but the other two still use mercurial.
<p>In 2019 we are in the process of migrating from mercurial to git for the repos.
<h4>Mercurial Website Hack 2019</h4>
<p> Currently (December 2019) after commiting and pushing your changes to expoweb to the mercurial server, you will need to
login to expo.survex.com using ssh, cd to /expoweb/ and issue a "<a href="https://www.selenic.com/mercurial/hg.1.html">hg update</a>" command to make your changes noticed by the webserver. This problem will go away before Expo 2020 - we hope - when we finish migrating from mercurial to git.
<p>All the scans, photos, presentations, fat documents and videos are
stored just as files (not in version control) in 'expofiles'. See
@ -95,8 +100,18 @@ below for details on that.</p>
<h3><a id="howitworks">How the data management system works</a></h3>
<p>Part of the data management system is static HTML, but quite a lot is generated by scripts.
So anything you check in which affects cave data or descriptions won't appear on the site until
<p>Part of the data management system is static HTML, but quite a lot is generated by scripts and troggle (a web framework built using Django).
<p>Examples of troggle-generated pages from data:
<ul>
<li><a href="http://expo.survex.com/caves">expo.survex.com/caves</a> - list of caves surveyed and links to guidebook descriptions
<li>< href="http://expo.survex.com/pubs.htm">http://expo.survex.com/pubs.htm</a> - reports, accounts and logbooks
<li><a href="http://expo.survex.com/expedition/2018">expo.survex.com/expedition/2018</a> - Members on expo 2018: . Scroll down for a list of all the data typed in from survey trips.
<li><a href="http://expo.survex.com/survexfile/caves/">expo.survex.com/survexfile/caves/</a> - List of caves with all the surveys done for each.
<li><a href="http://expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx">expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx</a> - Cave survey data from 1983 in Schnellzughohle.
<li><a href="http://expo.survex.com/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="http://expo.survex.com/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
<p>Anything you check in which affects cave data or descriptions won't appear on the site until
the data management system update scripts are run.
This happens automatically every 30 mins, but you can also kick off a manual update.
See 'The expoweb-update script' below for details.</p>
@ -106,25 +121,36 @@ are not the same as the version-controlled "master" expoweb repo.
So in order that your committed and pushed changes become visible on the website,
they have to be 'pulled' from the repo onto the webserver before your changes are reflected.</p>
<h3><a id="yourownlaptop">Your own laptop</a></h3>
<p>Setting your own laptop so that it can do everything the <i>expo laptop</i> can do is quite a
complicated process. At a minimum you will be an experienced software nerd already and will have git, mercurial and a text editor installed and you will know how to use them.
You will have done the
<a href="computing/keyexchange.html">key exchange</a> process - which you can only do entirely on your own if
you have access to the <i>expo laptop</i>.
<p>See <a href="computing/yourlaptop.html">setting up your own laptop</a> for the full list of software we use and where to get it.
<p>Note that the instructions are primarily for people using Linux with some help for those using Windows. If you are a Mac user then you are on your own.
<h3><a id="editthispage">Using 'Edit This Page'</a></h3>
<p>This edits the file served by the webserver on
the expo server but it does not update the copy of the file in the
repository. To properly finish the job you need to
<p>This can be used to edit web pages without installing any software or doing any key exchange. It even works if your laptop is a Mac.
<p>This is the capability that you can see in the top-left-hand menu on any website page if you <a href="/accounts/login/">log in to troggle</a> using the <a href="#usernamepassword">cavey:beery password</a>.
<p>'Edit This Page' is a troggle capability edits the file served by the webserver but it does not update the copy of the file in the
repository (the invese of the problem described above as 'Mercurial Website Hack'). To properly finish the job you need to
<ul>
<li>
ssh into expo@expo.survex.com (use putty on a Windows machine)
<li>cd to the directory containing the repo you want, i.e. "cd loser" for
cave data or "cd expoweb" for the handbook and visible data management system, which takes you to /home/expo/expoweb
<li>Then run "hg status" (to check what
<li>Then run "<a href="https://www.selenic.com/mercurial/hg.1.html">hg status</a>" (to check what
changes are pending),
<li>then "hg diff" to see the changes in detail
(or "hg diff|less" if you know how to use "less" or "more") and
<li>then DO NOT just run 'hg commit' unless you know how emacs works as it will dump
<li>then DO NOT just run '<a href="https://www.selenic.com/mercurial/hg.1.html">hg commit</a>' unless you know how emacs works as it will dump
you into an emacs editing window (C-x C-C is the way to exit emacs). Instead, do
'hg commit -m "found files left over - myName" '
which submits the obligatory comment witht he commit operation.
</ul>
<p>Again, we hope that this issue will go away when we migrate the expoweb repo from mercurial to git before the 2020 Expo.
<h3><a id="quickstart">Quick start</a></h3>

10
handbook/onlinesystems.html
View File

@ -116,6 +116,16 @@ The same goes for holiday photographs and GPS logs.</p>
Troggle is the software collection (not really a "package") based on <a href="https://www.djangoproject.com/">Django</a>
originally intended to manage all expo data in a logical and accessible way
and publish it on the web.
<p>Examples of troggle-generated pages from data:
<ul>
<li><a href="http://expo.survex.com/caves">expo.survex.com/caves</a> - list of caves surveyed and links to guidebook descriptions
<li>< href="http://expo.survex.com/pubs.htm">http://expo.survex.com/pubs.htm</a> - reports, accounts and logbooks
<li><a href="http://expo.survex.com/expedition/2018">expo.survex.com/expedition/2018</a> - Members on expo 2018: . Scroll down for a list of all the data typed in from survey trips.
<li><a href="http://expo.survex.com/survexfile/caves/">expo.survex.com/survexfile/caves/</a> - List of caves with all the surveys done for each.
<li><a href="http://expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx">expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx</a> - Cave survey data from 1983 in Schnellzughohle.
<li><a href="http://expo.survex.com/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="http://expo.survex.com/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
<p>Only a small part of troggle's original plan was fully implemented and deployed.
Many of the things it was intended to replace are still operating as a motley collection written by many different people in
several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).

2
handbook/survey/getin.htm
View File

@ -58,7 +58,7 @@ don't leave any blanks! Copy <em>and rename this</em> to where you want to put
<em>before</em> editing it. Too many people have overwritten the
template in the past, try not to do this yourself.
<h3>Filename and data-entry conventions</h3>
<h3><a id="filenames">Filename and data-entry conventions</a></h3>
<ul>
<li>Use Unix line endings (i.e. \n not \r\n).

11
handbook/troggle-ish.html
View File

@ -35,6 +35,17 @@ and publish it on the web. It was first used on the 2009 expo - see <a href="../
<p>Only a small part of troggle's original plan was fully implemented and deployed.
Many of the things it was intended to replace are still operating as a motley collection written by many different people in
several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).
<p>Examples of troggle-generated pages from data:
<ul>
<li><a href="http://expo.survex.com/caves">expo.survex.com/caves</a> - list of caves surveyed and links to guidebook descriptions
<li>< href="http://expo.survex.com/pubs.htm">http://expo.survex.com/pubs.htm</a> - reports, accounts and logbooks
<li><a href="http://expo.survex.com/expedition/2018">expo.survex.com/expedition/2018</a> - Members on expo 2018: . Scroll down for a list of all the data typed in from survey trips.
<li><a href="http://expo.survex.com/survexfile/caves/">expo.survex.com/survexfile/caves/</a> - List of caves with all the surveys done for each.
<li><a href="http://expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx">expo.survex.com/survexfile/caves-1623/115/cucc/futility.svx</a> - Cave survey data from 1983 in Schnellzughohle.
<li><a href="http://expo.survex.com/survey_scans/">expo.survex.com/survey_scans/</a> - List of all scanned original survey notes.
<li><a href="http://expo.survex.com/survey_scans/2018%252343/">expo.survex.com/survey_scans/2018%252343/</a> - list of links to scanned notes for wallet #43 during the 2018 expo.
</ul>
Today troggle is used for only three things:
<ol>
<li>Reformatting all the visible webpages such that they have a coherent style and have a contents list at the top-left

4
handbook/website-history.html
View File

@ -60,8 +60,8 @@ more straightforward.
<h3>Version control</h3>
<p>Another important element of this system was version control. The entire data structure was
stored initially in a Concurrent Version System repository, and later migrated to
Subversion [<em>now using a <a href="onlinesystems.html#mercurial">DVCS</a> in 2019</em>].
stored initially in a <a href="https://en.wikipedia.org/wiki/Concurrent_Versions_System">Concurrent Version System</a> repository, and later migrated to
<a href="https://en.wikipedia.org/wiki/Apache_Subversion">Subversion</a> [<em>now using a <a href="onlinesystems.html#mercurial">DVCS</a> in 2019</em>].
Any edits to the spreadsheets which caused the scripts to fail, breaking the
website, could be easily reversed.