CUCC Expedition Handbook: Server

The Expo server

Troggle is the software which runs the the expo cave survey data management and website. It is not the only thing running on the expo server.

Server configuration: directory structure

This is the folder structure of the repo :expoweb: which is also the root of the website. Note that the webserver (apache) presents many more apparent folders, such as expofiles, than there are folders because that's what it and troggle do.

In the list below only the handbook folder has been expanded. The years folder includes 42 subfolders from 1976 to 2022. expoweb (repo) ├── 1623 ├── 1626 ├── cave_data ├── css ├── documents ├── entrance_data ├── folk ├── gallery ├── handbook │ ├── computing │ ├── festering │ ├── i │ ├── l │ ├── putty │ ├── rig │ ├── survey │ ├── t │ └── troggle ├── images ├── map ├── maps ├── noinfo ├── others ├── pending ├── photos ├── piclinks ├── scripts ├── signup ├── templates ├── tinypix ├── topos └── years

The handbook illustrates the i/t/l idiom whereby an image file (i) is displayed with a paragraph of text as an HTML file (l) and there is a thumbnail image (t) which is included in another document, e.g. see the photographic guide to the walk from the toll road car park.

CSS and HTML

The expoweb repo contains all the HTML files and basic CSS files. The troggle repo contains the HTML templates, CSS for the troggle report pages generated dynamically and a copy of the CSS, thumbnail images and javascript for the Django admin package which has to be available under the url '/site_media/' (which resolves to expo/troggle/media/) for the admin system to work.

See the live report on which urls resolve to which actual folders at pathsreport.

Server configuration: software

• apache webserver *
• expofiles *
• MySQL/MariaDB database * sudo apt install mariadb-server -y
• xapian (search function) - xapian.org. Wookey did this in 2020.
• kanboard (task planning) - kanboard.org. Wookey did this in 2022.
• boe (bank of expo - not troggle)
• cgit (live webpage showing git status of repos) - qit.zx2c4.com/cgit
• bins (used to regenerate the photo albums)
• packaged javascript
• cron
• Makefile (obsolete?)
• Running a test Troggle server

You will also need everythingto run Django, as documented in troggle laptop including all the python modules listed there and installed using pip.

Now, if you don't know your wsgi from your asgi, read this article first on how to generically configure a Django server in the simplest possible way.

The server configuration scripts are in the file troggle/debian/serversetup and are also documented with notes in troggle/README.txt. It is intended that the full documentation will be copied here in due course.

It is hoped that we will develop fully automated server setup scripts (such as are used by CUYC for their Django system)

• to create a development server running locally on a personal machine
• to install the master system following a new operating system install on the public server

Apache needs to run as user 'expo', not 'www-data' as standard. This is due to a basic incompatibility in permissions between apache and git: git does not honour existing permissions exactly. See How to run apache as an alternate user.

Although troggle will appear to work with sqlite database, it needs a proper concurrrent access database to manage multiple users. sqlite is single-user (effectively a separate instance of django is created for each page access, so even one person looking at several pages at once is "multi-user").

The folder structure on the server is as shown below. It is all in the user folder for the user expo i.e. in
/home/user/expo: expo ├── boe │ └── ... ├── cuccfiles │ └── ... ├── loser (repo) │ ├── caves │ ├── caves-1623 │ ├── caves-1624 │ ├── caves-1626 │ ├── caves-1627 │ ├── docs │ ├── fixedpts │ ├── gpx │ ├── kataster │ ├── qms │ ├── routes │ ├── surface │ └── template ├── drawings (repo) │ ├── 107 │ ├── 161 │ ├── 2010-03 │ ├── 2011-01 │ ├── 2014-ms-14 │ ├── 2014-neo-01 │ ├── 2014-sd-01 │ ├── 2015-mf-06 Purple Lupin Cave │ ├── 2017-cucc-23 │ ├── 2017-cucc-24 │ ├── 2017-cucc-28 │ ├── 2017-gmh │ ├── 2017-pw-01 │ ├── 204 │ ├── 234 │ ├── 264-and-258 │ ├── minorCavesNear204 │ ├── oldframes │ ├── oldframes2016 │ ├── styles │ └── therion ├── expoweb (repo) │ └── [see above] ├── troggle (repo) │ ├── core │ ├── debian │ ├── docker │ ├── docsEtc │ ├── export │ ├── flatpages │ ├── helper │ ├── imagekit │ ├── media │ ├── parsers │ ├── profiles │ ├── templates │ └── wiki ├── static │ └── ... ├── repositories │ ├── cvs │ ├── git │ └── hg ├── expofiles │ └── ... ├── expofiles-private │ └── ... └── webphotos └── ...

expofiles

expofiles contains ~40GB of files which are published by the webserver but which are not parsed by troggle. 28GB of these are photographs in /expofiles/photos/ and there are over 4GB of scanned images of surveys in /expofiles/surveyscans/. There is a cleaned, complete copy of the documentation for the tunnelX cave plan drawing package in /expofiles/tunnelwiki/.

expofiles ├── accounts │ ├── 2016 │ ├── 2017 │ ├── misery │ └── policy ├── aerial_photos ├── defaults ├── documents │ ├── 2014kataster_applications │ ├── 2020-expo-planning │ ├── DistoX │ ├── GPF │ ├── SAP │ ├── austria-1984 │ ├── bierbook │ ├── cave_science_articles │ ├── excursions2018 │ ├── grants │ ├── hardware │ ├── proc.ubss │ ├── surveying │ ├── troggle │ ├── tshirts │ └── website-link-checks ├── geotiffsurveys │ └── qgis-examples ├── geotiffsurveys_backup │ └── qgis-examples ├── gpslogs │ ├── 2017 │ ├── 2018 │ ├── 2019 │ ├── OpenStreetMap │ └── essentials ├── location_maps ├── photos │ ├── 1978 │ ├── 1981 │ ├── 1982 │ ├── 1983 │ ├── 1997 │ ├── 1999 │ ├── 2004 │ ├── 2005 │ ├── 2007 │ ├── 2008 │ ├── 2009 │ ├── 2010 │ ├── 2011 │ ├── 2012 │ ├── 2013 │ ├── 2014 │ ├── 2015 │ ├── 2016 │ ├── 2017 │ ├── 2018 │ ├── 2019 │ ├── OriginalPhotoCD-png │ ├── WinklerChapterPhotos │ └── kletteratlas ├── presentations │ ├── CroatiaTrip2019 │ ├── HiddenEarth │ ├── SpeleoAustria │ ├── location_maps │ ├── misc_2018-2019 │ └── test3js ├── qgis_projects ├── qgis_resources │ ├── centreline-data │ ├── country_borders │ └── terrain ├── rigging_topos │ ├── 107 │ ├── 204 │ ├── 258 │ ├── 264 │ ├── 76 │ ├── rigging-guides-found-in-hut-2019 │ └── rigging_topos2017-18 ├── surveys │ ├── 107 │ ├── 148 │ ├── 161 │ ├── 1987-02 │ ├── 2004-01 │ ├── 2004-03 │ ├── 2007-71 │ ├── 204 │ ├── 234 │ ├── 258 │ ├── 258_and_264 │ ├── 264 │ ├── 267 │ ├── 271 │ ├── 40 │ ├── 76 │ ├── drafts │ ├── junk │ ├── minisurveys2012 │ ├── minisurveys2014 │ ├── minisurveys2016 │ ├── minisurveys2017 │ ├── minisurveys2018 │ ├── overview │ ├── smkhs │ └── tunnocks ├── surveyscans │ ├── 1975 │ ├── 1983 │ ├── 1984 │ ├── 1989LUSS │ ├── 1989surveybook │ ├── 1991surveybook │ ├── 1992-94Surveybookkh │ ├── 1994 │ ├── 1995-96kh │ ├── 1996-1999NotKHbook │ ├── 1997-99kh │ ├── 1999 │ ├── 2000 │ ├── 2001 │ ├── 2002 │ ├── 2003 │ ├── 2004 │ ├── 2005 │ ├── 2006 │ ├── 2007 │ ├── 2008 │ ├── 2009 │ ├── 2010 │ ├── 2011 │ ├── 2012 │ ├── 2013 │ ├── 2014 │ ├── 2015 │ ├── 2016 │ ├── 2017 │ ├── 2018 │ ├── 2019 │ ├── 2021 │ └── 90Surveybookkh ├── terrain ├── training-info ├── tunnelwiki │ ├── pdf │ ├── skins │ ├── training-info │ ├── uploads │ └── wiki ├── uploads ├── video │ ├── 2004 │ ├── 2007 │ ├── 2009 │ ├── 2010 │ ├── 2011 │ ├── 2012 │ ├── 2013 │ ├── 2014 │ ├── 2016 │ ├── 2017 │ ├── 2018 │ └── 2019 └── writeups ├── 1992 ├── 1993 ├── 1994 ├── 1999 ├── 2000 ├── 2003 ├── 2007 ├── 2008 ├── 2009 ├── 2010 ├── 2011 ├── 2012 ├── 2013 ├── 2014 ├── 2015 ├── 2016 ├── 2017 ├── 2018 ├── 2019 └── 2021

Javascript

• CaveScript for showing 3D views of cave centreline data,
• CodeMirror for syntax-colouring survex files
• openlayers for geographic mapping integration, for out slippy map showing paths and entrances at /map/map.html
• leaflets and experimental mapping test (only on /map/index-leaflet.html )

Currently installed on the server (October 2021) are the following javascript packages which are not used at all by troggle or any of the expo website:

• jquery
• jquery-ui
• mootools
• sphinxdoc
• underscore

Presumably these are used by something else hosted on the server ? Anyway, if you are setting up a new troggle sever you don't need them.

Search: xapian

Installed independently of troggle simply with apt install xapian-omega and then configured into the troggle-generated menus in css/main2.css. You can see it at the bottom of the top-left menu on this page and on nearly all pages of the handbook. The function is connected with an apache configuration ScriptAlias /search /usr/lib/cgi-bin/omega/omega in ~expo/config/apache/expo.conf. Installed by Wookey in May 2020.

Planning: kanboard

This is installed on the server and accessed at /kanboard It is an open source equivalent of the Trello kanban card task planning system. The 2022 expo uses Trello itself (separate login required) but we intend to move to our own kanboard from 2023.

boe

This is a perl script, and served by the webserver using the url apache configuration

#bank of expo
#current expedition
ScriptAlias /boe /home/expo/boe/boc/boc.pl
<Directory /home/expo/boe/boc>
        AddHandler cgi-script .pl
        SetHandler cgi-script
        Options +ExecCGI
        Require all granted
</Directory>

Handbook documentation for its use is at The Bank of Expo.

cgit - web interface to server git repos

This is a compiled executable written in C which, like boe, is installed as an Apache CGI redirection. The installation instructions are at https://git.zx2c4.com/cgit/tree/README but we use the Debian package https://packages.debian.org/stable/cgit.

cron - automatically run tasks to schedule

This is currently disabled in Feb. 2022. If you need anything that would be done frequently (e.g. bins) you currently have to run it manually.

The server runs it's hourly, daily and weekly scripts using the anacron system. In ~expo/config/cron/ on the server, there are expo.hourly and expo.daily scripts and these are (or should be) launched at the appropriate times by the server root from /etc/crontab. This is not obviously working on the server at present.

Running a test Troggle server

For high volume use, Troggle should be run using a web server like apache. However, a quick way to get started is to use the development server built into Django. This is limited though: directory redirection needs apache so CodeMirror and CaveView won't work as the url /javascript/ will not exist.

To do this, run $ python manage.py runserver 8000 -v 3 from the troggle directory. This runs it on port 8000 so you see the website at http://localhost:8000/

gunicorn also works. This runs with 9 workers (suitable for a 4-core processor, -w takes n+1 where n is the number of cores of your processor): $ gunicorn --reload -w 9 -b :8000 wsgi