<!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Handbook - Expo Server</title> <link rel="stylesheet" type="text/css" href="../../css/main2.css" /> </head> <body> <style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style> <h2 id="tophead">CUCC Expedition Handbook: Server</h2> <h1>The Expo server</h1> <p>Troggle is the software which runs the the expo cave survey data management and website. It is not the only thing running on the expo server. <h3 id="what">Server configuration: directory structure</a></h3> <p>This is the folder structure of the repo <var>:expoweb:</var> which is also the root of the website. Note that the webserver (apache) presents many more apparent folders, such as <var>expofiles</var>, than there are folders because that's what it and troggle do. <p>In the list below only the <var>handbook</var> folder has been expanded. The <var>years</var> folder includes 42 subfolders from <var>1976</var> to <var>2022</var>. <code><textarea id="folders" rows="30" cols="30"> expoweb (repo) ├── 1623 ├── 1626 ├── cave_data ├── css ├── documents ├── entrance_data ├── folk ├── gallery ├── handbook │ ├── computing │ ├── festering │ ├── i │ ├── l │ ├── putty │ ├── rig │ ├── survey │ ├── t │ └── troggle ├── images ├── map ├── maps ├── noinfo ├── others ├── pending ├── photos ├── piclinks ├── scripts ├── signup ├── templates ├── tinypix ├── topos └── years</textarea></code> <p>The handbook illustrates the i/t/l idiom whereby an image file (i) is displayed with a paragraph of text as an HTML file (l) and there is a thumbnail image (t) which is included in another document, e.g. see <a href="/guidebook/walkin.htm">the photographic guide to the walk from the toll road car park</a>. <h4 id="css">CSS and HTML</a></h4> <p>The <var>expoweb</var> repo contains all the HTML files and basic CSS files. The <var>troggle</var> repo contains the HTML templates, CSS for the troggle report pages generated dynamically and a copy of the CSS, thumbnail images and javascript for the Django admin package which has to be available under the url '/site_media/' (which resolves to expo/troggle/media/) for the admin system to work. <p> See the live report on which urls resolve to which actual folders at <a href="/pathsreport">pathsreport</a>. <h3 id="what">Server configuration: software</a></h3> <ul> <li><b>apache webserver *</b> <li><b>expofiles *</b> <li><b>MySQL/MariaDB database *</b> <var>sudo apt install mariadb-server -y </var> <li><a href="#xapian">xapian</a> (search function) - <a href="https://xapian.org/">xapian.org</a>. Wookey did this in 2020. <li><a href="#kanboard">kanboard</a> (task planning) - <a href="https://kanboard.org/">kanboard.org</a>. Wookey did this in 2022. <li><a href="#boe">boe</a> (bank of expo - not troggle) <li><a href="#cgit">cgit</a> (live webpage showing git status of repos) - <a href="https://git.zx2c4.com/cgit/about/">qit.zx2c4.com/cgit</a> <li>bins (used to regenerate the photo albums) <li>packaged javascript <li>cron <li>Makefile (obsolete?) <li><a href="#runserver">Running a test Troggle server</a> </ul> <b> * These are essential</b> to make troggle work at all. <p>You will also need everythingto run Django, as documented in <a href="/handbook/troggle/troglaptop.html">troggle laptop</a> including all the python modules listed there and installed using pip. <hr /> <p><b>Now, if you don't know</b> your <em>wsgi</em> from your <em>asgi</em>, read <a href= "https://james.walters.click/what-django-deployment-is-really-about.html">this article </a> first on how to generically configure a Django server in the simplest possible way. <hr /> <p>The server configuration scripts are in the file <var>troggle/_deploy/debian-server/wookey-exposerver-recipe.txt</var> and are also documented with notes in <var>troggle/README.txt</var>. It is intended that the full documentation will be moved here in due course. <p>It is hoped that we will develop fully automated server setup scripts (such as are used by CUYC for their Django system) <ul> <li>to create a development server running locally on a personal machine <li>to install the master system following a new operating system install on the public server </ul> <p>Apache needs to run as user 'expo', not 'www-data' as standard. This is due to a basic incompatibility in permissions between apache and git: git does not honour existing permissions exactly. See <a href="https://nim-labs.com/run-apache-alternate-user/">How to run apache as an alternate user</a>. <p>Although troggle will appear to work with sqlite database, it needs a proper concurrrent access database to manage multiple users. sqlite is single-user (effectively a separate instance of django is created for each page access, so even one person looking at several pages at once is "multi-user"). <p>The folder structure on the server is as shown below. It is all in the user folder for the user <var>expo</var> i.e. in <br>/home/user/expo: <code><textarea id="folders" rows="30" cols="40"> expo ├── boe │ └── ... ├── cuccfiles │ └── ... ├── loser (repo) │ ├── caves │ ├── caves-1623 │ ├── caves-1624 │ ├── caves-1626 │ ├── caves-1627 │ ├── docs │ ├── fixedpts │ ├── gpx │ ├── kataster │ ├── qms │ ├── routes │ ├── surface │ └── template ├── drawings (repo) │ ├── 107 │ ├── 161 │ ├── 2010-03 │ ├── 2011-01 │ ├── 2014-ms-14 │ ├── 2014-neo-01 │ ├── 2014-sd-01 │ ├── 2015-mf-06 Purple Lupin Cave │ ├── 2017-cucc-23 │ ├── 2017-cucc-24 │ ├── 2017-cucc-28 │ ├── 2017-gmh │ ├── 2017-pw-01 │ ├── 204 │ ├── 234 │ ├── 264-and-258 │ ├── minorCavesNear204 │ ├── oldframes │ ├── oldframes2016 │ ├── styles │ └── therion ├── expoweb (repo) │ └── [see above] ├── troggle (repo) │ ├── core │ ├── debian │ ├── docker │ ├── docsEtc │ ├── export │ ├── flatpages │ ├── helper │ ├── imagekit │ ├── media │ ├── parsers │ ├── profiles │ ├── templates │ └── wiki ├── static │ └── ... ├── repositories │ ├── cvs │ ├── git │ └── hg ├── expofiles │ └── ... ├── expofiles-private │ └── ... └── webphotos └── ...</textarea></code> <h4 id="expofiles">expofiles</a></h3> <p><var>expofiles</var> contains ~40GB of files which are published by the webserver but which are not parsed by troggle. 28GB of these are photographs in <var>/expofiles/photos/</var> and there are over 4GB of scanned images of surveys in <var>/expofiles/surveyscans/</var>. There is a cleaned, complete copy of the documentation for the <var>tunnelX</var> cave plan drawing package in <var>/expofiles/tunnelwiki/</var>. <p> <code><textarea id="folders" rows="30" cols="40"> expofiles ├── accounts │ ├── 2016 │ ├── 2017 │ ├── misery │ └── policy ├── aerial_photos ├── defaults ├── documents │ ├── 2014kataster_applications │ ├── 2020-expo-planning │ ├── DistoX │ ├── GPF │ ├── SAP │ ├── austria-1984 │ ├── bierbook │ ├── cave_science_articles │ ├── excursions2018 │ ├── grants │ ├── hardware │ ├── proc.ubss │ ├── surveying │ ├── troggle │ ├── tshirts │ └── website-link-checks ├── geotiffsurveys │ └── qgis-examples ├── geotiffsurveys_backup │ └── qgis-examples ├── gpslogs │ ├── 2017 │ ├── 2018 │ ├── 2019 │ ├── OpenStreetMap │ └── essentials ├── location_maps ├── photos │ ├── 1978 │ ├── 1981 │ ├── 1982 │ ├── 1983 │ ├── 1997 │ ├── 1999 │ ├── 2004 │ ├── 2005 │ ├── 2007 │ ├── 2008 │ ├── 2009 │ ├── 2010 │ ├── 2011 │ ├── 2012 │ ├── 2013 │ ├── 2014 │ ├── 2015 │ ├── 2016 │ ├── 2017 │ ├── 2018 │ ├── 2019 │ ├── OriginalPhotoCD-png │ ├── WinklerChapterPhotos │ └── kletteratlas ├── presentations │ ├── CroatiaTrip2019 │ ├── HiddenEarth │ ├── SpeleoAustria │ ├── location_maps │ ├── misc_2018-2019 │ └── test3js ├── qgis_projects ├── qgis_resources │ ├── centreline-data │ ├── country_borders │ └── terrain ├── rigging_topos │ ├── 107 │ ├── 204 │ ├── 258 │ ├── 264 │ ├── 76 │ ├── rigging-guides-found-in-hut-2019 │ └── rigging_topos2017-18 ├── surveys │ ├── 107 │ ├── 148 │ ├── 161 │ ├── 1987-02 │ ├── 2004-01 │ ├── 2004-03 │ ├── 2007-71 │ ├── 204 │ ├── 234 │ ├── 258 │ ├── 258_and_264 │ ├── 264 │ ├── 267 │ ├── 271 │ ├── 40 │ ├── 76 │ ├── drafts │ ├── junk │ ├── minisurveys2012 │ ├── minisurveys2014 │ ├── minisurveys2016 │ ├── minisurveys2017 │ ├── minisurveys2018 │ ├── overview │ ├── smkhs │ └── tunnocks ├── surveyscans │ ├── 1975 │ ├── 1983 │ ├── 1984 │ ├── 1989LUSS │ ├── 1989surveybook │ ├── 1991surveybook │ ├── 1992-94Surveybookkh │ ├── 1994 │ ├── 1995-96kh │ ├── 1996-1999NotKHbook │ ├── 1997-99kh │ ├── 1999 │ ├── 2000 │ ├── 2001 │ ├── 2002 │ ├── 2003 │ ├── 2004 │ ├── 2005 │ ├── 2006 │ ├── 2007 │ ├── 2008 │ ├── 2009 │ ├── 2010 │ ├── 2011 │ ├── 2012 │ ├── 2013 │ ├── 2014 │ ├── 2015 │ ├── 2016 │ ├── 2017 │ ├── 2018 │ ├── 2019 │ ├── 2021 │ └── 90Surveybookkh ├── terrain ├── training-info ├── tunnelwiki │ ├── pdf │ ├── skins │ ├── training-info │ ├── uploads │ └── wiki ├── uploads ├── video │ ├── 2004 │ ├── 2007 │ ├── 2009 │ ├── 2010 │ ├── 2011 │ ├── 2012 │ ├── 2013 │ ├── 2014 │ ├── 2016 │ ├── 2017 │ ├── 2018 │ └── 2019 └── writeups ├── 1992 ├── 1993 ├── 1994 ├── 1999 ├── 2000 ├── 2003 ├── 2007 ├── 2008 ├── 2009 ├── 2010 ├── 2011 ├── 2012 ├── 2013 ├── 2014 ├── 2015 ├── 2016 ├── 2017 ├── 2018 ├── 2019 └── 2021</textarea></code> <h3 id="js">Javascript</a></h3> There are javascript packages installed directly on the operating system on the server, i.e. in<br> <code>/usr/share/javascript</code> which are used by some <var>expoweb</var> pages: <ul> <li><var>CaveScript</var> for showing 3D views of cave centreline data, <li><var>CodeMirror</var> for syntax-colouring survex files <li><var>openlayers</var> for geographic mapping integration, for out slippy map showing paths and entrances at <a href="/map/map.html">/map/map.html</a> <li><var>leaflets</var> and experimental mapping test (only on /map/index-leaflet.html ) </ul> These javascript packages are not documented but some are mentioned in <a href="troglaptop.html">the soft dev laptop</a> configuration. <p>Currently installed on the server (October 2021) are the following javascript packages which are <em>not used at all by troggle or any of the expo website</em>: <ul> <li>jquery <li>jquery-ui <li>mootools <li>sphinxdoc <li>underscore </ul> <p>Presumably these are used by something else hosted on the server ? Anyway, if you are setting up a new troggle sever you don't need them. <h3 id="xapian">Search: xapian</h3> <p>Installed independently of troggle simply with <var>apt install xapian-omega</var> and then configured into the troggle-generated menus in <var>css/main2.css</var>. You can see it at the bottom of the top-left menu on this page and on nearly all pages of the handbook. The function is connected with an apache configuration <code>ScriptAlias /search /usr/lib/cgi-bin/omega/omega</code> in <var>~expo/config/apache/expo.conf</var>. Installed by Wookey in May 2020. <h3 id="kanboard">Planning: kanboard</h3> <p>This is installed on the server and accessed at <a href="/kanboard"><var>/kanboard</var></a> It is an open source equivalent of the Trello kanban card task planning system. The 2022 expo uses <a href="https://trello.com/b/WddB62HI/expo-jobs">Trello itself</a> (separate login required) but we intend to move to our own kanboard from 2023. <h3 id="boe">boe</h3> <p>This is a perl script, and served by the webserver using the url apache configuration<pre><code>#bank of expo #current expedition ScriptAlias /boe /home/expo/boe/boc/boc.pl <Directory /home/expo/boe/boc> AddHandler cgi-script .pl SetHandler cgi-script Options +ExecCGI Require all granted </Directory></code></pre> in <var>~expo/config/apache/expo.conf</var>. <p>Handbook documentation for its use is at <a href="../bankofexpo.html">The Bank of Expo</a>. <h3 id="cgit">cgit - web interface to server git repos</h3> <p>This is a compiled executable written in C which, like boe, is installed as an Apache CGI redirection. The installation instructions are at <a href="https://git.zx2c4.com/cgit/tree/README">https://git.zx2c4.com/cgit/tree/README</a> but we use the Debian package <a href="https://packages.debian.org/stable/cgit">https://packages.debian.org/stable/cgit</a>. <h3>cron - automatically run tasks to schedule</h3> <p>This is currently disabled in Feb. 2022. If you need anything that would be done frequently (e.g. bins) you currently have to run it manually. <p>The server runs it's hourly, daily and weekly scripts using the anacron system. In <var>~expo/config/cron/</var> on the server, there are <var>expo.hourly</var> and <var>expo.daily</var> scripts and these are (or should be) launched at the appropriate times by the server root from <var>/etc/crontab</var>. This is not obviously working on the server at present. <h3 id="runserver">Running a test Troggle server</h3> <p> For high volume use, Troggle should be run using a web server like apache. However, a quick way to get started is to use the development server built into Django. This is limited though: directory redirection needs apache so CodeMirror and CaveView won't work as the url /javascript/ will not exist. <p> To do this, run <code> $ python manage.py runserver 8000 -v 3 </code> from the troggle directory. This runs it on port 8000 so you see the website at http://localhost:8000/ <p> gunicorn also works. This runs with 9 workers (suitable for a 4-core processor, -w takes n+1 where n is the number of cores of your processor): <code> $ gunicorn --reload -w 9 -b :8000 wsgi </code> <hr /> Return to: <a href="trogdesign.html">Troggle design and future implementations</a><br /> Return to: <a href="trogintro.html">Troggle intro</a><br /> Troggle index: <a href="trogindex.html">Index of all troggle documents</a><br /> <hr /></body> </html>