2020-04-09 21:58:40 +01:00
<!DOCTYPE html>
< html >
< head >
< meta http-equiv = "Content-Type" content = "text/html; charset=utf-8" / >
< title > Handbook - Expo Server< / title >
< link rel = "stylesheet" type = "text/css" href = "../../css/main2.css" / >
< / head >
2022-06-24 22:34:22 +01:00
< body >
< style > body { background : #fff url ( /images/style/bg-system.png ) repeat-x 0 0 } < / style >
2020-05-04 00:49:40 +01:00
< h2 id = "tophead" > CUCC Expedition Handbook: Server< / h2 >
2020-04-09 21:58:40 +01:00
< h1 > The Expo server< / h1 >
< p > Troggle is the software which runs the the expo cave survey data management and website. It is not the only thing running on the expo server.
2020-05-04 00:49:40 +01:00
< h3 id = "what" > Server configuration: directory structure< / a > < / h3 >
< p > This is the folder structure of the repo < var > :expoweb:< / var > which is also the root
of the website. Note that the webserver (apache) presents many more apparent folders, such as < var > expofiles< / var > , than there are folders because that's what it and troggle do.
< p > In the list below only the < var > handbook< / var > folder has been expanded. The < var > years< / var > folder includes
2022-12-10 14:23:08 +00:00
42 subfolders from < var > 1976< / var > to < var > 2022< / var > .
2020-05-04 00:49:40 +01:00
< code > < textarea id = "folders" rows = "30" cols = "30" >
2020-05-19 19:08:53 +01:00
expoweb (repo)
2022-03-19 20:05:54 +00:00
├ ─ ─ 1623
├ ─ ─ 1626
├ ─ ─ cave_data
├ ─ ─ css
├ ─ ─ documents
├ ─ ─ entrance_data
├ ─ ─ folk
├ ─ ─ gallery
├ ─ ─ handbook
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ computing
│ ├ ─ ─ festering
│ ├ ─ ─ i
│ ├ ─ ─ l
│ ├ ─ ─ putty
│ ├ ─ ─ rig
│ ├ ─ ─ survey
│ ├ ─ ─ t
│ └ ─ ─ troggle
2022-03-19 20:05:54 +00:00
├ ─ ─ images
├ ─ ─ map
├ ─ ─ maps
├ ─ ─ noinfo
├ ─ ─ others
├ ─ ─ pending
├ ─ ─ photos
├ ─ ─ piclinks
├ ─ ─ scripts
├ ─ ─ signup
├ ─ ─ templates
├ ─ ─ tinypix
├ ─ ─ topos
└ ─ ─ years< / textarea > < / code >
2020-05-04 00:49:40 +01:00
2022-06-24 22:34:22 +01:00
< p > The handbook illustrates the i/t/l idiom whereby an image file (i) is displayed with a paragraph of text as an HTML file (l) and there is a thumbnail image (t) which is included in another document, e.g. see < a href = "/guidebook/walkin.htm" > the photographic guide to the walk from the toll road car park< / a > .
2020-05-04 00:49:40 +01:00
2021-10-29 20:16:55 +01:00
< h4 id = "css" > CSS and HTML< / a > < / h4 >
2021-03-21 21:46:21 +00:00
2021-10-29 20:16:55 +01:00
< p > The < var > expoweb< / var > repo contains all the HTML files and basic CSS files. The < var > troggle< / var > repo contains the HTML templates, CSS for the troggle report pages generated dynamically and a copy of the CSS, thumbnail images and javascript for the Django admin package which has to be available under the url '/site_media/' (which resolves to expo/troggle/media/) for the admin system to work.
< p >
See the live report on which urls resolve to which actual folders at < a href = "/pathsreport" > pathsreport< / a > .
2021-03-21 21:46:21 +00:00
2022-02-27 22:17:18 +00:00
2020-05-04 00:49:40 +01:00
< h3 id = "what" > Server configuration: software< / a > < / h3 >
< ul >
2022-02-27 22:17:18 +00:00
< li > < b > apache webserver *< / b >
< li > < b > expofiles *< / b >
2022-06-25 13:54:54 +01:00
< li > < b > MySQL/MariaDB database *< / b > < var > sudo apt install mariadb-server -y < / var >
2022-02-27 22:17:18 +00:00
< li > < a href = "#xapian" > xapian< / a > (search function) - < a href = "https://xapian.org/" > xapian.org< / a > . Wookey did this in 2020.
< li > < a href = "#kanboard" > kanboard< / a > (task planning) - < a href = "https://kanboard.org/" > kanboard.org< / a > . Wookey did this in 2022.
< li > < a href = "#boe" > boe< / a > (bank of expo - not troggle)
< li > < a href = "#cgit" > cgit< / a > (live webpage showing git status of repos) - < a href = "https://git.zx2c4.com/cgit/about/" > qit.zx2c4.com/cgit< / a >
2021-10-29 20:16:55 +01:00
< li > bins (used to regenerate the photo albums)
2022-02-27 22:17:18 +00:00
< li > packaged javascript
2020-05-04 00:49:40 +01:00
< li > cron
2022-02-27 22:17:18 +00:00
< li > Makefile (obsolete?)
< li > < a href = "#runserver" > Running a test Troggle server< / a >
2020-05-04 00:49:40 +01:00
< / ul >
2022-02-27 22:17:18 +00:00
< b > * These are essential< / b > to make troggle work at all.
2022-08-03 15:22:33 +01:00
< p > You will also need everythingto run Django, as documented in < a href = "/handbook/troggle/troglaptop.html" > troggle laptop< / a > including all the python modules listed there and installed using pip.
2023-02-08 23:36:35 +00:00
< hr / >
< p > < b > Now, if you don't know< / b > your < em > wsgi< / em > from your < em > asgi< / em > , read < a href =
"https://james.walters.click/what-django-deployment-is-really-about.html">this article
< / a > first on how to generically configure a Django server in the simplest possible
way.
2023-07-22 19:40:05 +01:00
2023-02-08 23:36:35 +00:00
< hr / >
2023-07-22 19:40:05 +01:00
< p > The server configuration scripts are in the file < var > troggle/_deploy/debian-server/wookey-exposerver-recipe.txt< / var > and are also
2021-03-21 21:46:21 +00:00
documented with notes in < var > troggle/README.txt< / var > . It is intended that the full documentation will
2023-07-22 19:40:05 +01:00
be moved here in due course.
2021-10-29 20:16:55 +01:00
< p > It is hoped that we will develop fully automated server setup scripts (such as are used by CUYC for their Django system)
2021-03-21 21:46:21 +00:00
< ul >
< li > to create a development server running locally on a personal machine
< li > to install the master system following a new operating system install on the public server
< / ul >
2020-05-19 19:08:53 +01:00
2022-03-29 16:26:53 +01:00
< p > Apache needs to run as user 'expo', not 'www-data' as standard. This is due to a basic incompatibility in permissions between apache and git: git does not honour existing permissions exactly. See
< a href = "https://nim-labs.com/run-apache-alternate-user/" > How to run apache as an alternate user< / a > .
< p > Although troggle will appear to work with sqlite database, it needs a proper concurrrent access database to manage multiple users. sqlite is
single-user (effectively a separate instance of django is created for each page access, so even one person looking at several pages at once is
"multi-user").
2020-05-19 19:08:53 +01:00
< p > The folder structure on the server is as shown below. It is all
in the user folder for the user < var > expo< / var > i.e. in
< br > /home/user/expo:
< code > < textarea id = "folders" rows = "30" cols = "40" >
expo
2022-03-19 20:05:54 +00:00
├ ─ ─ boe
2022-03-19 20:13:33 +00:00
│ └ ─ ─ ...
2022-03-19 20:05:54 +00:00
├ ─ ─ cuccfiles
2022-03-19 20:13:33 +00:00
│ └ ─ ─ ...
2022-03-19 20:05:54 +00:00
├ ─ ─ loser (repo)
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ caves
│ ├ ─ ─ caves-1623
│ ├ ─ ─ caves-1624
│ ├ ─ ─ caves-1626
│ ├ ─ ─ caves-1627
│ ├ ─ ─ docs
│ ├ ─ ─ fixedpts
│ ├ ─ ─ gpx
│ ├ ─ ─ kataster
│ ├ ─ ─ qms
│ ├ ─ ─ routes
│ ├ ─ ─ surface
│ └ ─ ─ template
2022-03-19 20:05:54 +00:00
├ ─ ─ drawings (repo)
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 107
│ ├ ─ ─ 161
│ ├ ─ ─ 2010-03
│ ├ ─ ─ 2011-01
│ ├ ─ ─ 2014-ms-14
│ ├ ─ ─ 2014-neo-01
│ ├ ─ ─ 2014-sd-01
│ ├ ─ ─ 2015-mf-06 Purple Lupin Cave
│ ├ ─ ─ 2017-cucc-23
│ ├ ─ ─ 2017-cucc-24
│ ├ ─ ─ 2017-cucc-28
│ ├ ─ ─ 2017-gmh
│ ├ ─ ─ 2017-pw-01
│ ├ ─ ─ 204
│ ├ ─ ─ 234
│ ├ ─ ─ 264-and-258
│ ├ ─ ─ minorCavesNear204
│ ├ ─ ─ oldframes
│ ├ ─ ─ oldframes2016
│ ├ ─ ─ styles
│ └ ─ ─ therion
2022-03-19 20:05:54 +00:00
├ ─ ─ expoweb (repo)
│ └ ─ ─ [see above]
├ ─ ─ troggle (repo)
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ core
│ ├ ─ ─ debian
│ ├ ─ ─ docker
│ ├ ─ ─ docsEtc
│ ├ ─ ─ export
│ ├ ─ ─ flatpages
│ ├ ─ ─ helper
│ ├ ─ ─ imagekit
│ ├ ─ ─ media
│ ├ ─ ─ parsers
│ ├ ─ ─ profiles
│ ├ ─ ─ templates
│ └ ─ ─ wiki
2022-03-19 20:05:54 +00:00
├ ─ ─ static
│ └ ─ ─ ...
├ ─ ─ repositories
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ cvs
│ ├ ─ ─ git
│ └ ─ ─ hg
2022-03-19 20:05:54 +00:00
├ ─ ─ expofiles
│ └ ─ ─ ...
├ ─ ─ expofiles-private
│ └ ─ ─ ...
└ ─ ─ webphotos
└ ─ ─ ...< / textarea > < / code >
2020-05-19 19:08:53 +01:00
2021-03-21 21:46:21 +00:00
< h4 id = "expofiles" > expofiles< / a > < / h3 >
< p > < var > expofiles< / var > contains ~40GB of files which are published by the webserver but which are not parsed by troggle. 28GB of these are photographs in < var > /expofiles/photos/< / var > and there are over 4GB of scanned images of surveys in < var > /expofiles/surveyscans/< / var > . There is a cleaned, complete copy of the documentation for the < var > tunnelX< / var > cave plan drawing package in < var > /expofiles/tunnelwiki/< / var > .
< p >
2020-05-19 19:08:53 +01:00
< code > < textarea id = "folders" rows = "30" cols = "40" >
expofiles
2022-03-19 20:05:54 +00:00
├ ─ ─ accounts
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 2016
│ ├ ─ ─ 2017
│ ├ ─ ─ misery
│ └ ─ ─ policy
2022-03-19 20:05:54 +00:00
├ ─ ─ aerial_photos
├ ─ ─ defaults
├ ─ ─ documents
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 2014kataster_applications
│ ├ ─ ─ 2020-expo-planning
│ ├ ─ ─ DistoX
│ ├ ─ ─ GPF
│ ├ ─ ─ SAP
│ ├ ─ ─ austria-1984
│ ├ ─ ─ bierbook
│ ├ ─ ─ cave_science_articles
│ ├ ─ ─ excursions2018
│ ├ ─ ─ grants
│ ├ ─ ─ hardware
│ ├ ─ ─ proc.ubss
│ ├ ─ ─ surveying
│ ├ ─ ─ troggle
│ ├ ─ ─ tshirts
│ └ ─ ─ website-link-checks
2022-03-19 20:05:54 +00:00
├ ─ ─ geotiffsurveys
2022-03-19 20:13:33 +00:00
│ └ ─ ─ qgis-examples
2022-03-19 20:05:54 +00:00
├ ─ ─ geotiffsurveys_backup
2022-03-19 20:13:33 +00:00
│ └ ─ ─ qgis-examples
2022-03-19 20:05:54 +00:00
├ ─ ─ gpslogs
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 2017
│ ├ ─ ─ 2018
│ ├ ─ ─ 2019
│ ├ ─ ─ OpenStreetMap
│ └ ─ ─ essentials
2022-03-19 20:05:54 +00:00
├ ─ ─ location_maps
├ ─ ─ photos
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 1978
│ ├ ─ ─ 1981
│ ├ ─ ─ 1982
│ ├ ─ ─ 1983
│ ├ ─ ─ 1997
│ ├ ─ ─ 1999
│ ├ ─ ─ 2004
│ ├ ─ ─ 2005
│ ├ ─ ─ 2007
│ ├ ─ ─ 2008
│ ├ ─ ─ 2009
│ ├ ─ ─ 2010
│ ├ ─ ─ 2011
│ ├ ─ ─ 2012
│ ├ ─ ─ 2013
│ ├ ─ ─ 2014
│ ├ ─ ─ 2015
│ ├ ─ ─ 2016
│ ├ ─ ─ 2017
│ ├ ─ ─ 2018
│ ├ ─ ─ 2019
│ ├ ─ ─ OriginalPhotoCD-png
│ ├ ─ ─ WinklerChapterPhotos
│ └ ─ ─ kletteratlas
2022-03-19 20:05:54 +00:00
├ ─ ─ presentations
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ CroatiaTrip2019
│ ├ ─ ─ HiddenEarth
│ ├ ─ ─ SpeleoAustria
│ ├ ─ ─ location_maps
│ ├ ─ ─ misc_2018-2019
│ └ ─ ─ test3js
2022-03-19 20:05:54 +00:00
├ ─ ─ qgis_projects
├ ─ ─ qgis_resources
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ centreline-data
│ ├ ─ ─ country_borders
│ └ ─ ─ terrain
2022-03-19 20:05:54 +00:00
├ ─ ─ rigging_topos
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 107
│ ├ ─ ─ 204
│ ├ ─ ─ 258
│ ├ ─ ─ 264
│ ├ ─ ─ 76
│ ├ ─ ─ rigging-guides-found-in-hut-2019
│ └ ─ ─ rigging_topos2017-18
2022-03-19 20:05:54 +00:00
├ ─ ─ surveys
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 107
│ ├ ─ ─ 148
│ ├ ─ ─ 161
│ ├ ─ ─ 1987-02
│ ├ ─ ─ 2004-01
│ ├ ─ ─ 2004-03
│ ├ ─ ─ 2007-71
│ ├ ─ ─ 204
│ ├ ─ ─ 234
│ ├ ─ ─ 258
│ ├ ─ ─ 258_and_264
│ ├ ─ ─ 264
│ ├ ─ ─ 267
│ ├ ─ ─ 271
│ ├ ─ ─ 40
│ ├ ─ ─ 76
│ ├ ─ ─ drafts
│ ├ ─ ─ junk
│ ├ ─ ─ minisurveys2012
│ ├ ─ ─ minisurveys2014
│ ├ ─ ─ minisurveys2016
│ ├ ─ ─ minisurveys2017
│ ├ ─ ─ minisurveys2018
│ ├ ─ ─ overview
│ ├ ─ ─ smkhs
│ └ ─ ─ tunnocks
2022-03-19 20:05:54 +00:00
├ ─ ─ surveyscans
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 1975
│ ├ ─ ─ 1983
│ ├ ─ ─ 1984
│ ├ ─ ─ 1989LUSS
│ ├ ─ ─ 1989surveybook
│ ├ ─ ─ 1991surveybook
│ ├ ─ ─ 1992-94Surveybookkh
│ ├ ─ ─ 1994
│ ├ ─ ─ 1995-96kh
│ ├ ─ ─ 1996-1999NotKHbook
│ ├ ─ ─ 1997-99kh
│ ├ ─ ─ 1999
│ ├ ─ ─ 2000
│ ├ ─ ─ 2001
│ ├ ─ ─ 2002
│ ├ ─ ─ 2003
│ ├ ─ ─ 2004
│ ├ ─ ─ 2005
│ ├ ─ ─ 2006
│ ├ ─ ─ 2007
│ ├ ─ ─ 2008
│ ├ ─ ─ 2009
│ ├ ─ ─ 2010
│ ├ ─ ─ 2011
│ ├ ─ ─ 2012
│ ├ ─ ─ 2013
│ ├ ─ ─ 2014
│ ├ ─ ─ 2015
│ ├ ─ ─ 2016
│ ├ ─ ─ 2017
│ ├ ─ ─ 2018
│ ├ ─ ─ 2019
│ ├ ─ ─ 2021
│ └ ─ ─ 90Surveybookkh
2022-03-19 20:05:54 +00:00
├ ─ ─ terrain
├ ─ ─ training-info
├ ─ ─ tunnelwiki
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ pdf
│ ├ ─ ─ skins
│ ├ ─ ─ training-info
│ ├ ─ ─ uploads
│ └ ─ ─ wiki
2022-03-19 20:05:54 +00:00
├ ─ ─ uploads
├ ─ ─ video
2022-03-19 20:13:33 +00:00
│ ├ ─ ─ 2004
│ ├ ─ ─ 2007
│ ├ ─ ─ 2009
│ ├ ─ ─ 2010
│ ├ ─ ─ 2011
│ ├ ─ ─ 2012
│ ├ ─ ─ 2013
│ ├ ─ ─ 2014
│ ├ ─ ─ 2016
│ ├ ─ ─ 2017
│ ├ ─ ─ 2018
│ └ ─ ─ 2019
2022-03-19 20:05:54 +00:00
└ ─ ─ writeups
├ ─ ─ 1992
├ ─ ─ 1993
├ ─ ─ 1994
├ ─ ─ 1999
├ ─ ─ 2000
├ ─ ─ 2003
├ ─ ─ 2007
├ ─ ─ 2008
├ ─ ─ 2009
├ ─ ─ 2010
├ ─ ─ 2011
├ ─ ─ 2012
├ ─ ─ 2013
├ ─ ─ 2014
├ ─ ─ 2015
├ ─ ─ 2016
├ ─ ─ 2017
├ ─ ─ 2018
├ ─ ─ 2019
└ ─ ─ 2021< / textarea > < / code >
2020-05-04 00:49:40 +01:00
2022-02-27 22:17:18 +00:00
< h3 id = "js" > Javascript< / a > < / h3 >
There are javascript packages installed directly on the operating system on the server, i.e. in< br >
< code > /usr/share/javascript< / code >
which are used by some < var > expoweb< / var > pages:
< ul >
< li > < var > CaveScript< / var > for showing 3D views of cave centreline data,
< li > < var > CodeMirror< / var > for syntax-colouring survex files
< li > < var > openlayers< / var > for geographic mapping integration, for out slippy map showing paths and entrances at < a href = "/map/map.html" > /map/map.html< / a >
< li > < var > leaflets< / var > and experimental mapping test (only on /map/index-leaflet.html )
< / ul >
These javascript packages are not documented but some are mentioned in < a href = "troglaptop.html" > the soft dev laptop< / a > configuration.
< p > Currently installed on the server (October 2021) are the following javascript packages which are < em > not used at all by troggle or any of the expo website< / em > :
< ul >
< li > jquery
< li > jquery-ui
< li > mootools
< li > sphinxdoc
< li > underscore
< / ul >
< p > Presumably these are used by something else hosted on the server ? Anyway, if you are setting up a new troggle sever you don't need them.
2020-05-04 00:49:40 +01:00
2022-02-27 22:17:18 +00:00
< h3 id = "xapian" > Search: xapian< / h3 >
2020-05-04 00:49:40 +01:00
< p > Installed independently of troggle simply with < var > apt install xapian-omega< / var > and
2022-02-27 22:17:18 +00:00
then configured into the troggle-generated menus in < var > css/main2.css< / var > .
You can see it at the bottom of the top-left menu on this page and on nearly all pages of the handbook.
The function is
2020-05-04 00:49:40 +01:00
connected with an apache configuration
< code > ScriptAlias /search /usr/lib/cgi-bin/omega/omega< / code > in < var > ~expo/config/apache/expo.conf< / var > .
Installed by Wookey in May 2020.
2022-02-27 22:17:18 +00:00
< h3 id = "kanboard" > Planning: kanboard< / h3 >
< p > This is installed on the server and accessed at < a href = "/kanboard" > < var > /kanboard< / var > < / a > It is an open source
equivalent of the Trello kanban card task planning system. The 2022 expo uses
< a href = "https://trello.com/b/WddB62HI/expo-jobs" > Trello itself< / a > (separate login required) but we intend to move to our own kanboard
from 2023.
< h3 id = "boe" > boe< / h3 >
< p > This is a perl script, and served by the webserver using the url apache configuration< pre > < code > #bank of expo
#current expedition
ScriptAlias /boe /home/expo/boe/boc/boc.pl
< Directory /home/expo/boe/boc>
AddHandler cgi-script .pl
SetHandler cgi-script
Options +ExecCGI
Require all granted
< /Directory> < / code > < / pre > in < var > ~expo/config/apache/expo.conf< / var > .
< p > Handbook documentation for its use is at < a href = "../bankofexpo.html" > The Bank of Expo< / a > .
< h3 id = "cgit" > cgit - web interface to server git repos< / h3 >
< p > This is a compiled executable written in C which, like boe, is installed as an Apache CGI redirection.
The installation instructions are at < a href = "https://git.zx2c4.com/cgit/tree/README" > https://git.zx2c4.com/cgit/tree/README< / a >
but we use the Debian package < a href = "https://packages.debian.org/stable/cgit" > https://packages.debian.org/stable/cgit< / a > .
< h3 > cron - automatically run tasks to schedule< / h3 >
< p > This is currently disabled in Feb. 2022. If you need anything that would be done frequently (e.g. bins) you currently have to run it manually.
< p > The server runs it's hourly, daily and weekly scripts using the anacron system. In
< var > ~expo/config/cron/< / var > on the server, there are < var > expo.hourly< / var >
and < var > expo.daily< / var > scripts and these are (or should be) launched at the appropriate times by
the server root from < var > /etc/crontab< / var > . This is not obviously working on the server at present.
< h3 id = "runserver" > Running a test Troggle server< / h3 >
2021-10-29 20:16:55 +01:00
< p >
For high volume use, Troggle should be run using a web server like apache.
However, a quick way to get started is to use the development server built into Django.
This is limited though: directory redirection needs apache so CodeMirror and CaveView won't work as the url /javascript/ will not exist.
< p >
To do this, run
< code >
$ python manage.py runserver 8000 -v 3
< / code >
from the troggle directory. This runs it on port 8000 so you see the website
at http://localhost:8000/
< p >
gunicorn also works. This runs with 9 workers (suitable for a 4-core processor,
-w takes n+1 where n is the number of cores of your processor):
< code >
$ gunicorn --reload -w 9 -b :8000 wsgi
< / code >
2020-04-09 21:58:40 +01:00
< hr / >
2020-07-27 01:42:09 +01:00
Return to: < a href = "trogdesign.html" > Troggle design and future implementations< / a > < br / >
Return to: < a href = "trogintro.html" > Troggle intro< / a > < br / >
Troggle index:
< a href = "trogindex.html" > Index of all troggle documents< / a > < br / >
2022-06-24 22:34:22 +01:00
< hr / > < / body >
2020-04-09 21:58:40 +01:00
< / html >
