troggle/README.txt

Updated 3 May 2021

Troggle is an application for caving expedition data management, 
originally created for use on Cambridge University Caving Club (CUCC)expeditions 
and licensed under the GNU Lesser General Public License.

Troggle has been forked into two projects. The original one is maintained by Aaron Curtis 
and was used for Erebus caves. The CUCC variant uses files as the definitive data, 
not the database and lives at http://expo.survex.com/repositories/troggle/.git/

See copyright notices in
http://expo.survex.com/handbook/computing/contribute.html

See online documentation at http://expo.survex.com/handbook/troggle/serverconfig.html
and at troggle/debian/serversetup .

Troggle setup
=============
0. read the very extensive oinline documentation and stop reading this README.
1. git clone troggle into correct directory structure
2. install pip, django & patch django
3. configure django to recognise troggle 

See http://expo.survex.com/handbook/computing/onlinesystems.html 

Setting up directories
----------------------
- create a directory in which you also have the loser/, expoweb/ and drawings/ repositories
- git clone (see below) the current latest troggle from the 'python3' git branch into 
  a folder called 'troggle'
- all the 4 repos should now be siblings, and also siblings with expofiles/ and expowebcache/
- if you have those repos elsewhere, set up symlinks in the directory above troggle
  so that troggle thinks they are siblings

Troggle itself
-------------
Choose the directory where you will keep troggle, (which must be named "troggle")
and git clone troggle into it using the following command:

git clone git://expo.survex.com/troggle
or more reliably
git clone ssh://expo@expo.survex.com/home/expo/troggle
but you will have to set up the ssh key-exchange with the server to make that work.
Key-exchange instructions: http://expo.survex.com/handbook/computing/keyexchange.html

If you want to work on the source code and be able to commit, your account will need to be 
added to the troggle project members list. Contact wookey  at wookware dot org to get this set up.

Next, you need to fill in your local settings. Copy _deploy/WSL/localsettingsWSL.py 
to a new file called localsettings.py and edit it and settings.py to match
your machine's file locations. 
Follow the instructions contained in the file to fill out your settings.

{ in _deploy/old/ we have these which are all very out of date:
  localsettings-expo-live.py is the python2.7 settings for the server.
  localsettingsubuntu.py
  localsettingsdocker.py
  localsettingswindows.py
  localsettingspotatohut.py
}

Python3, Django, and Database setup
-----------------------------------
We are now using Django 2.2.19 

Read this: https://docs.djangoproject.com/en/3.0/topics/install/

We are installing with python3.7
Also : https://linuxize.com/post/how-to-install-pip-on-ubuntu-20.04/ ]

Install Django using pip, not with apt, on your test system.
You will also definitely need sqlite3 even if you are planning to use another
database. Sqlite3 is used by the test harness system.
Your Linux installation almost certainly already includes python3 and pip3 but
in case it doesn't install those like this:

$ sudo apt update
$ sudo apt dist-upgrade
$ sudo apt install python3
$ sudo apt install python3-pip
$ sudo apt install sqlite3 sqlite3-doc
$ cd /usr/bin
$ sudo ln -s python3 python
$ sudo ln -s pip3 pip

Now install django etc.
$ cd [troggle]
$ sudo pip install -r requirements.txt

where requirements.txt is:
confusable-homoglyphs==3.2.0
Django==2.2
docutils==0.14
gunicorn==20.1.0
Pillow==5.4.1
pytz==2019.1
sqlparse==0.2.4
Unidecode==1.0.23

These minor things are as-standard on Debian Buster (10).

if you don't use sudo it will install them all in ~/.local/
and so will only be available for you, not everyone; and
the paths won't work to find troggle properly.

Pillow is an image handling package used to make
the prospecting map. tinymce is the wysiwyg in-browser
editor (disabled pending reinstatement)

$ pip list -o
will list all the pip python packages installed.

[NB we should test whether later verisons of tinymce work.]

venv option
-----------
Or use a python3 virtual environment: (python3.5 or later)
$ cd troggle
$ cd ..
$ python3.7 -m venv pyth37d2
(creates folder with virtual env)
$ cd pyth37d2
$ source bin/activate
(now install everything )
$ pip install -r requirements.txt

Testing the django installation
-------------------------------
Test things by running these commands:
$ django-admin --version
$ django-admin 
It should show a list of commands and complain:
"..only Django core commands are listed as settings are not properly configured" 

If you get an error when running
$ django-admin
Then run django-admin like this:
python /usr/local/lib/python3.8/dist-packages/django/bin/django-admin.py
which should fix the paths. From now on you should be able to run 
$ django-admin 
from within any folder on your machine.

Now do
$ cd troggle
$ python manage.py

You will get this error if everything is OK:  (error: No module named 'localsettings')
So now rename one of the relevant platform files, e.g. if you are on WSL on Windows:
$ mv localsettingsWSl.py localsettings.py

The git repo copies have got munged passwords.
So use sFTP to download the copies from expo.survex.com

otherwise, if you are on debian:
$ cp _deploy/debian/localsettings.py .

WARNING: only the WSL and debian variants are current in May 2021. All the others are so old that 
they will need serious work to be useable. Copy what you need from WSL and debian variants of
localsettings.py

Now try this again:
$ python manage.py

and in additon to the [django] command list, you will now gets command lsists for [auth],
[contenttypes], [core], and [sessions]. These are the modules (plugins) loaaded into django. 
[core] is the core of troggle (but not all of it: the input file parsers are not in [core]).

$ python manage.py check -v 3 --traceback

Ideally this will list settings imports and then say
System check identified no issues (0 silenced).

If you get an error your python sys.path is probably not set correctly yet. Do
$ python -m site
when you are in your troggle directory to see the list of paths python looks 
for when it is searching for packages (both django and troggle). 
Ensure that the path to the troggle/ directory is in the list.
It should be at the top, which is where the current working directory is.

Now:
python  manage.py check -v 3 --deploy
which will give security warnings for deployment. Various middleware settings and 
a warning not to use DEBUG=true in deployment. We always have DEBUG=True set for troggle
as otherwise the users get useless error messages.

python -Wall manage.py check 
Gives warnings of deprecated Django which should be fixed asap.

Now you need to edit the following settings in your localsettings.py file to match your
development machine:

FILES = Path('/mnt/f/expofiles/')
EXPOFILES = Path('/mnt/f/expofiles/')

All the other settings (drawings, expoweb etc.) will work fine if they are parallel directories
to the directory you installed troggle into. The troggle code can find out itself where it
is living.

If you do not have a local copy of /expofiles/ (40 GB), you can use the expo server copy if
you set:
EXPOFILESREMOTE = TRUE
and then the FILES and EXPOFILES setings will be ignored.

You will probably want to change the various EMAIL settings too.

Now do
python  manage.py test -v 2
Runs our test suite.

If you get an error, and you probably will, have a look in the test file, e.g. for this error:
FAIL: test_page_folk (troggle.core.TESTS.tests.PageTests) 
look in the file      troggle/code/TESTS/tests.py in the class PageTests.
It will also say:
  File "/mnt/c/EXPO/troggle/core/TESTS/tests.py", line 266, in test_page_folk
which means that the asssert failure in on line 266 of troggle/code/TESTS/tests.py
and that the failure function is test_page_folk()
If you look at this you will see that line 264 is:
        response = self.client.get('/folk/index.htm')

so this file is missing. Duh. Of course it is. We downloaded troggle from git but we didn't run
the standalone script to generate the folk list. It is top of the list in
http://expo.survex.com/handbook/troggle/scriptscurrent.html#folk

So do this:
cd ../expoweb/folk
python ../scripts/make-folklist.py <folk.csv >index.htm
cd ../../troggle

Or just sFTP a copy from expo.survex.com

and run the tests again:

python  manage.py test -v 2

Yeah if you were paying attention, you will see that this has done a git commit for a test file
in the drawings repo locally.
Sorry about that. the test harness does not yet undo that. So you will need ot manually reverse that commit: go into your favourite git tool. In VS code the command is Commit: Undo last commit

If yo have run the test suite several times you will need to undo several commits (and delete the files producd by the test suite).

Registering troggle as a django application
-------------------------------------------
In your troggle directory run
$ django-admin 
and check you got the same output as before.

Then try 
$ python manage.py
It may not work. But if it does it will
now show a superset of the previous output: the [django] commands
available but also other options. Run
$ python manage.py check
$ python manage.py diffsettings
This last one shows everything set in global settings, settings and localsettings.
Anything different from global settings  (django built-in) has '###' appended.

$ python manage.py test -v 2
Tests that it can create a database from all the model files.

$ python manage.py migrate
Tests the uptodateness of your sqlite database.
$ python manage.py help migrate
explains what this does and gives extra command line options.

If you got an error traceback with
$ python manage.py
then the settings registration of troggle with django is incomplete.
Delete all your cached .pyc files and try again.
You probably have a mistake in your settings.py or localsettings.py files.

CSS and media files
-------------------
Temporarily we are not using the STATICFILES capability but are instead
serving css files from troggle/media/.. (see urls.py) using 
view_surveys.cssfilessingle 
i.e.
cssfilessingle() in core/view_surveys.py

Setting up survex
-----------------
You need to have survex installed as the command line tools 'cavern' is
used as part of the survex import process.
$ sudo apt install survex

Setting up tables and importing survey data
-------------------------------------------
Run 
$ sudo python databaseReset.py
from the troggle directory will give you instructions.

[ NB Adding a new year/expedition requires adding a column to the
folk/folk.csv table - a year doesn't exist until that is done.]

Database
--------
If you want to use MySQL or Postgresql, download and install them. 
However, you can also use Django with sqlite3

MariaDB database
----------------
Start it up with
$ sudo mysql -u -p
when it will prompt you to type in the password. Get this by reading the settings.py file in use on the server.
then
> CREATE DATABASE troggle;
> use troggle;
> exit;

Note the semicolons.

You can check the status of the db service:
$ sudo systemctl status mysql

You can start and stop the db service with
$ sudo systemctl restart mysql.service
$ sudo systemctl stop mysql.service
$ sudo systemctl start mysql.service

While logged in at a terminal session as expo on expo.survex.,com

$ mysql -h localhost -u expo -p<password>
will get you the MariasDb command prompt: https://www.hostwinds.com/guide/how-to-use-mysql-mariadb-from-command-line/

then (Note the SEMICOLONS !):
>drop database troggle;
>create database troggle;
>quit
Somewhere I have notes for the GRANT PRIVS type runes...

Ah yes:
CREATE DATABASE troggle;
GRANT ALL PRIVILEGES ON troggle.* TO 'expo'@'localhost' IDENTIFIED BY 'somepassword'; FLUSH PRIVILEGES; (at mysql root prompt)

(explained on https://chartio.com/resources/tutorials/how-to-grant-all-privileges-on-a-database-in-mysql/)
(but you need to create the database too)

The GRANT ALL PRIVILEGES  bit requires you to logon in to MariaDB as root. sudo doesn't cut it.
these permissions are set in a different 'info' database which usually is untouched even if database troggle gets creamed.


PERMISSIONS
https://linuxize.com/post/usermod-command-in-linux/

sudo usermod -a expo www-data
adds expo to the www-data group which is what the webserver uses, and thus so the user troggle is acting as when
running live.

sudo usermod -a expo expocvs
the expocvs group is used for git & hg

all the users should bve in this group

Running a Troggle server
------------------------
For high volume use, Troggle should be run using a web server like apache. 
However, a quick way to get started is to use the development server built into Django. 
This is limited though: directory redirection needs apache.

To do this, run 
$ python manage.py runserver 8000 -v 3
from the troggle directory. This runs it on port 8000 so you see the website
at http://localhost:8000/

gunicorn also works. This runs with 9 workers (suitable for a 4-core processor):
$ gunicorn --reload -w 9 -b :8000 wsgi

EXTRAS
------
cgit - https://git.zx2c4.com/cgit/about/
search - https://www.ibm.com/developerworks/opensource/library/os-xapianomega/index.html

Running a Troggle server with Apache
------------------------------------
Troggle also needs these aliases to be configured. These are set in 
/home/expo/config/apache/expo.conf
on the expo server.

At least these need setting:
DocumentRoot              /home/expo/expoweb
WSGIScriptAlias        /  /home/expo/troggle/wsgi.py
<Directory /home/expo/troggle>
	<Files wsgi.py>
		Require all granted
	</Files>
</Directory>

Alias       /expofiles    /home/expo/expofiles
Alias       /photos       /home/expo/webphotos
Alias       /map          /home/expo/expoweb/map
Alias       /javascript   /usr/share/javascript
Alias       /static/      /home/expo/static/
ScriptAlias /repositories /home/expo/config/apache/services/hgweb/hgweb.cgi

(The last is just for mercurial which will be remoived during 2020).

These two are not necessary as Django will serve these (see urls.py), but 
it may be faster for apache to serve them first:
Alias       /expofiles    /home/expo/expofiles
Alias       /static/      /home/expo/static/


Unlike the "runserver" method, apache requires a restart before it will use
any changed files:

sudo service apache2 restart

Olly's comments 20 July 2020:
olly: looking at /lib/systemd/system/apache2.service suggests so

olly: ExecStart=/usr/sbin/apachectl start
olly: ExecStop=/usr/sbin/apachectl stop
olly: ExecReload=/usr/sbin/apachectl graceful

Experimental additions
----------------------
These are untried tools which help us document how troggle works.

pip install pygraphviz 
pip install pyparsing pydot # installs fine
django extension graph_models # https://django-extensions.readthedocs.io/en/latest/graph_models.html