2021-10-27 19:25:37 +01:00
<!DOCTYPE html>
2021-10-22 14:18:09 +01:00
< html >
< head >
< meta http-equiv = "Content-Type" content = "text/html; charset=utf-8" / >
< title > Handbook Troggle Laptop< / title >
2024-02-09 00:01:17 +00:00
< link rel = "stylesheet" type = "text/css" href = "/css/main2.css" / >
2021-10-22 14:18:09 +01:00
< / head >
2022-06-24 23:17:32 +01:00
< body >
< style > body { background : #fff url ( /images/style/bg-system.png ) repeat-x 0 0 } < / style >
2021-10-22 14:18:09 +01:00
< h2 id = "tophead" > CUCC Expedition Handbook< / h2 >
< h1 > Troggle - software development laptop< / h1 >
< img src = "https://www.python.org/favicon.ico" width = 64 hspace = "20" align = "right" alt = 'python logo' >
2021-10-27 19:25:37 +01:00
2021-10-22 14:18:09 +01:00
< h2 > Software development machine< / h2 >
2024-12-15 21:46:01 +00:00
< p > For troggle itself, you need a linux machine. We all use Debian or Debian-derived machines (Debian itself, Ubuntu, Xubuntu etc.) but other forms of linux may work. Running Debian or Ubuntu under WSL on Windows 10 or 11 works fine.
2021-10-22 19:04:23 +01:00
< p > If you want to work on the troggle source code and be able to commit to the server git repo,
your account will need to be added to the troggle project members list. Contact wookey at wookware dot org to get this set up.
2021-10-22 14:18:09 +01:00
2023-01-26 23:36:45 +00:00
< h3 > Before you start< / h3 >
< p > Make sure you are familiar with the debugging tools already built into troggle. You may not
need to write any new software for what you need to do. Look at the tools and reports listed on the
< a href = "/controlpanel" > Troggle Control Panel< / a > page (needs 'expoadmin' login password.)
< p > Maybe what you need is simply some fixes or workarounds in the imported data files, not in the python code.
2021-10-27 19:25:37 +01:00
< h3 > Prerequisites< / h3 >
2024-12-15 21:46:01 +00:00
< p > You need to already know really quite a lot about what troggle does, and how it is used in practice. Also you will have installed git and set up the key-pair as documented in < a href = "../computing/bulkupdatelaptop.html" > your bulk update laptop configuration< / a > . In particular read the information there about VS code and git.
2021-10-27 19:25:37 +01:00
2024-12-15 21:46:01 +00:00
< div style = "margin-left: 10%" >
< h3 > Short-cut< / h3 >
2024-12-15 21:49:05 +00:00
If you have never done any bulk updates, and don't understand git, but want to look at the code and maybe run it, then there is a shortcut. You do not actually need to do all the Bulk Update machine configuration first.
2024-12-15 21:46:01 +00:00
< p > You can download the code without having a key-pair like this:
< pre > < code > $ mkdir ~/expo
$ cd ~/expo
$ git clone http://expo.survex.com/repositories/troggle/.git< / code > < / pre >
This will create < var > ~/expo/troggle/< / var > and get you a copy of the configuration scripts
< ul > < var >
< li > troggle/_deploy/wsl/os-trog24.04.sh
< li > troggle/_deploy/wsl/venv-trog24.04.sh
< / var >
< / ul >
but you need to make them executable before they will work:
< pre > < code > $ cd ~/expo/troggle
$ chmod +x *.sh< / code > < / pre >
2024-12-15 21:49:05 +00:00
Now go to < a href = "#scripts" > scripts< / a >
2024-12-15 21:46:01 +00:00
< / div >
< h3 > Read more here..< / h3 >
2021-10-22 14:18:09 +01:00
< p > This page is a work in progess. Text will be moved here from
2021-10-22 19:04:23 +01:00
2021-10-22 14:18:09 +01:00
< pre >
http://expo.survex.com/repositories/troggle/.git/tree/README.txt
< / pre >
2021-10-22 21:43:02 +01:00
< img src = "https://www.debian.org/logos/openlogo-100.jpg" align = "right" hspace = "20" alt = 'debian logo' >
2021-10-22 14:18:09 +01:00
2021-10-22 19:04:23 +01:00
< h2 id = "os" > Installing linux< / h2 >
2022-10-26 20:30:48 +01:00
< p > If you don't already know how to do this, then you should probably not be attempting to work on the troggle code. But in case you are an experienced linux user who has always had someone else set up the system for them, then Ubuntu is the easiest and more forgiving to install, either directly on the computer or inside WSL. Ubuntu installs python and various essential python dev tools by default.
2024-11-25 22:29:54 +00:00
< p > The server is running fairly old, stable releases of Debian and Django, but you will probably want Ubuntu-24.04 which is compatible.
2021-10-22 19:04:23 +01:00
< ul >
2022-10-26 20:30:48 +01:00
< li > Linux machine
< ul >
2024-12-15 01:40:07 +00:00
< li > Install < a href = "https://ubuntu.com/download/desktop" > Ubuntu 24.04 LTS< / a >
2022-10-26 20:30:48 +01:00
< / ul >
< li > Windows 10 or 11 machine
2024-12-15 01:40:07 +00:00
< ul >
2024-12-15 02:12:08 +00:00
< li > Configure ☯ < a href = "../computing/wsllaptop.html" > WSL on a WIndows laptop< / a > . The standard documentation for Ubuntu or debian below all works, but you should first skim the
☀ < a href = "../computing/winlaptop.html" > Windows expo laptop< / a > configuration too.
2024-12-15 01:40:07 +00:00
< / ul >
2021-10-22 19:04:23 +01:00
< / ul >
2023-05-02 16:06:41 +01:00
< p > Before you do anything else, get yourself set up with a < a href = "../computing/keyexchange.html" > key-pair< / a > to access the software on the expo server properly.
2024-12-08 16:28:09 +00:00
2024-12-15 21:49:05 +00:00
< h3 id = "folders" > Directory (folder) structure< / h3 >
2024-12-08 16:28:09 +00:00
< p > This is up to you of course, but a suggested structure is like this ("~" means your own user home folder, e.g. < var > /home/philip/< / var > ):
< ul >
< li > ~/expo
< ul >
< li > ~/expo/troggle
< li > ~/expo/expoweb
< li > ~/expo/drawings
< li > ~/expo/loser< br > < br >
2024-12-15 00:32:40 +00:00
< li > ~/expo/.venv/ - virtual environment
2024-12-08 16:28:09 +00:00
< / ul >
< / ul >
2024-12-15 02:12:08 +00:00
< p > Do familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in < a href = "serverconfig.html" > The Expo Server< / a > . We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on Django.
2024-12-15 21:49:05 +00:00
< h4 id = "scripts" > Scripts< / h4 >
2022-10-26 20:30:48 +01:00
< p > There are two scripts in the < var > troggle< / var > folder which will do semi-automatically what is described below. Have a look at
< ul >
2024-12-15 00:32:40 +00:00
< li > < var > os-trog24.04.sh< / var >
2022-10-26 20:30:48 +01:00
< li > < var > venv-trog.sh< / var >
< / ul >
2024-12-15 00:32:40 +00:00
which you can read without installing by looking in:< br >
< a href = "https://expo.survex.com/repositories/troggle/.git/tree/_deploy/wsl" > troggle/.git/tree/_deploy/wsl< / a > .
2024-12-08 16:28:09 +00:00
2024-12-15 02:12:08 +00:00
< p > You will run < var > os-trog24.04.sh< / var > just once to install all the basics,and you will run < var > venv-trog.sh< / var > just once initially.
< p > We have two configurations for the virtual environment: 'dev' which uses the latest python and Django for speed, and 'server' which mimics the versions currently running on the server. [In December 2024 Django is 5 releases ahead of the version on the server (5.1 versus 3.2). Each has a list of ancilliary packages with the appropriate versions in dev.toml and server.toml.] < var > venv-trog.sh< / var > deals with all this python-specific stuff, libraries and Django plug-ins.
2022-10-27 12:19:16 +01:00
2024-12-15 21:46:01 +00:00
< p > < var > os-trog.sh< / var > takes a few minutes: it installs the subset of /expofiles/ you need to work with troggle. If you now want to install survex, therion etc. then run < var > os-survey.sh< / var > as these drag in a huge number of dependencies.
2024-12-15 02:12:08 +00:00
< h3 > Why no Docker container?< / h3 >
2024-11-25 22:29:54 +00:00
< p > Yes, it is true that this would greatly speed up on-boarding new programmers. Or < a href = "https://podman.io/" > podman< / a > .
2022-03-16 12:46:33 +00:00
< p > But there is the significant danger that containers would get copied around and deployed without being properly cleaned up:
resulting in configuration drift and a < a href = "https://martinfowler.com/bliki/SnowflakeServer.html" > snowflake server situation< / a > .
File permissions are a big issue.
2024-11-25 22:29:54 +00:00
< p > We should do both: create a Docker or Podman system for getting started, then transition programmers to script-based or recipe-based
2022-03-16 12:46:33 +00:00
provisioning so that systems are rebuilt cleanly. < a href = "http://www.cuyc.org.uk" > CUYC< / a > (who also use Django) have a bash script which sets up a new django
2024-12-15 00:32:40 +00:00
development system. We should copy that in the first instance. Alas, we haven't got around to doing any of this yet. However < var > uv< / var > now makes everything much, much simpler than when were were using < var > pip< / var > .
2021-12-30 22:21:41 +00:00
2023-07-09 13:01:39 +01:00
< h2 id = "git" > Configuring ubuntu< / h2 >
2024-12-15 00:32:40 +00:00
< p > In your home folder run < br >
~$ < var > bash os-trog24.04.sh< / var >
2024-12-15 02:12:08 +00:00
< br > copy it from the .git repo online to get hold of it. Run it in < var > /home/username< / var > and it will create a folder < var > expo< / var > and clone all the repositories into it after installing the software you need. It installs some of < var > expofiles< / var > but not < var > expofiles/photos/< / var > .
2022-03-16 12:46:33 +00:00
2024-12-15 00:32:40 +00:00
<!--
2022-11-12 14:09:02 +00:00
< pre > < code > sudo apt install python3 python3-pip -y
sudo apt install sqlite3 sqlite3-doc -y
sudo apt install survex -y
2022-11-15 17:50:16 +00:00
sudo apt install git gitk -y
2023-07-09 12:56:00 +01:00
< / code > < / pre >
Martin Green is not sure why, but historically we may have also installed:
< pre > < code >
sudo apt install software-properties-common -y
2022-11-12 16:02:23 +00:00
sudo apt install default-jdk -y
2022-11-12 14:09:02 +00:00
sudo apt install binutils binfmt-support -y
2023-07-09 12:56:00 +01:00
< / code > < / pre >
2024-12-15 00:32:40 +00:00
-->
2022-11-12 16:02:23 +00:00
2023-07-09 12:56:00 +01:00
< h4 > key exchange< / h4 >
< p > You need this so that you can upload your edited code to the git repo on the server.
< p > Follow this link to < a href = "../computing/keyexchange.html" > register a key with the expo server< / a > to get git access if you have
not already cloned the < var > :troggle:< / var > repo.
2021-10-22 19:04:23 +01:00
2023-07-09 12:56:00 +01:00
< h4 > Set your indentity< / h4 >
< p > On a new machine you need to configure your git identity:
< pre > < code > git config --global user.email "you@example.com"
git config --global user.name "Your Name"
git config --global pull.rebase true
< / code > < / pre >
2021-10-22 19:04:23 +01:00
2024-12-15 02:12:08 +00:00
2022-03-16 12:46:33 +00:00
< h4 id = "EXPOFILESREMOTE" > Remote EXPOFILES< / h4 >
< p > If you do not have a local copy of the 40GB /expofiles/, don't worry. Later on we can set < var > 'EXPOFILESREMOTE = True'< / var > in the
< var > localsettings.py< / var > file and your test system will use the live expofiles on < var > expo.survex.com< / var > (read only).
< p > If you do have < var > 'EXPOFILESREMOTE = True'< / var > then the forms which upload scans and photos to the server will not work as you expect.
They will upload to your local machine, but read the status of the folders from < var > expo.survex.com< / var > . So you will get
confusing and apparently inconsistent behaviour: e.g. you will upload a file but then be unable to see it.
< p > For development, you mostly only need a local copy of the wallets and scanned survey notes and sketches in
< var > expofiles/surveyscans< / var > which is less than 5GB.
2023-04-26 20:31:15 +01:00
< p > You can, if you like, have the expo photo archive collection somewhere else, not inside < var > expofiles< / var > , on a troggle development machine. To do this set the PHOTOS_ROOT appropriately in < var > troggle/localsettings.py< / var > . By default it is < var > PHOTOS_ROOT = EXPOFILES / 'photos'< / var > . This is handy if you want everything else in expofiles in your Linux home drive for speed, but don't care about speed for the 29GB of photos which can sit on an SDdrive.
2021-10-22 19:04:23 +01:00
2023-07-09 13:01:39 +01:00
2023-07-09 12:56:00 +01:00
< h2 id = "python" > Installing python libraries< / h2 >
< a href = "https://xkcd.com/1987/" > < img src = "https://imgs.xkcd.com/comics/python_environment.png" align = "right" hspace = "20" width = "230" alt = 'XKCD python install' > < / a >
< p > We do < em > not< / em > install Django at this point. We will be installing Django in a separate virtual
environment (a 'venv'), not in the main linux system.
2021-10-22 19:04:23 +01:00
< h3 id = "venv" > Installing a venv< / h3 >
2024-12-15 00:32:40 +00:00
< p > [We are revising this process. We are now (December 2024) using uv instead of pip, see
< a href = "https://nielscautaerts.xyz/python-dependency-management-is-a-dumpster-fire.html" > "dumpster fire".< / a > . DOCUMENTATION HERE NEEDS UPDATING URGENTLY, we do NOT use REQUIREMENTS.TXT
2024-11-25 22:29:54 +00:00
]
< p > In order to avoid compatability issues when deploying our code, we should develop using the same libraries that will be used in production of expo.survex.com. Expo.survex.com currently uses python 3.11.2, python 3.12 (standard with Ubuntu 24.04) is backwardly compatable as long as you do not use any new functions it should be fine. Venv allows us to specify which python libraries to use.< / p >
< p > [We also develop with Django 5 even though the server is still running Django 3.2, but so long as we are carefule, and test code before pushng to the server, this is fine.]
< p >
2024-12-15 02:12:08 +00:00
< strike >
2023-07-09 12:56:00 +01:00
Create and activate the virtual enviroment:
2021-10-22 19:04:23 +01:00
2023-07-09 12:56:00 +01:00
< pre > < code > cd ~/expo
python3 -m venv venvexpo
source venvexpo/bin/activate
< / code > < / pre >
2021-10-22 19:04:23 +01:00
2023-07-09 12:56:00 +01:00
Install the python libraries
2021-10-23 22:40:12 +01:00
2023-07-09 12:56:00 +01:00
< pre > < code > pip install -r requirements.txt< / code > < / pre >
< p > In case you have python version problems with venv, look at the instructions for < a href = "/handbook/computing/wsllaptop.html_edit" > python3.11 on WSL2< / a > .
2021-10-23 22:40:12 +01:00
2024-12-15 02:12:08 +00:00
< / strike >
< pre > < code > cd ~/expo
source .venv/bin/activate
< / code > < / pre >
will mean that you can start checking that the development version of the system (Django 5, python 3.13 etc.) is working.
< p > To swap to the server mimic environment, do:
< pre > < code > deactivate
cd ~/expo/troggle
cp < font color = red > server.toml< / font > pyproject.toml
uv sync
source .venv/bin/activate
< / code > < / pre >
to swap back to the development environment, do
< pre > < code > deactivate
cd ~/expo/troggle
cp < font color = red > dev.toml< / font > pyproject.toml
uv sync
source .venv/bin/activate
< / code > < / pre >
boh of these are extremely quick because they us < var > uv< / var > and everything is cached.
2021-10-23 22:40:12 +01:00
2021-10-29 20:16:55 +01:00
< h4 > do the basic Django health checks< / h4 >
2021-10-23 22:40:12 +01:00
< p > This all checks that the installation has completed properly.
2021-10-29 20:16:55 +01:00
< pre > < code > django-admin< / code > < / pre >
2021-10-27 19:25:37 +01:00
< p > The first line < var > django-admin< / var > will complain that it has not got a SETTINGS file, but that's fine.
See < a href = "djangostart.html" > django-admin initial output< / a > for what you should expect to see at this point.
2021-10-29 20:16:55 +01:00
If it crashes though, you have not managed to install the software completely.
If you get an error when running
< pre > < code > django-admin --version< / code > < / pre >
The version number is the version of Django you have installed. Double check that it is the one you meant to install and check with < a href = "trogdjangup.html" > our Django versions page< / a > .
< h4 > do the basic troggle health checks< / h4 >
< p > Now try
2023-07-09 12:56:00 +01:00
< pre > < code > python manage.py< / code > < / pre >
2021-10-29 20:16:55 +01:00
You will get an error: No module named 'localsettings'. Fixing this is described below, but for now try:
< pre > < code > python manage.py check -v 3 --traceback< / code > < / pre > < p > The most important is the < var > python manage.py check< / var > . If this works, then you have installed the software correctly.
< p >
The next task is to edit the SETTINGS files to match your machine and folder structure. So find the appropriate copy of the localsettings in /_deploy/ and copy it into the main troggle folder:
2021-10-23 22:40:12 +01:00
< pre > < code > cp _deploy/wsl/localsettingsWSL.py localsettings.py< / code > < / pre >
2021-10-29 20:35:52 +01:00
The git repo copies have got munged passwords.
localsettings.py is not stored in git, but the copy on the server does have the correct passwords.
So use sFTP to download localsettings.py from expo.survex.com to get these.
< p >
We have at one time made localsettings in /_deploy/ appropriate for
< ul >
< li > debian (server install, not development)
< li > Ubuntu
< li > Ubuntu under WSL (Windows System for Linux)
< li > docker container
< li > potatohut (local network only)
< / ul >
< p >
2021-12-06 01:01:21 +00:00
WARNING: only the WSL and debian variants are current in December 2021. All the others are so old that
2021-10-29 20:35:52 +01:00
they will need serious work to be useable. Copy what you need from WSL and debian variants of
localsettings.py
< p > Now edit localsettings.py and insert useful values for EXPOUSERPASS [e.g. cavey:beery], EXPOADMINUSERPASS [e.g. beery:cavey], SECRET_KEY. SECRET_KEY can be anything, it just has to be unique to each installation and invisible to anyone not a developer.
2022-03-16 12:46:33 +00:00
< p > Set < a href = "https://docs.djangoproject.com/en/4.0/topics/email/#s-configuring-email-for-development" > EMAIL_HOST and EMAIL_HOST_PASSWORD< / a > to an email account you control that can send email. Then troggle can email you when some things go wrong. This may mean having to set EMAIL_PORT and MAIL_USE_TLS too (this is not used in troggle currently). Set EXPOUSER_EMAIL and EXPOADMINUSER_EMAIL to your own email address while you are doing software development. All these will be different when troggle is deployed on the public server.
2021-10-23 22:40:12 +01:00
2021-10-29 20:35:52 +01:00
< p >
Now you need to edit the following settings in your localsettings.py file to match your
development machine, e.g. if you have /expofiles/ mounted on another disc:
< pre > < code > FILES = Path('/mnt/f/expofiles/')
EXPOFILES = Path('/mnt/f/expofiles/')< / code > < / pre >
< p >
All the other settings (drawings, expoweb etc.) will work fine if they are parallel directories
to the directory you installed troggle into. The troggle code can find out itself where it
is living.
< p >
If you do not have a local copy of /expofiles/ (40 GB), you can use the expo server copy if
you set:
< pre > < code > EXPOFILESREMOTE = TRUE< / code > < / pre >
2022-03-16 12:46:33 +00:00
and then the FILES and EXPOFILES setings will be ignored. (Except for the upload forms which will 'upload' files
< a href = "#EXPOFILESREMOTE" > to your local disc< / a > . )
2021-10-29 20:35:52 +01:00
2021-10-29 20:16:55 +01:00
< p >
Now try this again:
< pre > < code > python manage.py< / code > < / pre >
2021-10-29 20:35:52 +01:00
and in addition to the [django] command list, you will now gets command lists for [auth],
2021-10-29 20:16:55 +01:00
[contenttypes], [core], and [sessions]. These are the modules (plugins) loaaded into django.
[core] is the core of troggle (but not all of it: the input file parsers are not in [core]).
2021-10-31 08:05:26 +00:00
< p > Now:
< pre > < code > python -Wall manage.py check < / code > < / pre >
This loads the settings files and checks that all the Django packages and python libraries
imports all work. It gives warnings of deprecated Django which should be fixed.
You should not get any warnings on a fresh install on a new machine but you will
see warning and error messages when you are
trying to upgrade troggle to use later versions of Django.
2021-10-29 20:16:55 +01:00
< pre > < code > python manage.py check -v 3 --traceback< / code > < / pre >
Ideally this will list settings imports and then say< br >
< var > System check identified no issues (0 silenced)< / var > .
< p >
If you get an error your python sys.path is probably not set correctly yet. Do
< pre > < code > python -m site< / code > < / pre >
when you are in your troggle directory to see the list of paths python looks
for when it is searching for packages (both django and troggle). < br >
Ensure that the path to the troggle/ directory is in the list.
It should be at the top, which is where the current working directory is.
< p >
Now:
2021-10-31 08:05:26 +00:00
2021-10-29 20:16:55 +01:00
< pre > < code > python manage.py check -v 3 --deploy< / code > < / pre >
2021-10-31 08:05:26 +00:00
which will give security warnings for deployment. You will get various middleware settings
because we have not got https:// properly configured everywhere (we have unresolved issues with
the Django admin control panel and https:// in October 2021) and
2021-10-29 20:16:55 +01:00
a warning not to use DEBUG=true in deployment. We always have DEBUG=True set for troggle
2021-10-31 08:05:26 +00:00
as otherwise the users get useless error messages. This is completely contrary to Django
official security advice.
< details >
< summary > < em > Click triangle to see security warnings in detail< / em > < / summary >
< font color = "brown" > < pre >
?: (security.W001) You do not have 'django.middleware.security.SecurityMiddleware'
in your MIDDLEWARE so the SECURE_HSTS_SECONDS, SECURE_CONTENT_TYPE_NOSNIFF,
SECURE_BROWSER_XSS_FILTER, and SECURE_SSL_REDIRECT settings will have no effect.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only
session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your
MIDDLEWARE, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only
CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment. < / pre > < / font >
< / details >
2021-10-29 20:16:55 +01:00
2021-10-31 08:05:26 +00:00
< p >
2021-10-29 20:16:55 +01:00
If you got an error traceback with
< pre > < code > python manage.py< / code > < / pre >
then the settings registration of troggle with django is incomplete.
Delete all your cached .pyc files and try again.
You probably have a mistake in your settings.py or localsettings.py files.
< pre > < code > python manage.py diffsettings< / code > < / pre >
This last one shows everything set in global settings, settings and localsettings.
Anything different from global settings (django built-in) has '###' appended.
< pre > < code > python manage.py help migrate
python manage.py migrate< / code > < / pre >
Tests the uptodateness of your sqlite database. The help option
explains what this does and gives extra command line options.
2022-06-24 23:17:32 +01:00
< h4 id = "fullfunct" > check full functionality< / h4 >
2021-10-29 20:16:55 +01:00
< p > Now run the test suite:
< pre > < code > python manage.py test -v 3 --traceback< / code > < / pre >
2023-02-24 22:54:47 +00:00
< p > This will run the entire troggle test suite of ~90 tests (it takes only a few seconds).
2021-10-29 20:16:55 +01:00
2021-10-31 20:48:05 +00:00
< p >
If you get an error, and you probably will, have a look in the source code of the test, e.g. for this error:
< pre > < code > FAIL: test_page_folk (troggle.core.TESTS.tests.PageTests) < / code > < / pre >
look in the file
< var > troggle/code/TESTS/tests.py< / var > in the class < var > PageTests< / var > .
2021-10-29 20:16:55 +01:00
It will also say:
2021-10-31 20:48:05 +00:00
< pre > < code > File "/mnt/c/EXPO/troggle/core/TESTS/tests.py", line 266, in test_page_folk< / code > < / pre >
which means that the asssert failure in on line 266 of < var > troggle/code/TESTS/tests.py< / var >
and that the failure function is test_page_folk() .
2021-10-29 20:16:55 +01:00
If you look at this you will see that line 264 is:
2021-10-31 20:48:05 +00:00
< pre > < code > response = self.client.get('/folk/index.htm')< / code > < / pre >
2021-10-29 20:16:55 +01:00
so this file is missing. Duh. Of course it is. We downloaded troggle from git but we didn't run
2021-10-31 20:48:05 +00:00
the standalone script to generate the folk list.
The quickest thing to do when installing troggle is to simply download a copy from the server:
< pre > < code > wget -O ../expoweb/folk/index.htm expo.survex.com/folk/index.htm< / code > < / pre >
and run the tests again:
2021-10-29 20:16:55 +01:00
2021-10-31 20:48:05 +00:00
< pre > < code > python manage.py test -v 2< / code > < / pre >
2021-12-01 21:12:07 +00:00
< details > < summary > Click on the triangle to see how to run the < var > folk< / var > script. You don't need to do this now.< / summary >
2021-10-31 20:48:05 +00:00
< font color = "blue" >
The < var > folk< / var > generation script is top of the list in
< var >
http://expo.survex.com/handbook/troggle/scriptscurrent.html#folk< / var >
< p >
2021-10-29 20:16:55 +01:00
So do this:
2021-10-31 20:48:05 +00:00
< pre >
2021-10-29 20:16:55 +01:00
cd ../expoweb/folk
2022-06-25 15:00:07 +01:00
python ../scripts/make-folklist.py < folk.csv > index.htm
2021-10-29 20:16:55 +01:00
cd ../../troggle
2021-10-31 20:48:05 +00:00
< / pre >
< / font >
< / details >
2021-10-29 20:16:55 +01:00
2022-03-16 12:46:33 +00:00
< p > The test suite now tidies up after itself, so there should not be any temporary files left behind or local git commits that you
will need to clean up.
2021-10-29 20:35:52 +01:00
2023-02-24 22:54:47 +00:00
< p > The test suite has ~90 tests but does not cover all of what troggle does and does not use any real data. You need to manually test these too, < em > after< / em > you have done a full < a href = "trogimport.html" > data import< / a > :
2021-10-23 22:40:12 +01:00
< br > - < var > < a href = "http://localhost:8000/pathsreport" > http://localhost:8000/pathsreport< / a > < / var >
< br > - < var > < a href = "http://localhost:8000/stats" > http://localhost:8000/stats< / a > < / var >
2021-10-27 19:25:37 +01:00
< br > - < var > < a href = "http://localhost:8000/people" > http://localhost:8000/people< / a > < / var > (takes a minute or so)
2022-08-06 18:29:47 +01:00
< br > - < var > < a href = "http://localhost:8000/wallets/year/2019" > http://localhost:8000/wallets/year/2019l< / a > < / var >
2021-12-01 21:12:07 +00:00
< br > - < var > < a href = "http://localhost:8000/survexfile/caves/" > http://localhost:8000/survexfile/caves/< / a > < / var >
2021-10-23 22:40:12 +01:00
< br > - < var > < a href = "http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx" > http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx< / a > < / var >
2022-06-25 17:30:35 +01:00
< br > - < var > < a href = "http://localhost:8000/1623/291/291" > http://localhost:8000/1623/291/291< / a > < / var >
2021-10-23 22:40:12 +01:00
< br > - < var > < a href = "http://localhost:8000/caves" > http://localhost:8000/caves< / a > < / var >
< br > - < var > < a href = "http://localhost:8000/admin/doc/models/core.expedition/" > ttp://localhost:8000/admin/doc/models/core.expedition/< / a > < / var > (admin login required)
< br > - < var > < a href = "http://localhost:8000/survexfile/204" > http://localhost:8000/survexfile/204< / a > < / var >
2021-10-27 19:25:37 +01:00
< h4 > Check parsing and importing< / h4 >
< p >
Nearly half the code deals with importing and parsing data, so you need to test that a full data import works. Run the full data import < var > troggle$ python databaseReset.py reset R000< / var > . It should take about 5 minutes to import everything. See < a href = "trogimport.html" > Troggle Full Import< / a > for what you should expect to see.
2021-10-23 22:40:12 +01:00
< h4 > Use git to commit your edits< / h4 >
< p > You need to know git. Sorry, but there it is. See < a href = "../computing/repos.html" > our git repositories< / a > and < a href = "../computing/qstart-git.html" > our git cheat sheet< / a > .
2021-10-22 19:04:23 +01:00
2021-10-24 14:31:22 +01:00
< h3 id = "dbtools" > Helpful database tools and scripts< / h3 >
< img class = "onleft" width = "100px" src = "https://mariadb.com/kb/static/images/logo-2018-black.95f5978ae14d.png" >
< img class = "onright" width = "80px" src = "https://sqlite.org/images/sqlite370_banner.gif" >
< p > The public server uses a < a href = "https://mariadb.org/about/" > MariaDB SQL database< / a > and development is usually done using a single-user < a href = "https://sqlite.org/about.html" > sqlite database< / a > which is a standard Django option.
< p >
2022-06-25 13:54:54 +01:00
You will find it very, very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case)
and browse the data in the tables. This is vital when doing Django migrations between Django versions. A light-weight, simple db browser is < a href =
"https://sqlitebrowser.org/">DB Browser for SQLite< / a > . Connecting directly the the MariaDB database with a control panel or < a href =
"https://www.mysql.com/products/workbench/">workbench< / a > gives even more tools and documentation capabilities. See the < a href = "serverconfig.html" >
troggle server documentation< / a > for how to install MariaDB.
< p > When Ubuntu is running on WSL, it does not use systemctl. So you need specific instructions for installing MariaDB under WSL,
2022-06-25 14:19:07 +01:00
do what it says in these instructions first (sudo apt install ...etc.):
2022-06-25 13:54:54 +01:00
< ul >
< li > < a href = "https://segmentfault.com/a/1190000040671057/en" > Ubuntu-20.04 (WSL) install MySQL (MariaDB)< / a > .
< li > < a href = "https://stackoverflow.com/questions/52487644/install-mariadb-in-windows-subsystem-linux-wsl" >
Install MariaDB in Windows subsystem Linux (WSL)< / a > (if you have complications)
< / ul >
< p > Create a new dedicated administrative MariaDB user 'expo' who can access all databases. Log in to the MariaDB command with
< pre > sudo mysql
< / pre >
and execute these commands:
< pre >
GRANT ALL PRIVILEGES on *.* TO 'expo'@'%' IDENTIFIED BY 'my-secret-password-schwatzmooskogel' WITH GRANT OPTION;
SET PASSWORD FOR expo=PASSWORD('my-secret-password-schwatzmooskogel');
FLUSH PRIVILEGES;
QUIT;
< / pre >
and you will need to set this user and password in your < var > localsettings.py< / var > :
< pre >
DATABASE = {
'default': {
'ENGINE': 'django.db.backends.mysql', # 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
2022-06-25 14:19:07 +01:00
'NAME' : 'troggle',
2022-06-25 13:54:54 +01:00
'USER' : 'expo',
'PASSWORD' : 'my-secret-password-schwatzmooskogel',
2022-06-25 14:19:07 +01:00
'HOST' : '', # Set to empty string for localhost.
'PORT' : '', # Set to empty string for default.
2022-06-25 13:54:54 +01:00
}
}
< / pre >
2021-10-22 19:04:23 +01:00
2022-06-25 13:54:54 +01:00
< h4 > But it still does not work< / h4 >
2022-06-25 14:41:53 +01:00
< p > That is because we need to install the python tools that talk to mariadb. And while it 'will just work' for python3.9, the standard installed on the distro, with python3.10 it is a bit more work.
2022-06-25 14:19:07 +01:00
but note that there is a problem with using python 3.10 in that some bits of pip are not correct and you will get a
2022-06-25 14:41:53 +01:00
< var > ImportError: cannot import name 'html5lib'< / var > error.
TEMPORARILY use this hack from bootstrap:
2022-06-25 14:19:07 +01:00
< pre >
curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
pip install mysql-connector-python
< / pre >
installs the correct pip for python3.10 on Ubuntu. Which works, then
2022-06-25 13:54:54 +01:00
< pre >
2022-06-25 14:41:53 +01:00
sudo apt-get install python3.10-dev
sudo apt install libmariadbclient-dev
2022-06-25 13:54:54 +01:00
pip install mariadb
< / pre >
2022-06-25 14:41:53 +01:00
wwhich now has installed mariadb python stuff, but seems to have trashed my django installattion. Hmph. And pip.
2024-12-15 00:32:40 +00:00
< h2 > Getting a copy of live javascript libraries< / h2 >
To get the same javascript libraries as the expo server:
< pre > < code > scp -r expo@expo.survex.com:/usr/share/javascript/ ~/expo/troggle/javascript/
< / code > < / pre >
but you don't need these for most troggle development: copies of what is needed are in the troggle git repo. One example that is not in the troggle git repo is < a href = "https://aardgoose.github.io/CaveView.js/installation.html" > "CaveView"< / a > .
2021-10-22 14:18:09 +01:00
< hr / >
2024-12-15 20:19:07 +00:00
<!-- absolute addressing so that this fragment can be copied eveywhere and still work -->
Go on to: < a href = "/handbook/troggle/trogarch.html" > Troggle architecture< / a > < br / >
2021-10-22 14:18:09 +01:00
2024-12-15 20:19:07 +00:00
Return to: < a href = "/handbook/troggle/trognotes.html" > Troggle programmers' guide< / a > < br / >
2021-10-22 14:18:09 +01:00
Troggle index:
2024-12-15 20:19:07 +00:00
< a href = "/handbook/troggle/trogindex.html" > Index of all troggle documents< / a > < br / >
Go to: ※ < a href = "/handbook/computing/basiclaptop.html" > Basic laptop< / a > < br / >
Go to: ⁂ < a href = "/handbook/computing/surveylaptop.html" > Survey laptop< / a > < br / >
2024-12-15 20:26:35 +00:00
Go to: ☆ < a href = "/handbook/computing/bulkupdatelaptop.html" > Bulk Update laptop< / a >
☀ < a href = "/handbook/computing/winlaptop.html" > Windows Bulk Update laptop< / a > < br / >
Go to: ⚒ < a href = "/handbook/troggle/troglaptop.html" > Troggle development laptop< / a >
☯ < a href = "/handbook/computing/wsllaptop.html" > Troggle development WSL laptop< / a > < br / >
2022-06-24 23:17:32 +01:00
< hr / > < / body >
2021-10-22 14:18:09 +01:00
< / html >
