mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2024-11-29 21:32:00 +00:00
224 lines
16 KiB
HTML
224 lines
16 KiB
HTML
pythin<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<title>Handbook Troggle Laptop</title>
|
|
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
|
|
</head>
|
|
<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
|
|
<h2 id="tophead">CUCC Expedition Handbook</h2>
|
|
|
|
<h1>Troggle - software development laptop</h1>
|
|
<img src="https://www.python.org/favicon.ico" width=64 hspace="20" align="right" alt='python logo'>
|
|
|
|
<h2>Software development machine</h2>
|
|
<p>For troggle itself, you need a linux machine. We all use Debian or Debian-derived machines (Debian itself, Ubuntu, Xubuntu etc.) but other forms of linux may work. Running Debian or Ubuntu under WSL on Windows 10 works fine.
|
|
<p>If you want to work on the troggle source code and be able to commit to the server git repo,
|
|
your account will need to be added to the troggle project members list. Contact wookey at wookware dot org to get this set up.
|
|
|
|
<p>This page is a work in progess. Text will be moved here from
|
|
|
|
<pre>
|
|
http://expo.survex.com/repositories/troggle/.git/tree/README.txt
|
|
handbook/computing/yourlaptop.html
|
|
</pre>
|
|
<img src="https://www.debian.org/logos/openlogo-100.jpg" align="right" hspace="20" alt='debian logo'>
|
|
|
|
|
|
|
|
<h2 id="os">Installing linux</h2>
|
|
<p>If you don't already know how to do this, then you should probably not be attempting to work on the troggle code. But in case you are an experienced linux user who has always had someone else set up the system for them, then Ubuntu is the easiest and more forgiving to install, either directly on the computer or inside WSL. Because we are using fairly old releases of Django, you will want Ubuntu-20.04
|
|
<ul>
|
|
<li>Install in <a href="https://docs.microsoft.com/en-us/windows/wsl/install">WSL on Windows 10</a>.
|
|
<li>Install <a href="https://ubuntu.com/download/desktop">Ubuntu 20.04 LTS</a>.
|
|
</ul>
|
|
<p>and familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in <a href="serverconfig.html">The Expo Server</a>. We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on Django.
|
|
|
|
<h3>WSL on Windows</h3>
|
|
<p>The standard documentation for Ubuntu or debian below all works, but you should first skim the
|
|
<a href="../computing/winlaptop.html">Windows expo laptop</a> description of WSL1 and WSL2. The default
|
|
is that WSL2 will be installed, but all our practical experience so far is with WSL1.
|
|
|
|
<h2 id="python">Installing python</h2>
|
|
<p>Python is not installed by default usually, and in any case we need specific versions to be installed. For Ubuntu 20.04 the default is python3.9 but this is incompatible with standard debian Buster which is what is live on our server, so we also need python3.7 . If you are planning on helping the migration of troggle from debian Buster (v10) to Bullseye (v11) then you will also want python3.8 .
|
|
<pre><code>sudo apt install python3 python3-pip
|
|
sudo apt install software-properties-common
|
|
sudo add-apt-repository ppa:deadsnakes/ppa
|
|
sudo apt install python3.7 python3.7-venv python3.7-doc binutils binfmt-support
|
|
sudo apt install sqlite3 sqlite3-doc
|
|
sudo apt install survex
|
|
cd /usr/bin
|
|
sudo ln -s python3 python
|
|
sudo ln -s pip3 pip </code></pre>
|
|
|
|
<p>You will also definitely need sqlite3 even if you are planning to use another
|
|
database. Sqlite3 is used by the test harness system.
|
|
<p>Note that when you install survex it installs a shed load of packages that it needs.
|
|
<p>We do <em>not</em> install Django at this point. We will be installing Django in a separate virtual
|
|
environment (a 'venv'), not in the main linux system.
|
|
|
|
<h2 id="links">Creating folders and soft links</h2>
|
|
<p>We need to have the folder structure that troggle expects. On your own machine
|
|
you will have your own logon id, and you do not need to create an 'expo' user, but
|
|
you do need to create folders where and an 'expo' user would have them. Make links there
|
|
to wherever you have actually installed the repos. So if you have installed all your repos
|
|
in <var>/mnt/c/EXPO/</var>, then you would need to do this:
|
|
<pre><code>cd ~
|
|
cd ..
|
|
sudo mkdir expo
|
|
cd expo
|
|
sudo ln -s /mnt/c/EXPO/expoweb expoweb
|
|
sudo ln -s /mnt/c/EXPO/troggle troggle
|
|
sudo ln -s /mnt/c/EXPO/loser loser
|
|
sudo ln -s /mnt/c/EXPO/drawings drawings
|
|
sudo ln -s /mnt/c/EXPO/expofiles expofiles
|
|
sudo ln -s /mnt/c/EXPO/expowebcache expowebcache
|
|
sudo mkdir expowebcache/3d
|
|
cd ..
|
|
ls -tlA expo</pre></code>
|
|
<p>If you do not have a local copy of the 40GB /expofiles/, don't worry. Later on we can set 'EXPOFILESREMOTE = True' and your test system will use the live expofiles on expo.survex.com (read only).
|
|
|
|
<h2>Installing Django and troggle</h2>
|
|
<img src="Django_Logo-420x180.png" align="right" hspace="20" width='210' alt='django logo'>
|
|
<p>The important point to note
|
|
here is that unless you are doing something fairly trivial, or you are a git genius,
|
|
it is sensible to set up a python virtual environments to hold duplicate copies of both troggle and Django code.
|
|
Then you will be able to check very quickly that your edited version of troggle runs with old, current
|
|
and pre-release versions of python and of Django;
|
|
and you will more easily be able to manage problems with incompatible versions of Django plugins as installing and upgrading the dependent packages is very fast.
|
|
|
|
<h3 id="venv">Installing a venv</h3>
|
|
|
|
<p>We set up
|
|
a <var>venv</var> specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and <a href="https://docs.djangoproject.com/en/3.2/releases/2.2/">Django 2.2.19</a>. See the <a href="https://docs.python.org/3.7/library/venv.html">standard python documentation on venv</a> for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it <var>python3.7 -m venv py37d22</var>.
|
|
Note that we are creating it as a sibling folder to /expoweb/ . Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':
|
|
<pre><code>cd ~
|
|
cd ../expo
|
|
python3.7 -m venv py37d22
|
|
cd py37d22
|
|
source bin/activate
|
|
pip list -o</pre></code>
|
|
<p>The last command lists the default packages installed in the venv. This is for comparison later, after we
|
|
have installed troggle, Django and dependencies. You will get a warning that you have an out of date version of pipbut this is as we want: we are using a version of pip appropriate for the older version of python within
|
|
the venv.
|
|
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' hspace="20" alt='troggle logo'/></a>
|
|
|
|
<h3 id="venv">Installing the troggle dependencies</h3>
|
|
<p>The first time you do this on a new machine you can't complete the pip installation of Django as you have not yet got the
|
|
dependencies appropriate for troggle - because you have not yet cloned the troggle repo. So the first time it is easiest to just create requirements.txt yourself with a text editor. Without using git yourself, you can get the file from the website at <a href="http://expo.survex.com/repositories/troggle/.git/tree/requirements.txt">requirements.txt</a>. If you have already cloned all the repos, then just copy it.
|
|
<pre><code>cp ../troggle/requirements.txt .</pre></code>
|
|
where <var>requirements.txt</var> (note the capitalisation of the listed packages) is:
|
|
<pre><code>confusable-homoglyphs==3.2.0
|
|
Django==2.2
|
|
docutils==0.14
|
|
gunicorn==20.1.0
|
|
Pillow==5.4.1
|
|
pytz==2019.1
|
|
sqlparse==0.2.4
|
|
Unidecode==1.0.23</pre></code>
|
|
This will pick up the latest Django 2.2.x available, but all the other dependencies are pinned. These
|
|
dependencies are as-standard on debian Buster (10) and you will want to experiment with upgrading them
|
|
for Bullseye (v11).
|
|
<p>Once you have the file, install the listed dependencies like this:
|
|
<pre><code>pip install -r requirements.txt
|
|
pip list -o</pre></code>
|
|
|
|
|
|
|
|
<p>Pillow is an image handling package used to make
|
|
the prospecting map (currently disabled,
|
|
see <a href="http://expo.survex.com/prospecting_guide/">/prospecting_guide/</a>).<br>
|
|
tinymce is the wysiwyg in-browser
|
|
editor (disabled pending reinstatement)
|
|
|
|
<p>This is also documented in the <a href="trogdjangup.html">updating Django for troggle page</a>
|
|
where it describes upgrading and testing with later versions of Django.
|
|
|
|
<p>If you have not used pip before, read <a href="https://linuxize.com/post/how-to-install-pip-on-ubuntu-20.04/">this</a>
|
|
|
|
|
|
|
|
<h3 id="troginstall">Installing troggle</h3>
|
|
<p>The <var>:troggle:</var> repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/Django template code. This is over 600 files in over 400 folders, but only 38MB in size.
|
|
|
|
<h4>key exchange</h4>
|
|
<p>Follow this link to <a href="../computing/keyexchange.html">register a key with the expo server</a> to get git access if you have not already cloned the <var>:troggle:</var> repo.
|
|
|
|
<h4>git clone troggle</h4>
|
|
<p>This creates a folder /troggle/ containing all the troggle code. </p>
|
|
|
|
<pre><code>cd ~
|
|
cd ../expo
|
|
git clone ssh://expo@expo.survex.com/home/expo/troggle</code></pre>
|
|
|
|
|
|
<p>Now we create soft links within the venv to all the repo folders and wherever you have a local copy of /expofiles/ and /expowebcache/:
|
|
<pre><code>cd py37d22
|
|
sudo ln -s /mnt/c/EXPO/expoweb expoweb
|
|
sudo ln -s /mnt/c/EXPO/troggle troggle
|
|
sudo ln -s /mnt/c/EXPO/loser loser
|
|
sudo ln -s /mnt/c/EXPO/drawings drawings
|
|
sudo ln -s /mnt/c/EXPO/expofiles expofiles
|
|
sudo ln -s /mnt/c/EXPO/expowebcache expowebcache</code></pre>
|
|
|
|
<p>If you have not been using git on this machine until now, you will need to identify yourself before you go any further.
|
|
<pre><code>git config --global user.email "you@example.com"
|
|
git config --global user.name "Your Name"</code></pre>
|
|
|
|
<h4>do the basic health checks</h4>
|
|
<p>This all checks that the installation has completed properly.
|
|
<pre><code>django-admin
|
|
python manage.py
|
|
python manage.py check -v 3 --traceback</code></pre>
|
|
<p>The first line <var>django-admin</var> will complain that it has not got a SETTINGS file, but that's fine. If it crashes though, you have not managed to install the software completely.
|
|
<p>The most important is the <var>python manage.py check</var>. If this works, then you have installed the software correctly. Now we test whether troggle is configured correctly:
|
|
|
|
|
|
<pre><code>python manage.py test -v 3 --traceback</code></pre>
|
|
<p>This will run the entire troggle test suite of over 70 tests (it takes only a few seconds). It will probably have a dozen or so failures which will probably be because it can't find the various bits of the rest of the website. So the next task is to edit the SETTINGS files to match your machine and folder structure. So find the appropriate copy of the localsettings in /_deploy/ and copy it into the main troggle folder:
|
|
<pre><code>cp _deploy/wsl/localsettingsWSL.py localsettings.py</code></pre>
|
|
<p>Now edit it and insert useful values for EXPOUSERPASS [e.g. cavey:beery], EXPOADMINUSERPASS [e.g. beery:cavey], SECRET_KEY and make sure that LIBDIR refers to the actual version of python you are using (sorry, this should be automatic).
|
|
<p>Check that FILES and EXPOFILES are set to wherever you have put /expofiles/ or set 'EXPOFILESREMOTE = False' which will use expofiles on expo.survex.com instead.
|
|
<p>Set <a href="https://docs.djangoproject.com/en/3.2/topics/email/#s-configuring-email-for-development">EMAIL_HOST and EMAIL_HOST_PASSWORD</a> to an email account you control that can send email. Then troggle can email you when some things go wrong. This may mean having to set EMAIL_PORT and MAIL_USE_TLS too (this is not used in troggle currently). Set EXPOUSER_EMAIL and EXPOADMINUSER_EMAIL to your own email address while you are doing software development. All these will be different when troggle is deployed on the public server.
|
|
|
|
<h4>check full functionality</h4>
|
|
<p>The test suite hardly skims the surface of what troggle does. You need to manually test all these too:
|
|
<br>- <var><a href="http://localhost:8000/pathsreport">http://localhost:8000/pathsreport</a></var>
|
|
<br>- <var><a href="http://localhost:8000/stats">http://localhost:8000/stats</a></var>
|
|
<br>- <var><a href="http://localhost:8000/people">http://localhost:8000/people</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expedition/2018">http://localhost:8000/expedition/2018</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/surveyscans/2019/walletindex.html">http://localhost:8000/expofiles/surveyscans/2019/walletindex.html</a></var>
|
|
<br>- <var><a href="http://localhost:8000/survey_scans/">http://localhost:8000/survey_scans/</a></var>
|
|
<br>- <var><a href="http://localhost:8000/survexfile/caves/">http://localhost:8000/survexfile/caves/</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx">http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx</a></var>
|
|
<br>- <var><a href="http://localhost:8000/1623/291/291.html">http://localhost:8000/1623/291/291.htm</a></var>
|
|
<br>- <var><a href="http://localhost:8000/caves">http://localhost:8000/caves</a></var>
|
|
<br>- <var><a href="http://localhost:8000/admin/doc/models/core.expedition/">ttp://localhost:8000/admin/doc/models/core.expedition/</a></var> (admin login required)
|
|
<br>- <var><a href="http://localhost:8000/survexfile/204">http://localhost:8000/survexfile/204</a></var>
|
|
|
|
Nearly half the code deals with importing and parsing data, so you need to test that a full data import works. Run the full data import <var>troggle$ python databaseReset.py reset R000</var>. It should take about 5 minutes to import everything.
|
|
|
|
<h4>Use git to commit your edits</h4>
|
|
<p>You need to know git. Sorry, but there it is. See <a href="../computing/repos.html">our git repositories</a> and <a href="../computing/qstart-git.html">our git cheat sheet</a>.
|
|
|
|
<h3 id="dbtools">Helpful database tools and scripts</h3>
|
|
<img class="onleft" width = "100px" src="https://mariadb.com/kb/static/images/logo-2018-black.95f5978ae14d.png">
|
|
<img class="onright" width = "80px" src="https://sqlite.org/images/sqlite370_banner.gif">
|
|
<p>The public server uses a <a href="https://mariadb.org/about/">MariaDB SQL database</a> and development is usually done using a single-user <a href="https://sqlite.org/about.html">sqlite database</a> which is a standard Django option.
|
|
<p>
|
|
You will find it very, very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. This is vital when doing Django migrations between Django versions. A light-weight, simple db browser is <a href="https://sqlitebrowser.org/">DB Browser for SQLite</a>. Connecting directly the the MariaDB database with a control panel or <a href="https://www.mysql.com/products/workbench/">workbench</a> gives even more tools and documentation capabilities.
|
|
|
|
<hr />
|
|
|
|
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
|
|
|
|
Return to: <a href="trognotes.html">Troggle programmers' guide</a><br />
|
|
Troggle index:
|
|
<a href="trogindex.html">Index of all troggle documents</a><br />
|
|
<hr />
|
|
</body>
|
|
</html>
|