expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2025-12-15 18:37:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update docs to bullseye

Browse Source
This commit is contained in:
Philip Sargent
2022-03-16 12:46:33 +00:00
parent cc6605d06b
commit 8b0056ccd5
3 changed files with 113 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
handbook/troggle/troglaptop.html
View File

@@ -43,21 +43,28 @@ http://expo.survex.com/repositories/troggle/.git/tree/README.txt
<h3>Why no Docker container?</h3>
<p>Yes, it is true that this would greatly speed up on-boarding new programmers.
<p>But there is the significant danger that containers would get copied around and deployed without being properly cleaned up: resulting in configuration drift and a <a href="https://martinfowler.com/bliki/SnowflakeServer.html">snowflake server situation</a>. File permissions are a big issue.
<p>We should do both: create a Docker system for getting started, then transition programmers to script-based or recipe-based provisioning so that systems are rebuilt cleanly. CUYC (who also use Django) have a bash script which sets up a new django development system. We should copy that in the first instance. Alas, we haven't got around to doing any of this yet.
<p>But there is the significant danger that containers would get copied around and deployed without being properly cleaned up:
resulting in configuration drift and a <a href="https://martinfowler.com/bliki/SnowflakeServer.html">snowflake server situation</a>.
File permissions are a big issue.
<p>We should do both: create a Docker system for getting started, then transition programmers to script-based or recipe-based
provisioning so that systems are rebuilt cleanly. <a href="http://www.cuyc.org.uk">CUYC</a> (who also use Django) have a bash script which sets up a new django
development system. We should copy that in the first instance. Alas, we haven't got around to doing any of this yet.
<h2 id="python">Installing python</h2>
<a href="https://xkcd.com/1987/"><img src="https://imgs.xkcd.com/comics/python_environment.png" align="right" hspace="20" width="230" alt='XKCD python install'></a>
<p>Python is not installed by default usually, and in any case we need specific versions to be installed. For Ubuntu 20.04 the default is python3.9 but this is incompatible with standard debian Buster which is what is live on our server, so we also need python3.7 .
<p>If you are planning on helping the migration of troggle from debian Buster (v10) to Bullseye (v11) then you will also want python3.8 .
<p>Python is not installed by default usually, and in any case we need specific versions to be installed. For Ubuntu 20.04 the
default is python3.9
<p>If you are planning on eventually helping the migration of troggle from debian Bullseye (v11) to
<a href="https://www.debian.org/releases/">Bookworm (v12)</a> to then you will also want the appropriate later versions of python e.g. 3.10 below:
<pre><code>sudo apt install python3 python3-pip
sudo apt install sqlite3 sqlite3-doc
sudo apt install survex
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.7 python3.7-venv python3.7-doc binutils binfmt-support
sudo apt install python3.10 python3.10-venv python3.10-doc binutils binfmt-support
cd /usr/bin
sudo ln -s python3 python
sudo ln -s pip3 pip </code></pre>
@@ -87,7 +94,16 @@ sudo ln -s /mnt/c/EXPO/expowebcache expowebcache
sudo mkdir expowebcache/3d
cd ..
ls -tlA expo</pre></code>
<p>If you do not have a local copy of the 40GB /expofiles/, don't worry. Later on we can set 'EXPOFILESREMOTE = True' and your test system will use the live expofiles on expo.survex.com (read only).
<h4 id="EXPOFILESREMOTE">Remote EXPOFILES</h4>
<p>If you do not have a local copy of the 40GB /expofiles/, don't worry. Later on we can set <var>'EXPOFILESREMOTE = True'</var> in the
<var>localsettings.py</var> file and your test system will use the live expofiles on <var>expo.survex.com</var> (read only).
<p>If you do have <var>'EXPOFILESREMOTE = True'</var> then the forms which upload scans and photos to the server will not work as you expect.
They will upload to your local machine, but read the status of the folders from <var>expo.survex.com</var>. So you will get
confusing and apparently inconsistent behaviour: e.g. you will upload a file but then be unable to see it.
<p>For development, you mostly only need a local copy of the wallets and scanned survey notes and sketches in
<var>expofiles/surveyscans</var> which is less than 5GB.
<h2>Installing Django and troggle</h2>
<img src="Django_Logo-420x180.png" align="right" hspace="20" width='210' alt='django logo'>
@@ -100,13 +116,16 @@ ls -tlA expo</pre></code>
<h3 id="venv">Installing a venv</h3>
<p>We set up
a <var>venv</var> specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and <a href="https://docs.djangoproject.com/en/3.2/releases/2.2/">Django 2.2.19</a>. See the <a href="https://docs.python.org/3.7/library/venv.html">standard python documentation on venv</a> for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it <var>python3.7 -m venv py37d22</var>.
Note that we are creating it as a sibling folder to /expoweb/ . Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':
<p>We set up a <var>venv</var> specifically for python 3.9 (which is the standard version on our server which is running unmodified
Bullseye (debian v11) and <a href="https://docs.djangoproject.com/en/3.2/releases/3.2/">Django 3.2.12</a>. See the <a
href="https://docs.python.org/3.9/library/venv.html">standard python documentation on venv</a> for python 3.9.10.
Create a new venv for each version of python you are using. Get that venv installed right by explicitly stating the
python version to create it <var>python3.9 -m venv py39d32</var>. Note that we are creating it as a sibling folder to /expoweb/ .
Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':
<pre><code>cd ~
cd ../expo
python3.7 -m venv py37d22
cd py37d22
python3.9 -m venv py39d32
cd py39d32
source bin/activate
pip list -o</pre></code>
<p>The last command lists the default packages installed in the venv. This is for comparison later, after we
@@ -119,15 +138,17 @@ the venv.
dependencies appropriate for troggle - because you have not yet cloned the troggle repo. So the first time it is easiest to just create requirements.txt yourself with a text editor. Without using git yourself, you can get the file from the website at <a href="http://expo.survex.com/repositories/troggle/.git/tree/requirements.txt">requirements.txt</a>. If you have already cloned all the repos, then just copy it.
<pre><code>cp ../troggle/requirements.txt .</pre></code>
where <var>requirements.txt</var> (note the capitalisation of the listed packages) is:
<pre><code>confusable-homoglyphs==3.2.0
Django==2.2
<pre><code>asgiref==3.3.4
confusable-homoglyphs==3.2.0
coverage==5.5
Django==3.2
docutils==0.14
gunicorn==20.1.0
Pillow==5.4.1
pytz==2019.1
sqlparse==0.2.4
typing-extensions==3.7.4.3
Unidecode==1.0.23</pre></code>
This will pick up the latest Django 2.2.x available, but all the other dependencies are pinned. These
This will pick up the latest Django 3.2.x available, but all the other dependencies are pinned. These
dependencies are as-standard on debian Buster (10) and you will want to experiment with upgrading them
for Bullseye (v11).
<p>Once you have the file, install the listed dependencies like this:
@@ -136,7 +157,7 @@ pip list -o</pre></code>
<p>Pillow is an image handling package used to make
<p>Pillow (not currently in that package list) is an image handling package used to make
the prospecting map (currently disabled,
see <a href="http://expo.survex.com/prospecting_guide/">/prospecting_guide/</a>).<br>
tinymce is the wysiwyg in-browser
@@ -150,13 +171,17 @@ where it describes upgrading and testing with later versions of Django.
<h3 id="troginstall">Installing troggle</h3>
<p>The <var>:troggle:</var> repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/Django template code. This is over 600 files in over 400 folders, but only 38MB in size.
<p>The <var>:troggle:</var> repo is the python source code for troggle. This is what you will be editing. There are over 9,000 lines
of python code (excluding comments and blank lines) and nearly 3,000 lines of HTML/Django template code. This is over 600 files in
over 400 folders, but only 49MB in size (plus a .git repo of 32MB).
<h4>key exchange</h4>
<p>Follow this link to <a href="../computing/keyexchange.html">register a key with the expo server</a> to get git access if you have not already cloned the <var>:troggle:</var> repo.
<p>Follow this link to <a href="../computing/keyexchange.html">register a key with the expo server</a> to get git access if you have
not already cloned the <var>:troggle:</var> repo.
<h4>git clone troggle</h4>
<p>You will do a git clone to create a folder /troggle/ containing all the troggle code. You need to clone the 'python-3' branch, if you an see multiple branches.</p>
<p>You will do a git clone to create a folder /troggle/ containing all the troggle code. You need to clone the 'python-3' branch, if
you an see multiple branches.</p>
<pre><code>cd ~
cd ../expo
@@ -164,7 +189,7 @@ git clone ssh://expo@expo.survex.com/home/expo/troggle</code></pre>
<p>Now we create soft links within the venv to all the repo folders and wherever you have a local copy of /expofiles/ and /expowebcache/:
<pre><code>cd py37d22
<pre><code>cd py39d32
sudo ln -s /mnt/c/EXPO/expoweb expoweb
sudo ln -s /mnt/c/EXPO/troggle troggle
sudo ln -s /mnt/c/EXPO/loser loser
@@ -223,7 +248,7 @@ localsettings.py
<p>Now edit localsettings.py and insert useful values for EXPOUSERPASS [e.g. cavey:beery], EXPOADMINUSERPASS [e.g. beery:cavey], SECRET_KEY. SECRET_KEY can be anything, it just has to be unique to each installation and invisible to anyone not a developer.
<p>Set <a href="https://docs.djangoproject.com/en/3.2/topics/email/#s-configuring-email-for-development">EMAIL_HOST and EMAIL_HOST_PASSWORD</a> to an email account you control that can send email. Then troggle can email you when some things go wrong. This may mean having to set EMAIL_PORT and MAIL_USE_TLS too (this is not used in troggle currently). Set EXPOUSER_EMAIL and EXPOADMINUSER_EMAIL to your own email address while you are doing software development. All these will be different when troggle is deployed on the public server.
<p>Set <a href="https://docs.djangoproject.com/en/4.0/topics/email/#s-configuring-email-for-development">EMAIL_HOST and EMAIL_HOST_PASSWORD</a> to an email account you control that can send email. Then troggle can email you when some things go wrong. This may mean having to set EMAIL_PORT and MAIL_USE_TLS too (this is not used in troggle currently). Set EXPOUSER_EMAIL and EXPOADMINUSER_EMAIL to your own email address while you are doing software development. All these will be different when troggle is deployed on the public server.
<p>
Now you need to edit the following settings in your localsettings.py file to match your
@@ -240,7 +265,8 @@ is living.
If you do not have a local copy of /expofiles/ (40 GB), you can use the expo server copy if
you set:
<pre><code>EXPOFILESREMOTE = TRUE</code></pre>
and then the FILES and EXPOFILES setings will be ignored.
and then the FILES and EXPOFILES setings will be ignored. (Except for the upload forms which will 'upload' files
<a href="#EXPOFILESREMOTE">to your local disc</a>. )
<p>
Now try this again:
@@ -317,7 +343,7 @@ explains what this does and gives extra command line options.
<p>Now run the test suite:
<pre><code>python manage.py test -v 3 --traceback</code></pre>
<p>This will run the entire troggle test suite of over 70 tests (it takes only a few seconds).
<p>This will run the entire troggle test suite of ~80 tests (it takes only a few seconds).
<p>
If you get an error, and you probably will, have a look in the source code of the test, e.g. for this error:
@@ -356,9 +382,10 @@ cd ../../troggle
</font>
</details>
<p>The test suite now tidies up after itself, so there should not be any temporary files left behind or local git commits that you will need to clean up.
<p>The test suite now tidies up after itself, so there should not be any temporary files left behind or local git commits that you
will need to clean up.
<p>The test suite has over 70 tests but does not cover all of what troggle does and does not use any real data. You need to manually test these too, <em>after</em> you have done a full <a href="trogimport.html">data import</a>:
<p>The test suite has ~80 tests but does not cover all of what troggle does and does not use any real data. You need to manually test these too, <em>after</em> you have done a full <a href="trogimport.html">data import</a>:
<br>- <var><a href="http://localhost:8000/pathsreport">http://localhost:8000/pathsreport</a></var>
<br>- <var><a href="http://localhost:8000/stats">http://localhost:8000/stats</a></var>
<br>- <var><a href="http://localhost:8000/people">http://localhost:8000/people</a></var> (takes a minute or so)