expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

troggle and git installation

Browse Source
This commit is contained in:
Philip Sargent 2021-10-24 00:40:12 +03:00
parent 7ab44fec6d
commit 9e98d24bb6
4 changed files with 105 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
handbook/computing/qstart-git.html
View File

@ -2,7 +2,7 @@
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook placeholder page</title>
<title>Handbook - Git Quick</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
@ -17,6 +17,8 @@
<p>You can read or clone these repos without any control, but to write ("push") to them you will need to use <var>ssh://expo@expo.survex.com</var> and set up the <a href="keyexchange.html">key exchange</a>.</p>
<p>NOTE: always use user <var>'expo'</var> as the login user (<var>ssh://expo@...</var>) even though within git you will be identified by your own ssh key name.</p>
<p>Brendan wrote a guide to using git for expo on a Windows machine. It's worth reading: <a href="/expofiles/documents/idiots-guide-expo-git.pdf">Idiots guide to accessing expo git.pdf</a>.
<p>Open a terminal in a new directory, e.g. /tmp/experiments/ in which you want to create the repo. It will automatically create a folder with the repo name e.g.'troggle' in that directory.</p>
<dl>
@ -30,6 +32,14 @@
</dl>
<h3>Using git</h3>
<p>In Git, there are five places your source can exist: <ul>
<li>a stash,
<li>your local working directory,
<li>an index (or staging area),
<li>a local repository,
<li>and a remote repository.
</ul>
Knowing that these places even <em>exist</em> is often the first conceptual impediment.
<p><a href="https://stevebennett.me/2012/02/24/10-things-i-hate-about-git/"><img class="onright" src="git-arrows31.png" alt="git bread &amp; butter commands subset" width="250" /></a>You may find these useful:</p>
<ul>
<ul>

1
handbook/computing/winlaptop.html
View File

@ -41,6 +41,7 @@
<p>Most Windows software that we recommend "just works" if you have set up PuTTy and have done the <a href="keyexchange.html">key-pair setup</a> and are running a local ssh agent (pagent) automatically at boot up on your laptop.
<p>Some software, such as the commercial (but free) GitKraken, requires that you click a checkbox to say that you are "using local SSH agent" rather than specifying ssh private keys explicitly (File->Preferences->Authentication in GitKraken).
<p>Some software (such as Filezilla) defaults to using the local agent and it "just works".
<p>Brendan wrote a guide to using putty and git for expo on a Windows machine. It's worth reading and it has lots of screenshots: <a href="/expofiles/documents/idiots-guide-expo-git.pdf">Idiots guide to accessing expo git.pdf</a>.
<h3 id="problems">Things that cause problems</h3>

38
handbook/troggle/trogdjangup.html
View File

@ -85,18 +85,21 @@
<p>[ However django-extensions looks like it could be useful explicitly to help us through the upgrade process:
<a href="https://pypi.org/project/django-extensions/">pypi.org/project/django-extensions/</a> (only available for django 2.2 and later). ]
<h3 id="dev">Troggle software development</h3>
<p>Upgrading the version of django used by troggle is a serious programming job. It is not just a matter of editing a few config files. You will need a full troggle software development environment set up on your machine including, most definitely, the capability of running the troggle test suite. See <a href="troglaptop.html">how to set up a troggle software development laptop</a>.
<h3 id="djangotricks">Important Tricks</h3>
<p>There are six critical tricks that make everything much, much easier:
<ol>
<li>Use <a href="https://docs.python.org/3/library/venv.html">pip venv</a> or virtualenv to set up a virtual python environment within which you can easily and quickly change the specific releases of python, django, django's dependencies and django plugins.
With a previously created venv <var>py37d2</var> start it up like this:<br />
With a previously created venv <var>py37d22</var> start it up like this:<br />
<var>cd py37d22<br />
source bin/activate<br />
python --version<br />
cd troggle<br />
django-admin<br />
python manage.py<br />
python manage.py validate -v 3 --traceback<br />
python manage.py check -v 3<br />
django-admin check <br />
python manage.py test -v 3 --traceback</var>
<li>Use the <em>highest release number</em> when upgrading between minor-versions of django. <br />So we went from 1.8.19 to 1.9.13 to 1.10.8 to 1.11.29 to 2.0.13 to 2.1.15 then 2.2.19 (.20 was not released then) and next we will go to 3.2.x .
@ -119,7 +122,7 @@
<ol>
<li>ensure that you have the exact version of python installed on your machine as is live for troggle on the server, e.g. do <var>$ sudo apt install python3.7.5</var>.
<li>create a venv using the version of python to be used.
<li>do <a href="/repositories/troggle/.git/tree/README.txt">a clean install</a> of django and troggle using pip.
<li>do <a href="troglaptop.html">a clean install</a> of django and troggle.
<li>check the versions of plugins using <var>pip list -o</a>
<li>open two terminal windows:
<ul>
@ -127,33 +130,17 @@
<li>- the other one will be used for <var>python manage.py runserver 0.0.0.0:8000 -v 3</var> (or whatever portnumber you like)
</ul>
<li>rename the most recent localsettingsXXX.py to fit your local system and copy it to localsettings.py
<li>run a script to make a sanitised copy (no passwords) of localsettings.py as localsettingsXXX.py before every commit
<li>run a script <var>./pre-push.sh</var> to make a sanitised copy (no passwords) of localsettings.py as localsettingsXXX.py before every commit
<li>edit your localsettings.py to use sqlite with a file <var>troggle.sqlite</var>
<li>edit your localsettings.py to set a password for users "expo" & "expoadmin" (or just use default "nnn:gggggg")
<li>testing:
<ul>
<li>run <var>troggle$ python manage.py test</var>
<li>run the full data import <var>troggle$ python databaseReset.py reset R000</var>. It should take about 80 seconds to import everything.
<li>start the runserver in the other terminal window and open a web browser to http://localhost:8000 and check important pages which we don't have tests for (yet):
<br>- <var><a href="http://localhost:8000/pathsreport">http://localhost:8000/pathsreport</a></var>
<br>- <var><a href="http://localhost:8000/stats">http://localhost:8000/stats</a></var>
<br>- <var><a href="http://localhost:8000/people">http://localhost:8000/people</a></var>
<br>- <var><a href="http://localhost:8000/expedition/2018">http://localhost:8000/expedition/2018</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/surveyscans/2019/walletindex.html">http://localhost:8000/expofiles/surveyscans/2019/walletindex.html</a></var>
<br>- <var><a href="http://localhost:8000/survey_scans/">http://localhost:8000/survey_scans/</a></var>
<br>- <var><a href="http://localhost:8000/survexfile/caves/">http://localhost:8000/survexfile/caves/</a></var>
<br>- <var><a href="http://localhost:8000/dwgdata/">http://localhost:8000/dwgdata/</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx">http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx</a></var>
<br>- <var><a href="http://localhost:8000/1623/291/291.html">http://localhost:8000/1623/291/291.htm</a></var>
<br>- <var><a href="http://localhost:8000/caves">http://localhost:8000/caves</a></var>
<br>- <var><a href="http://localhost:8000/admin/doc/models/core.expedition/">ttp://localhost:8000/admin/doc/models/core.expedition/</a></var> (admin login required)
<br>- <var><a href="http://localhost:8000/survexfile/204">http://localhost:8000/survexfile/204</a></var>
<li>run the full data import <var>troggle$ python databaseReset.py reset R000</var>. It should take about 5 minutes to import everything (11 minutes on a slow core 2 duo laptop).
<li>start the runserver in the other terminal window and open a web browser to http://localhost:8000 and check important pages which we don't have tests for (yet). These are listed in <a href="troglaptop.html">the troggle laptop installation instuctions</a>.
</ul>
<li>Use the error dump tracebacks to find and correct the python code. [Running the server in a debugger would help: please add those instructioons for that to this page.]
<li>Use the error dump tracebacks to find and correct the python code. [Running the server in a debugger would help: please add those instructions for that to this page.]
<li>run the script <var>./pre-run.sh</var> to clean up everything before the next round of tests.
</ol>
@ -195,7 +182,8 @@ Unidecode==1.0.23</code></pre>
</code></pre>which is technically incompatible with Django 2.2 which requires 1.3.13. This incompatibility is a policy choice by the Django team. Wookey ran the Django system tests with this (April 2020) and it works fine.
<h4>To run Django's system test suite:</h4>
<p>This is only necessary if we hit a policy-imposed incomatibility, as we did with mysqlclient==1.3.10 . We have only ever had to do this once.
<p>
Follow the instructions in the "Unit tests" section installed locally at docs/internals/contributing/writing-code/unit-tests.txt, published online at <a href="https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#running-the-unit-tests">docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/</a>
This involves running <var>git clone</var> on the django source repo to download the tests. We hope we never have to do this again but in case we get incomprehensible bugs in future, we should be prepared to do so.

98
handbook/troggle/troglaptop.html
View File

@ -1,4 +1,4 @@
<!DOCTYPE html>
pythin<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
@ -21,14 +21,13 @@ your account will need to be added to the troggle project members list. Contact
<pre>
http://expo.survex.com/repositories/troggle/.git/tree/README.txt
handbook/computing/yourlaptop.html
handbook/troggle/trogdjangup.html
</pre>
<img src="https://www.debian.org/logos/openlogo-100.jpg" align="right" hspace="20" alt='debian logo'>
<h2 id="os">Installing linux</h2>
<p>If you don't already know how to do this, then you should probably not be attempting this. But in case you are an experienced linux user who has always had someone else set up the system for them, then Ubuntu is the easiest and more forgiving to install, either directly on the computer or inside WSL. Because we are using fairly old releases of Django, you will want Ubuntu-20.04
<p>If you don't already know how to do this, then you should probably not be attempting to work on the troggle code. But in case you are an experienced linux user who has always had someone else set up the system for them, then Ubuntu is the easiest and more forgiving to install, either directly on the computer or inside WSL. Because we are using fairly old releases of Django, you will want Ubuntu-20.04
<ul>
<li>Install in <a href="https://docs.microsoft.com/en-us/windows/wsl/install">WSL on Windows 10</a>.
<li>Install <a href="https://ubuntu.com/download/desktop">Ubuntu 20.04 LTS</a>.
@ -41,7 +40,7 @@ handbook/troggle/trogdjangup.html
is that WSL2 will be installed, but all our practical experience so far is with WSL1.
<h2 id="python">Installing python</h2>
<p>Python is not installed by default usually, and in any case we need specific versions to be installed. For Ubuntu 20.04 the default is python3.9 but this is incompatible with standard debian Buster, so we also need python3.7, if you are planning on migrating troggle from debian Buster (v10) to Bullseye (v11) then you will also want python3.8 .
<p>Python is not installed by default usually, and in any case we need specific versions to be installed. For Ubuntu 20.04 the default is python3.9 but this is incompatible with standard debian Buster which is what is live on our server, so we also need python3.7 . If you are planning on helping the migration of troggle from debian Buster (v10) to Bullseye (v11) then you will also want python3.8 .
<pre><code>sudo apt install python3 python3-pip
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
@ -68,15 +67,16 @@ in <var>/mnt/c/EXPO/</var>, then you would need to do this:
cd ..
sudo mkdir expo
cd expo
sudo ln -s /mnt/c/EXPO/expofiles expofiles
sudo ln -s /mnt/c/EXPO/expoweb expoweb
sudo ln -s /mnt/c/EXPO/troggle troggle
sudo ln -s /mnt/c/EXPO/loser loser
sudo ln -s /mnt/c/EXPO/drawings drawings
sudo ln -s /mnt/c/EXPO/expofiles expofiles
sudo ln -s /mnt/c/EXPO/expowebcache expowebcache
sudo mkdir expowebcache/3d
cd ..
ls -tlA expo</pre></code>
<p>If you do not have a local copy of the 40GB /expofiles/, don't worry. Later on we can set 'EXPOFILESREMOTE = True' and your test system will use the live expofiles on expo.survex.com (read only).
<h2>Installing Django and troggle</h2>
<img src="Django_Logo-420x180.png" align="right" hspace="20" width='210' alt='django logo'>
@ -90,7 +90,8 @@ ls -tlA expo</pre></code>
<h3 id="venv">Installing a venv</h3>
<p>We set up
a <var>venv</var> specifically for python 3.7 (which is the standard version on our server which is running Buster (debian v10) and <a href="https://docs.djangoproject.com/en/3.2/releases/2.2/">django 2.2.19</a>. See the <a href="https://docs.python.org/3.7/library/venv.html">standard python dcoumentation on venv</a> for python 3.7.12. Note that we are creating it as a sibling folder to /troggle/ .
a <var>venv</var> specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and <a href="https://docs.djangoproject.com/en/3.2/releases/2.2/">django 2.2.19</a>. See the <a href="https://docs.python.org/3.7/library/venv.html">standard python documentation on venv</a> for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it <var>python3.7 -m venv py37d22</var>.
Note that we are creating it as a sibling folder to /expoweb/ . Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':
<pre><code>cd ~
cd ../expo
python3.7 -m venv py37d22
@ -102,12 +103,10 @@ have installed troggle, django and dependencies. You will get a warning that you
the venv.
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' hspace="20" alt='troggle logo'/></a>
<p>The first time you do this you can't complete the pip installation of django as you have not yet got the
dependencies - because you have not yet cloned the troggle repo, so the first time it is easiest to just create requirements.txt yourself with a text editor.
<pre><code>cp ../troggle/requirements.txt .
pip install -r requirements.txt
pip list -o</pre></code>
<h3 id="venv">Installing the troggle dependencies</h3>
<p>The first time you do this on a new machine you can't complete the pip installation of django as you have not yet got the
dependencies appropriate for troggle - because you have not yet cloned the troggle repo. So the first time it is easiest to just create requirements.txt yourself with a text editor. Without using git yourself, you can get the file from the website at <a href="http://expo.survex.com/repositories/troggle/.git/tree/requirements.txt">requirements.txt</a>. If you have already cloned all the repos, then just copy it.
<pre><code>cp ../troggle/requirements.txt .</pre></code>
where <var>requirements.txt</var> (note the capitalisation of the listed packages) is:
<pre><code>confusable-homoglyphs==3.2.0
Django==2.2
@ -120,6 +119,11 @@ Unidecode==1.0.23</pre></code>
This will pick up the latest django 2.2.x available, but all the other dependencies are pinned. These
dependencies are as-standard on debian Buster (10) and you will want to experiment with upgrading them
for Bullseye (v11).
<p>Once you have the file, install the listed dependencies like this:
<pre><code>pip install -r requirements.txt
pip list -o</pre></code>
<p>Pillow is an image handling package used to make
the prospecting map (currently disabled,
@ -135,13 +139,72 @@ where it describes upgrading and testing with later versions of django.
<h3 id="troginstall">Installing troggle</h3>
<p>The <var>:troggle:</var> repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/django template code. This is over 600 files in over 400 folders, but only 38MB in size.
<dl><dt>key exchange</dt><dd>Follow this link to <a href="../computing/keyexchange.html">register a key with the expo server</a> to get git access. </dd>
<h4>key exchange</h4>
<p>Follow this link to <a href="../computing/keyexchange.html">register a key with the expo server</a> to get git access if you have not already cloned the <var>:troggle:</var> repo.
<dt>troggle</dt><dd>The <var>:troggle:</var> repo is the python code that runs troggle and generates 2,000 webpages of reports and tables. This is what you need for software development.<tt>cd /home/expo/p37d22<br>git clone ssh://expo@expo.survex.com/home/expo/troggle</tt> (read/write)</dd>
<h4>git clone troggle</h4>
<p>This creates a folder /troggle/ containing all the troggle code. </p>
<pre><code>cd ~
cd ../expo
git clone ssh://expo@expo.survex.com/home/expo/troggle</code></pre>
<p>Now we create soft links within the venv to all the repo folders and wherever you have a local copy of /expofiles/ and /expowebcache/:
<pre><code>cd py37d22
sudo ln -s /mnt/c/EXPO/expoweb expoweb
sudo ln -s /mnt/c/EXPO/troggle troggle
sudo ln -s /mnt/c/EXPO/loser loser
sudo ln -s /mnt/c/EXPO/drawings drawings
sudo ln -s /mnt/c/EXPO/expofiles expofiles
sudo ln -s /mnt/c/EXPO/expowebcache expowebcache</code></pre>
<p>If you have not been using git on this machine until now, you will need to identify yourself before you go any further.
<pre><code>git config --global user.email "you@example.com"
git config --global user.name "Your Name"</code></pre>
<h4>do the basic health checks</h4>
<p>This all checks that the installation has completed properly.
<pre><code>django-admin
python manage.py
python manage.py check -v 3 --traceback</code></pre>
<p>The first line <var>django-admin</var> will complain that it has not got a SETTINGS file, but that's fine. If it crashes though, you have not managed to install the software completely.
<p>The most important is the <var>python manage.py check</var>. If this works, then you have installed the software correctly. Now we test whether troggle is configured correctly:
<pre><code>python manage.py test -v 3 --traceback</code></pre>
<p>This will run the entire troggle test suite of over 70 tests (it takes only a few seconds). It will probably have a dozen or so failures which will probably be because it can't find the various bits of the rest of the website. So the next task is to edit the SETTINGS files to match your machine and folder structure. So find the appropriate copy of the localsettings in /_deploy/ and copy it into the main troggle folder:
<pre><code>cp _deploy/wsl/localsettingsWSL.py localsettings.py</code></pre>
<p>Now edit it and insert useful values for EXPOUSERPASS [e.g. cavey:beery], EXPOADMINUSERPASS [e.g. beery:cavey], SECRET_KEY and make sure that LIBDIR refers to the actual version of python you are using (sorry, this should be automatic).
<p>Check that FILES and EXPOFILES are set to wherever you have put /expofiles/ or set 'EXPOFILESREMOTE = False' which will use expofiles on expo.survex.com instead.
<p>Set <a href="https://docs.djangoproject.com/en/3.2/topics/email/#s-configuring-email-for-development">EMAIL_HOST and EMAIL_HOST_PASSWORD</a> to an email account you control that can send email. Then troggle can email you when some things go wrong. This may mean having to set EMAIL_PORT and MAIL_USE_TLS too (this is not used in troggle currently). Set EXPOUSER_EMAIL and EXPOADMINUSER_EMAIL to your own email address while you are doing software development. All these will be different when troggle is deployed on the public server.
<h4>check full functionality</h4>
<p>The test suite hardly skims the surface of what troggle does. You need to manually test all these too:
<br>- <var><a href="http://localhost:8000/pathsreport">http://localhost:8000/pathsreport</a></var>
<br>- <var><a href="http://localhost:8000/stats">http://localhost:8000/stats</a></var>
<br>- <var><a href="http://localhost:8000/people">http://localhost:8000/people</a></var>
<br>- <var><a href="http://localhost:8000/expedition/2018">http://localhost:8000/expedition/2018</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/surveyscans/2019/walletindex.html">http://localhost:8000/expofiles/surveyscans/2019/walletindex.html</a></var>
<br>- <var><a href="http://localhost:8000/survey_scans/">http://localhost:8000/survey_scans/</a></var>
<br>- <var><a href="http://localhost:8000/survexfile/caves/">http://localhost:8000/survexfile/caves/</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx">http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx</a></var>
<br>- <var><a href="http://localhost:8000/1623/291/291.html">http://localhost:8000/1623/291/291.htm</a></var>
<br>- <var><a href="http://localhost:8000/caves">http://localhost:8000/caves</a></var>
<br>- <var><a href="http://localhost:8000/admin/doc/models/core.expedition/">ttp://localhost:8000/admin/doc/models/core.expedition/</a></var> (admin login required)
<br>- <var><a href="http://localhost:8000/survexfile/204">http://localhost:8000/survexfile/204</a></var>
Nearly half the code deals with importing and parsing data, so you need to test that a full data import works. Run the full data import <var>troggle$ python databaseReset.py reset R000</var>. It should take about 5 minutes to import everything.
<h4>Use git to commit your edits</h4>
<p>You need to know git. Sorry, but there it is. See <a href="../computing/repos.html">our git repositories</a> and <a href="../computing/qstart-git.html">our git cheat sheet</a>.
</dl>
<hr />
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
@ -149,7 +212,6 @@ Go on to: <a href="trogarch.html">Troggle architecture</a><br />
Return to: <a href="trognotes.html">Troggle programmers' guide</a><br />
Troggle index:
<a href="trogindex.html">Index of all troggle documents</a><br />
<hr />
</body>
<hr />
</body>
</html>