mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2024-11-29 21:32:00 +00:00
140 lines
12 KiB
HTML
140 lines
12 KiB
HTML
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<title>Handbook Troggle and Django</title>
|
|
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
|
|
</head>
|
|
<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
|
|
<h2 id="tophead">CUCC Expedition Handbook</h2>
|
|
<h1>Upgrading Django for Troggle</h1>
|
|
<h3>Django Versions and troggle</h3>
|
|
<p>Troggle is built on top of the django framework. In 2019 we had let ourselves get rather behind in keeping up to date with the django version and this held us back from upgrading the operating system - which we needed to do. Troggle uses several community-contributed django plugins and each of these had their own version dependencies. We also got caught up with out of date and deprecated python language features.
|
|
<p>A ridiculous amount of work was required to get to a situation where troggle was running on relatively recent and nearly security-in-date versions of python, django, plugins and linux by Summer 2020. We don't want that to happen again.
|
|
|
|
<h4>Why this is difficult</h4>
|
|
<p>When django upgrades to a new version things break across the entire django package, including things which we don't conciously use but are internal dependencies within django. These were 'the way to do it' when troggle was first written for django 0.7 in 2006. So upgrading troggle to a new django version requires not just a broad beadth of knowledge across troggle, but also across the entire breadth of django itself. And the error messages are sometimes very unhelpful.
|
|
|
|
<h4>Major, minor and releases</h4>
|
|
<p>Django release 1.11.29 is major-version 1, minor-version 11, and patch-release 29. 1.11 is the "feature release".
|
|
<p>Things <em>will break</em> between <a href="https://docs.djangoproject.com/en/3.0/internals/release-process/">django minor-versions</a> (feature releases) which come out every 8 months.
|
|
|
|
<h4>Django versions and deprecation</h4>
|
|
<a href="https://www.djangoproject.com/download#supported-versions">
|
|
<img border="1" class="onright" width="250px" src='release-roadmap.3c7ece4f31b3.png'/></a></a>
|
|
<ul>
|
|
<li>Django has a mature and reliable release programme: <a href="http://docs.djangoproject.com/en/3.0/releases/">https://docs.djangoproject.com/en/3.0/releases/</a>
|
|
<li>Django pre-announces which features will be deprecated: <a href="https://docs.djangoproject.com/en/3.0/internals/deprecation/">docs.djangoproject.com/en/internals/deprecation/</a>
|
|
<li>Django release schedule: <a href="https://www.djangoproject.com/download/">supported versions timetable</a>
|
|
</ul>
|
|
|
|
<h4>Django plugins</h4>
|
|
<p>But the contributed plugins have no such management process and releases are rather random, e.g. for django-registration there are two sources of current information:<br />
|
|
<a href="https://django-registration.readthedocs.io/en/3.1/">Docs: django-registration 3.1</a><br />
|
|
<a href="https://pypi.org/project/django-registration/">PyPi: django-registration 3.1</a><br />
|
|
<p>but only one of these (PyPi) gives release history data - which is what you need if you get behind with the django upgrades.
|
|
|
|
<p>Django plugin documentation cannot be relied upon to tell you which version of django they require. They will complain when you run them if your version of django is too old though. Some experimentation is required.
|
|
|
|
<p>Every extra plugin increases the "vulnerability surface" of troggle with respect to django upgrade problems so only install a new plugin if it is really, really necessary. For example, when django-staticfiles broke during one upgrade we discovered that we could use flatfiles as a workaround, so we are not planning on reinstalling staticfiles ever again.
|
|
|
|
<p>[ However django-extensions looks like it could be useful explicitly to help us through the upgrade process:
|
|
<a href="https://pypi.org/project/django-extensions/">pypi.org/project/django-extensions/</a> (only available for django 2.2 and later). ]
|
|
|
|
<h4>Why we still use django</h4>
|
|
<p>Well we <a href="trog2030.html">might not use django indefinitely</a>, but unlike many frameworks the necessary functions are separately replaceable. So this gives us an evolution path.
|
|
<p>The stack is: database, request/response (GET/POST http),
|
|
<a href="https://docs.djangoproject.com/en/1.11/topics/http/urls/">URL mapping/dispatch</a> (declarative and bidrectional), and <a href="https://docs.djangoproject.com/en/3.0/topics/templates/">templates</a> (data to HTML). See the <a href="https://docs.djangoproject.com/en/3.0/misc/design-philosophies/">django design philosophy</a>. Note that having a URL dispatcher that supports <a href="https://docs.djangoproject.com/en/1.11/topics/http/urls#s-reverse-resolution-of-urls">reverse resolution</a> is important: it is a key reason why we need a framework and not just a mess of javascript. We have nearly 100 URL patterns.
|
|
|
|
<h3>Important Tricks</h3>
|
|
<p>There are five critical tricks that make everything much, much easier:
|
|
<ol>
|
|
<li>Use <a href="https://docs.python.org/3/library/venv.html">pip venv</a> to set up a virtual python environment within which you can easily and quickly change the specific releases of python, django, django's dependencies and django plugins.
|
|
With a previously created venv <var>t37</var> start it up like this:<br />
|
|
<var>cd t37<br />
|
|
source bin/activate<br />
|
|
python --version<br />
|
|
cd troggle<br />
|
|
python manage.py test -v 3</var>
|
|
<li>Use the <em>highest release number</em> when upgrading between minor-versions of django. <br />So we went from 1.8.19 to 1.9.13 to 1.10.8 to 1.11.29 . Next will be 2.0.13, then 2.1.15 and then 2.2.15 .
|
|
<li>Use the django 'check' maintenance system at the most verbose setting at each release<br />
|
|
<var>troggle$ python manage.py check -v 3 --deploy</var><br>
|
|
<var>troggle$ python -Wall manage.py check</var><br>
|
|
<li>Use our test suite (and if you see errors, run it with <var>-v 3</var>)<br />
|
|
<var>troggle$ python manage.py test [-v 3]</var><br>
|
|
<li>Read all the release notes for <em>all</em> the intermediate releases. So from 1.1.29 we will read <a href="https://docs.djangoproject.com/en/3.0/releases/">14 sets of notes</a>: for <a href="https://docs.djangoproject.com/en/3.0/releases/2.0/">2.0</a>, 2.0.1, 2.0.2... up to 2.0.13 .
|
|
</ol>
|
|
<p>The individual releases within a minor version don't break anything but do fix bugs. So if you are on 1.10.x there is no point in getting 1.11.1 to work and you should go straight to 1.11.29 (29 is the highest release number in minor version 1.11).
|
|
<p><var>--deploy</var> gives django warnings about security issues in your settings as well as django deprecation warnings.<br>
|
|
<var>-Wall</var> is a standard python option and gives warnings of deprecated python features used by django and all the current plugins. So it tells us that django 1.11.29 is using a deprecated python language feature which will be removed from the language in python 3.9 (in fact 1.11.29 is incompatible with python 3.8 too but we didn't get a warning about that).
|
|
|
|
<h3>The upgrade process</h3>
|
|
<ol>
|
|
<li>ensure that you have the exact version of python installed on your machine as is live for troggle on the server, e.g. do <var>$ sudo apt install python3.7.5</var>.
|
|
<li>create a venv using the version of python to be used.
|
|
<li>do <a href="/repositories/troggle/.git/tree/README.txt">a clean install</a> of django and troggle using pip.
|
|
<li>check the versions of plugins using <var>pip list -o</a>
|
|
<li>open two terminal windows:
|
|
<br>- one will be used for installations and tests.
|
|
<br>- the other one will be used for <var>python manage.py runserver 0.0.0.0:8000 -v 3</var> (or whatever portnumber you like)
|
|
<li>rename the most recent localsettingsXXX.py to fit your local system and make it symlink to localsettings.py
|
|
<li>edit your localsettingsXXX.py to use sqlite with a file <var>troggle.sqlite</var>
|
|
<li>edit your localsettingsXXX.py to set a password for user "expo" (or just use default "nnn:gggggg")
|
|
<li>testing:
|
|
<ul>
|
|
<li>run <var>troggle$ python manage.py test</var>
|
|
<li>run the full data import <var>troggle$ python databaseReset.py reset R000</var>. It should take about 80 seconds to import everything.
|
|
<li>start the runserver in the other terminal window and open a web browser to http://localhost:8000 and check important pages which we don't have tests for (yet):
|
|
<br>- <var><a href="http://localhost:8000/pathsreport">http://localhost:8000/pathsreport</a></var>
|
|
<br>- <var><a href="http://localhost:8000/stats">http://localhost:8000/stats</a></var>
|
|
<br>- <var><a href="http://localhost:8000/people">http://localhost:8000/people</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expedition/2018">http://localhost:8000/expedition/2018</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/surveyscans/2019/walletindex.html">http://localhost:8000/expofiles/surveyscans/2019/walletindex.html</a></var>
|
|
<br>- <var><a href="http://localhost:8000/survey_scans/">http://localhost:8000/survey_scans/</a></var>
|
|
<br>- <var><a href="http://localhost:8000/survexfile/caves/">http://localhost:8000/survexfile/caves/</a></var>
|
|
<br>- <var><a href="http://localhost:8000/tunneldata/">http://localhost:8000/tunneldata/</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg</a></var>
|
|
<br>- <var><a href="http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx">http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx</a></var>
|
|
<br>- <var><a href="http://localhost:8000/1623/291/291.html">http://localhost:8000/1623/291/291.htm</a></var>
|
|
<br>- <var><a href="http://localhost:8000/caves">http://localhost:8000/caves</a></var>
|
|
<br>- <var><a href="http://localhost:8000/admin/doc/models/core.expedition/">ttp://localhost:8000/admin/doc/models/core.expedition/</a></var> (admin login required)
|
|
<br>- <var><a href="http://localhost:8000/survexfile/204">http://localhost:8000/survexfile/204</a></var>
|
|
</ul>
|
|
<li>Use the error dump tracebacks to find and correct the python code. [Running the server in a debugger would help: please add those instructioons for that to this page.]
|
|
<li>run the script <var>./pre-run.sh</var> to clean up everything before the next round of tests.
|
|
|
|
</ol>
|
|
<p>You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right.
|
|
|
|
<p>Here is example of installed (Version) and most-recent-available (Latest) verisons of pip installations.
|
|
<pre><code>$ pip list -o
|
|
Package Version Latest Type
|
|
------------------- ------- ------ -----
|
|
Django 1.11.29 3.0.8 wheel
|
|
django-registration 2.5.2 3.1 wheel
|
|
Pillow 7.1.2 7.2.0 wheel
|
|
setuptools 45.2.0 49.2.0 wheel</code></pre>
|
|
<p>This only gives any output for packages which are not the most recent. To see what is actually installed use
|
|
<pre><code>$ pip freeze
|
|
confusable-homoglyphs==3.2.0
|
|
Django==1.11.29
|
|
django-registration==2.5.2
|
|
docutils==0.16
|
|
Pillow==7.2.0
|
|
pytz==2020.1
|
|
Unidecode==1.1.1</code></pre>
|
|
<p>To upgrade pip istelf, do <var>$ pip install --upgrade pip</var>
|
|
<p>We will not need to update the django-registration until we move from django 2.2.15+ to 3.0.9+ which <a href="trog2030.html">we may never do</a>.
|
|
|
|
<p>Current upgrade status is documented: <a href="trogstatus.html">here</a>.
|
|
<hr />
|
|
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
|
|
Return to: <a href="trogintro.html">Troggle intro</a><br />
|
|
Troggle index:
|
|
<a href="trogindex.html">Index of all troggle documents</a><br />
|
|
<hr />
|
|
</body>
|
|
</html>
|