mirror of
https://expo.survex.com/repositories/expoweb/.git/
synced 2024-12-18 14:32:23 +00:00
269 lines
15 KiB
HTML
269 lines
15 KiB
HTML
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<title>Handbook Troggle and Django</title>
|
|
<link rel="stylesheet" type="text/css" href="/css/main2.css" />
|
|
</head>
|
|
<body>
|
|
<style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
|
|
<h2 id="tophead">CUCC Expedition Handbook</h2>
|
|
<h1>Upgrading Django for Troggle</h1>
|
|
<h4>Why this used to be difficult</h4>
|
|
<p>Before Django 3.0, whenever Django was upgraded to a new version things broke across the entire django package, including things which we did't conciously use but which were internal dependencies within Django. These were 'the way to do it' when troggle was first written for Django 0.7 in 2006. So upgrading troggle to a new Django version required not just a broad beadth of knowledge across troggle, but also across the entire breadth of Django itself. And the error messages are sometimes very unhelpful. This is <em>no longer the case</em> and the Django organisation has a very clear "non-breaking" policy between LTS releases.
|
|
<p>Now read <a href="trogdjango.html">Django and Troggle</a> if you haven't already.
|
|
|
|
<p>Note that Django 3.2 is out of support since April 2024.
|
|
|
|
<h4>Django versions and update schedule</h4>
|
|
<div align="center">
|
|
<a href="https://www.djangoproject.com/download#supported-versions">
|
|
<img border="1" width="65%" src='django-v4.2-7.0-schedule.png'/></a></a>
|
|
<table style="border:1px solid black; background-color: #C9F0DD;font-size:130%" >
|
|
|
|
|
|
<tr>
|
|
<th>Release Series</th>
|
|
<th>Latest Release</th>
|
|
<th>End of support</th>
|
|
<th>End of security fixes</th>
|
|
</tr>
|
|
<tr>
|
|
<td>6.1</td>
|
|
<td>August 2026</td>
|
|
<td>April 2027</td>
|
|
<td>December 2027</td>
|
|
</tr>
|
|
<tr>
|
|
<td>6.0</td>
|
|
<td>December 2025</td>
|
|
<td>August 2026</td>
|
|
<td>April 2027</td>
|
|
</tr>
|
|
<tr>
|
|
<td>5.2 LTS</td>
|
|
<td>April 2025</td>
|
|
<td>December 2025</td>
|
|
<td>April 2028</td>
|
|
</tr>
|
|
<tr>
|
|
<td>5.1</td>
|
|
<td>5.1.3</td>
|
|
<td>April 2025</td>
|
|
<td>December 2025</td>
|
|
</tr>
|
|
<tr>
|
|
<td>5.0</td>
|
|
<td>5.0.9</td>
|
|
<td>August 7, 2024</td>
|
|
<td>April 2025</td>
|
|
</tr>
|
|
<tr>
|
|
<td>4.2 LTS</td>
|
|
<td>4.2.16</td>
|
|
<td>December 4, 2023</td>
|
|
<td>April 2026</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>4.1</td>
|
|
<td>4.1.13</td>
|
|
<td>April 5, 2023</td>
|
|
<td>December 1, 2023</td>
|
|
</tr>
|
|
<tr>
|
|
<td>4.0</td>
|
|
<td>4.0.10</td>
|
|
<td>August 3, 2022</td>
|
|
<td>April 1, 2023</td>
|
|
</tr>
|
|
<tr>
|
|
<td>3.2 LTS</td>
|
|
<td>3.2.8</td>
|
|
<td>December 2021</td>
|
|
<td>April 2024</td>
|
|
</tr>
|
|
<tr>
|
|
<td>3.1</td>
|
|
<td>3.1.13</td>
|
|
<td>April 2021</td>
|
|
<td>December 2021</td>
|
|
</tr>
|
|
<tr>
|
|
<td>3.0</td>
|
|
<td>3.0.14</td>
|
|
<td>August 2020</td>
|
|
<td>April 2021</td>
|
|
</tr>
|
|
<tr>
|
|
<td>2.2 LTS</td>
|
|
<td>2.2.20</td>
|
|
<td>December 2019</td>
|
|
<td>April 2022</td>
|
|
</tr>
|
|
|
|
</table>
|
|
</div>
|
|
<h4>Major, minor and releases</h4>
|
|
<p>Django release 4.2.16 is major-version 4, minor-version 2, and patch-release 16. 4.2 is the "feature release" and patch releases within each
|
|
feature release are not meant to break anything. They are just to fix bugs.
|
|
<p>Things <em>will break</em> between <a href="https://docs.djangoproject.com/en/dev/internals/release-process/">feature releases</a> which come
|
|
out every 8 months. We plan on upgrading troggle on the server whenever we upgrade the server operating system, which we do only every 2-3 years
|
|
between <a href="https://wiki.debian.org/LTS">Debian LTS releases</a>.
|
|
<p>You will come to rely extensively on <a href="https://docs.djangoproject.com/en/3.2/releases/">the release notes and versions
|
|
documentation</a> which is maintained by django.
|
|
<p>You will also need to read the django <a href="https://docs.djangoproject.com/en/3.2/howto/upgrade-version/">guide to upgrading to newer
|
|
versions</a>.
|
|
|
|
<h4>Django plugins ("apps")</h4>
|
|
<p>Documentation for the plugins is highly variable and plugin projects, being run by volunteers, can just die unexpectedly. For the django-registration plugin there are two sources of current information:<br />
|
|
<a href="https://django-registration.readthedocs.io/en/3.1/">Docs: django-registration 3.1</a><br />
|
|
<a href="https://pypi.org/project/django-registration/">PyPi: django-registration 3.1</a><br />
|
|
<p>but only one of these (PyPi) gives release history data - which is what you need if you get behind with the django upgrades.
|
|
|
|
<p>Django plugin documentation cannot be relied upon to tell you which version of django they require. They will complain when you run them if your version of django is too old though. Some experimentation is required.
|
|
|
|
<p>[ However django-extensions looks like it could be useful explicitly to help us through the upgrade process:
|
|
<a href="https://pypi.org/project/django-extensions/">pypi.org/project/django-extensions/</a> (only available for django 2.2 and later). ]
|
|
|
|
<h3 id="dev">Troggle software development</h3>
|
|
<p>Upgrading the version of django used by troggle can be a serious programming job. It is not just a matter of editing a few config files. You
|
|
will need a full troggle software development environment set up on your machine including, most definitely, the capability of running the
|
|
troggle test suite. See <a href="troglaptop.html">how to set up a troggle software development laptop</a>. Note particularly that you will find
|
|
sqlite database browser software very helpful and that you will need to know git.
|
|
|
|
<h3 id="djangotricks">Important Tricks</h3>
|
|
|
|
<p>There are six critical tricks that make everything much, much easier:
|
|
<ol>
|
|
<li>Use a virtual python environment within which you can easily and quickly change the specific releases of python, django, django's dependencies and django plugins.
|
|
With a previously created envornment <var>.venv</var> start it up like this:<br />
|
|
|
|
cd ~/expo/troggle<br />
|
|
source .venv/bin/activate<br />
|
|
python --version<br />
|
|
|
|
django-admin<br />
|
|
python manage.py<br />
|
|
python manage.py check -v 3<br />
|
|
django-admin check <br />
|
|
python manage.py test -v 3 --traceback</var>
|
|
<li>Use the <em>highest release number</em> when upgrading between minor-versions of django. <br />
|
|
So we went from 1.8.19 to 1.9.13 to 1.10.8 to 1.11.29 to 2.0.13 to 2.1.15 then 2.2.19 (variously patched at .20, .24, .25) and then 3.2.10 when
|
|
we upgraded debian 10 (buster) to 11 (bullseye).
|
|
<li>Use the django 'check' maintenance system at the most verbose setting at each release<br />
|
|
<var><font color=fuchsia>troggle$</font> python manage.py check -v 3 --deploy</var><br>
|
|
<var><font color=fuchsia>troggle$</font> python -Wall manage.py check</var><br>
|
|
<li>Use our test suite (and if you see errors, run it with <var>-v 3</var>)<br />
|
|
<var>troggle$ python manage.py test [-v 3] [--parallel auto]</var><br>
|
|
The --parallel option runs the tests in parallel on your machine which is much faster.
|
|
<li>Read all the release notes for <em>all</em> the intermediate releases. So from 1.1.29 we read <a
|
|
href="https://docs.djangoproject.com/en/3.0/releases/">14 sets of notes</a>: for <a
|
|
href="https://docs.djangoproject.com/en/3.0/releases/2.0/">2.0</a>, 2.0.1, 2.0.2... up to 2.0.13 .
|
|
<li>Seriously learn how to use the traceback webpage produced by django when a page crashes. There is a full record of every variable value
|
|
hidden in there.
|
|
<li>You will be doing a lot of local testing just on your development machine. The default is to use the webserver built into Django:
|
|
<var>python manage.py runserver 0.0.0.0:8000 -v 3</var>. Alternatively you can install and use gunicorn:
|
|
<var><a href="https://docs.gunicorn.org/en/stable/design.html">gunicorn</a> --reload -w 9 -b :8000 wsgi</var>. the '-w' flag is for the number of worker threads and should be 2n+1 where n is the number of cpu cores on your machine.
|
|
</ol>
|
|
|
|
<h3>Deprecation warnings and python versions</h3>
|
|
<p>The individual releases within a minor version don't break anything but do fix bugs. So if you are on 4.1.x there is no point
|
|
in getting 4.2.1 to work and you should go straight to the latest available on your machine of the 4.2 feature release, e.g. 4.2.5 on debian 13.
|
|
<p><var>check --deploy</var> gives django warnings about security issues in your settings as well as django deprecation warnings.<br>
|
|
<var>-Wall</var> is a standard python option and gives warnings of deprecated python features used by django and all
|
|
the current plugins. So it tells us that django 1.11.29 is using a deprecated python language feature which will be
|
|
removed from the language in python 3.9 . Python version compatibilities are documented at the top of each y.x.0
|
|
release note. <a href="https://docs.djangoproject.com/en/4.2/releases/4.2/">From Django 4.2</a> it requires python
|
|
3.8, but python 3.11 is recommended.
|
|
|
|
<h3>The upgrade process</h3>
|
|
<h4>Django migrations - prior reading</h4>
|
|
<p>Even if you have made no changes to the data model, and thus the database scheme is unchanged, you still have to run the Django migration software between different Django Feature Releases, e.g. between 3.1.x and 3.2.y .
|
|
<p>Even if you have not made any changes, Django itself may have made substantial changes to its admin data model and so the database structure needs to be migrated to the new structure - even though it is an empty database since we recreate it all every time we do a data import.
|
|
<ul>
|
|
<li><a href="https://docs.djangoproject.com/en/3.2/topics/migrations/">Django migrations documentation</a>. Read it all even if you don't understand it all yet. You will come back to it.
|
|
<li><a href="https://realpython.com/django-migrations-a-primer/">Django Migrations: A Primer</a>. A training course in migrations. Really useful.
|
|
<li><a href="https://docs.djangoproject.com/en/3.2/ref/django-admin/#django-admin-makemigrations">command line documentation</a> for the 'makemigrations' and 'migrate' commands.
|
|
</ul>
|
|
<img class="onright" width = "80px" src="https://sqlite.org/images/sqlite370_banner.gif">
|
|
<p>Now might also be a good idea to get familiar with using a database browser (see <a href="troglaptop.html#dbtools">configuring a troggle laptop</a>) and use it with the example project described in the Primer article: which you should run through on your own machine.
|
|
|
|
<h4>Running troggle before and after a Django migration</h4>
|
|
<ol>
|
|
<li>ensure that you have the exact same version of python installed on your machine as is live for troggle on the server, so check the line <br />
|
|
<code>requires-python = ">=3.11,<3.12"</code><br />
|
|
in the file <var>server.toml</var>.
|
|
<li>Copy <var>server.toml</var> on top of <var>pyprojectr.toml</var> and synchronise the environment: <var>uv sync</var>.
|
|
|
|
<li>check the versions of plugins using <var>uv tree</var>
|
|
<pre><code><font color=fuchsia>(troggle) philip@Mohawk:~/expo/troggle$</font> uv tree --all-groups
|
|
Resolved 11 packages in 0.54ms
|
|
troggle-server v2024.12.1
|
|
├── beautifulsoup4 v4.11.2 (group: server)
|
|
│ └── soupsieve v2.6
|
|
├── coverage v7.6.9 (group: server)
|
|
├── django v3.2.19 (group: server)
|
|
│ ├── asgiref v3.8.1
|
|
│ ├── pytz v2024.2
|
|
│ └── sqlparse v0.5.3
|
|
├── piexif v1.1.3 (group: server)
|
|
├── pillow v9.4.0 (group: server)
|
|
└── unidecode v1.3.6 (group: server)</code></pre>
|
|
<li>open two terminal windows:
|
|
<ul>
|
|
<li>- one will be used for installations and tests.
|
|
<li>- the other one will be used for <var>python manage.py runserver 0.0.0.0:8000 -v 3</var> (or whatever portnumber you like)
|
|
</ul>
|
|
|
|
<li>run a script <var>./pre-push.sh</var> to make a sanitised copy (no passwords) of localsettings.py as localsettingsXXX.py before every commit
|
|
<li>edit your localsettings.py to use sqlite with a file <var>troggle.sqlite</var>
|
|
<li>edit your localsettings.py to set a password for users "expo" & "expoadmin" (or just use default "nnn:gggggg")
|
|
<li>testing:
|
|
<ul>
|
|
<li>run <var>troggle$ python manage.py test</var>
|
|
<li>run the full data import <var>troggle$ python databaseReset.py reset R000</var>. It should take between 40s and 1 minute or so to import everything (11 minutes on an old slow core 2 duo laptop). ("R000" is just a label, use whatever you like.)
|
|
<li>start the runserver in the other terminal window and open a web browser to http://localhost:8000 and check important pages which we don't have tests for (yet). These are listed in <a href="troglaptop.html">the troggle laptop installation instuctions</a>.
|
|
|
|
</ul>
|
|
<li>Use the error dump tracebacks to find and correct the python code. [Running the server in a debugger would help: please add those instructions for that to this page.]
|
|
<li>run the script <var>./pre-run.sh</var> to clean up everything before the next round of tests.
|
|
|
|
</ol>
|
|
|
|
|
|
<h3>Django database dependencies</h3>
|
|
<p>On the expo server we run MariaDB as the database which has its own dependencies. In particular under debian buster we
|
|
had to use MariaDB v1.3.10 which was the latest version supported by Debian buster:
|
|
|
|
<pre><code>mysqlclient==1.3.10
|
|
</code></pre>which is technically incompatible with Django 2.2 which requires 1.3.13.
|
|
This incompatibility is a policy choice by the Django team. Wookey ran the Django system tests with 1.3.10 (April 2020) and it worked fine.
|
|
Django just didn't support it because they couldn't spare the effort to test it.
|
|
|
|
<h4>To run Django's system test suite:</h4>
|
|
<p>This is only necessary if we hit a policy-imposed incompatibility, as we did with mysqlclient==1.3.10 . We have only ever had to do this once.
|
|
<p>
|
|
Follow the instructions in the "Unit tests" section installed locally at docs/internals/contributing/writing-code/unit-tests.txt, published online at <a href="https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#running-the-unit-tests">docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/</a>
|
|
This involves running <var>git clone</var> on the django source repo to download the tests. We hope we never have to do this again but in case we get incomprehensible bugs in future, we should be prepared to do so.
|
|
|
|
<p>Current upgrade status is documented: <a href="trogstatus.html">here</a>.
|
|
<hr />
|
|
|
|
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
|
|
Return to: <a href="trogdjango.html">Django and Troggle</a><br />
|
|
Return to: <a href="trogintro.html">Troggle intro</a><br />
|
|
<!-- absolute addressing so that this fragment can be copied eveywhere and still work -->
|
|
|
|
|
|
Return to: <a href="/handbook/troggle/trognotes.html">Troggle programmers' guide</a><br />
|
|
|
|
Go to: ⚒ <a href="/handbook/troggle/troglaptop.html">Troggle development laptop</a>
|
|
☯ <a href="/handbook/computing/wsllaptop.html">Troggle development WSL laptop</a><br />
|
|
<br />
|
|
Troggle index:
|
|
<a href="/handbook/troggle/trogindex.html">Index of all troggle documents</a><br />
|
|
|
|
<hr /></body>
|
|
</html>
|