expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Django upgrade and troggle docm.

Browse Source
This commit is contained in:
Philip Sargent 2020-07-29 18:02:33 +01:00
parent 33c271625f
commit bdbb171762
10 changed files with 157 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
TODO
View File

@ -1,2 +0,0 @@
everything here moved to handbook/computing/x-todo.html
Philip Sargent 2020/03/28

BIN
handbook/computing/ioc-timeline.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
handbook/computing/spag.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

4
handbook/troggle/placeholder.html
View File

@ -8,12 +8,16 @@
<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Handbook Troggle XXX</h1>
<a href="https://www.goodreads.com/book/show/1155986.The_Tricky_Troggle">
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
<p>This is not the page you are looking for.
<p>This will be replaced with the information you want as soon as someone gets around to writing it. Why not find out how to do this yourself ?
<hr />
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
Return to: <a href="trogintro.html">Troggle intro</a><br />
Troggle index:
<a href="trogindex.html">Index of all troggle documents</a><br />
<hr />
</body>
</html>

BIN
handbook/troggle/release-roadmap.3c7ece4f31b3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

10
handbook/troggle/trogdesign.html
View File

@ -9,17 +9,23 @@
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Troggle Design Decisions</h1>
<p>Open issues being worked on:
<p>Specific, immediate problems:
<ul>
<li>New systems for <a href="menudesign.html">website menus</a>
<li>Short-term note on "logon" <a href="trogregistr.html">django-registration</a>
</ul>
<p>Open architectural issues being worked on:
<ul>
<li>Critiqued proposal on <a href="trogsimpler.html">future architecture</a>
<li>A possible <a href="trog2030.html">5 year plan</a>
<li>Speculations on <a href="trogspeculate.html">future architectures</a>
<li>Short-term note on "logon" <a href="trogregistr.html">django-registration</a>
</ul>
<hr />
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
Return to: <a href="trogintro.html">Troggle intro</a><br />
Troggle index:
<a href="trogindex.html">Index of all troggle documents</a><br />
<hr />
</body>
</html>

125
handbook/troggle/trogdjango.html Normal file
View File

@ -0,0 +1,125 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Handbook Troggle and Django</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
<h2 id="tophead">CUCC Expedition Handbook</h2>
<h1>Upgrading Django for Troggle</h1>
<h3>Django Versions and troggle</h3>
<p>Troggle is built on top of the django framework. In 2019 we had let ourselves get rather behind in keeping up to date with the django version and this held us back from upgrading the operating system - which we needed to do. Troggle uses several community-contributed django plugins and each of these had their own version dependencies. We also got caught up with out of date and deprecated python language features.
<p>A ridiculous amount of work was required to get to a situation where troggle was running on relatively recent and nearly security-in-date versions of python, django, plugins and linux by Summer 2020. We don't want that to happen again.
<h4>Why this is difficult</h4>
<p>When django upgrades to a new version things break across the entire django package, including things which we don't conciously use but are internal dependencies within django. These were 'the way to do it' when troggle was first written for django 0.7 in 2006. So upgrading troggle to a new django version requires not just a broad beadth of knowledge across troggle, but also across the entire breadth of django itself. And the error messages are sometimes very unhelpful.
<h4>Major, minor and releases</h4>
<p>Django release 1.11.29 is major-version 1, minor-version 11, and patch-release 29. 1.11 is the "feature release".
<p>Things <em>will break</em> between <a href="https://docs.djangoproject.com/en/3.0/internals/release-process/">django minor-versions</a> which come out every 8 months.
<h4>Django versions and deprecation</h4>
<a href="https://www.djangoproject.com/download#supported-versions">
<img border="1" class="onright" width="250px" src='release-roadmap.3c7ece4f31b3.png'/></a></a>
<ul>
<li>Django has a mature and reliable release programme: <a href="http://docs.djangoproject.com/en/3.0/releases/">https://docs.djangoproject.com/en/3.0/releases/</a>
<li>Django pre-announces which features will be deprecated: <a href="https://docs.djangoproject.com/en/3.0/internals/deprecation/">docs.djangoproject.com/en/internals/deprecation/</a>
<li>Django release schedule: <a href="https://www.djangoproject.com/download/">supported versions timetable</a>
</ul>
<h4>Django plugins</h4>
<p>But the contributed plugins have no such management process and releases are rather random, e.g. for django-registration there are two sources of current information:<br />
<a href="https://django-registration.readthedocs.io/en/3.1/">Docs: django-registration 3.1</a><br />
<a href="https://pypi.org/project/django-registration/">PyPi: django-registration 3.1</a><br />
<p>but only one of these (PyPi) gives release history data - which is what you need if you get behind with the django upgrades.
<p>Django plugin documentation cannot be relied upton to tell you which version of django they require. They will complain when you run them if your version of django is too old though. Some experimentation is required.
<p>Every extra plugin increases the "vulnerability surface" of troggle with respect to django upgrade problems so only install a new plugin if it is really, really necessary. For example, when django-staticfiles broke during one upgrade we discovered that we could use flatfiles as a workaround, so we are not planning on reinstalling staticfiles ever again.
<p>[ However django-extensions looks like it could be useful explicitly to help us through the upgrade process:
<a href="https://pypi.org/project/django-extensions/">pypi.org/project/django-extensions/</a> (only available for django 2.2 and later). ]
<h3>Important Tricks</h3>
<p>There are five critical tricks that make everything much, much easier:
<ol>
<li>Use pip venv to set up a virtual python environment within which you can easily and quickly change the specific releases of python, django, django's dependencies and django plugins.<li>Use the <em>highest release number</em> when upgrading between minor-versions of django. <br />So we went from 1.8.19 to 1.9.13 to 1.10.8 to 1.11.29 . Next will be 2.0.13, then 2.1.15 and then 2.2.15 .
<li>Use the django 'check' maintenance system at the most verbose setting at each release<br />
<var>troggle$ python manage.py check -v 3 --deploy</var><br>
<var>troggle$ python -Wall manage.py check</var><br>
<li>Use our test suite (and if you see errors, run it with <var>-v 3</var>)<br />
<var>troggle$ python manage.py test [-v 3]</var><br>
<li>Read all the release notes for <em>all</em> the intermediate releases. So from 1.1.29 we will read <a href="https://docs.djangoproject.com/en/3.0/releases/">14 sets of notes</a>: for <a href="https://docs.djangoproject.com/en/3.0/releases/2.0/">2.0</a>, 2.0.1, 2.0.2... up to 2.0.13 .
</ol>
<p>The individual releases within a minor version don't break anything but do fix bugs. So if you are on 1.10.x there is no point in getting 1.11.1 to work and you should go straight to 1.11.29 (29 is the highest release number in minor version 1.11).
<p><var>--deploy</var> gives django warnings about security issues in your settings as well as django deprecation warnings.<br>
<var>-Wall</var> is a standard python option and gives warnings of deprecated python features used by django and all the current plugins. So it tells us that django 1.11.29 is using a deprecated python language feature which will be removed from the language in python 3.9 (in fact 1.11.29 is incompatible with python 3.8 too but we didn't get a warning about that).
<h3>The upgrade process</h3>
<ol>
<li>ensure that you have the exact version of python installed on your machine as is live for troggle on the server, e.g. do <var>$ sudo apt install python3.7.5</var>.
<li>create a venv using the version of python to be used.
<li>do <a href="/repositories/troggle/.git/tree/README.txt">a clean install</a> of django and troggle using pip.
<li>check the versions of plugins using <var>pip list -o</a>
<li>open two terminal windows:
<br>- one will be used for installations and tests.
<br>- the other one will be used for <var>python manage.py runserver 0.0.0.0:8000 -v 3</var> (or whatever portnumber you like)
<li>rename the most recent localsettingsXXX.py to fit your local system and make it symlink to localsettings.py
<li>edit your localsettingsXXX.py to use sqlite with a file <var>troggle.sqlite</var>
<li>edit your localsettingsXXX.py to set a password for user "expo" (or just use default "nnn:gggggg")
<li>testing:
<ul>
<li>run <var>troggle$ python manage.py test</var>
<li>run the full data import <var>troggle$ python databaseReset reset R000</var>. It should take about 80 seconds to import everything.
<li>start the runserver in the other terminal window and open a web browser to http://localhost:8000 and check important pages which we don't have tests for (yet):
<br>- <var><a href="http://localhost:8000/pathsreport">http://localhost:8000/pathsreport</a></var>
<br>- <var><a href="http://localhost:8000/stats">http://localhost:8000/stats</a></var>
<br>- <var><a href="http://localhost:8000/people">http://localhost:8000/people</a></var>
<br>- <var><a href="http://localhost:8000/expedition/2018">http://localhost:8000/expedition/2018</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/surveyscans/2019/walletindex.html">http://localhost:8000/expofiles/surveyscans/2019/walletindex.html</a></var>
<br>- <var><a href="http://localhost:8000/survey_scans/">http://localhost:8000/survey_scans/</a></var>
<br>- <var><a href="http://localhost:8000/survexfile/caves/">http://localhost:8000/survexfile/caves/</a></var>
<br>- <var><a href="http://localhost:8000/tunneldata/">http://localhost:8000/tunneldata/</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.pdf</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo_2016.svg</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg">http://localhost:8000/expofiles/rigging_topos/264/entrance_topo1_2016.jpeg</a></var>
<br>- <var><a href="http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx">http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx</a></var>
<br>- <var><a href="http://localhost:8000/1623/291/291.html">http://localhost:8000/1623/291/291.htm</a></var>
<br>- <var><a href="http://localhost:8000/caves">http://localhost:8000/caves</a></var>
<br>- <var><a href="http://localhost:8000/admin/doc/models/core.expedition/">ttp://localhost:8000/admin/doc/models/core.expedition/</a></var> (admin login required)
<br>- <var><a href="http://localhost:8000/survexfile/204">http://localhost:8000/survexfile/204</a></var>
</ul>
<li>Use the error dump tracebacks to find and correct the python code. [Running the server in a debugger would help: please add those instructioons for that to this page.]
<li>run the script <var>./pre-run.sh</var> to clean up everything before the next round of tests.
</ol>
<p>You can upgrade the version of python installed within pip but not downgrade. So get that first venv installed right.
<p>Here is example of installed (Version) and most-recent-available (Latest) verisons of pip installations.
<pre><code>$ pip list -o
Package Version Latest Type
------------------- ------- ------ -----
Django 1.11.29 3.0.8 wheel
django-registration 2.5.2 3.1 wheel
Pillow 7.1.2 7.2.0 wheel
setuptools 45.2.0 49.2.0 wheel</code></pre>
<p>This only gives any output for packages which are not the most recent. To see what is actually installed use
<pre><code>$ pip freeze
confusable-homoglyphs==3.2.0
Django==1.11.29
django-registration==2.5.2
docutils==0.16
Pillow==7.2.0
pytz==2020.1
Unidecode==1.1.1</code></pre>
<p>To upgrade pip istelf, do <var>$ pip install --upgrade pip</var>
<p>We will not need to update the django-registration until we move from django 2.2.15+ to 3.0.9+ which <a href="trog2030.html">we may never do</a>.
<hr />
Go on to: <a href="trogarch.html">Troggle architecture</a><br />
Return to: <a href="trogintro.html">Troggle intro</a><br />
Troggle index:
<a href="trogindex.html">Index of all troggle documents</a><br />
<hr />
</body>
</html>

11
handbook/troggle/trogindex.html
View File

@ -12,7 +12,7 @@
<p>The number of different webpages describing troggle, configuring the server troggle runs on, maintaining the underlying Django system, current documentation, future design issues and historical decisions is all getting a bit voluminous. So we need a complete index:
<p><img border="1" class="onright" width="150px" src='tricky-troggle.jpg' alt='git logo'/></a>
<a href="trogintro.html">Troggle - Introduction</a> - what it is<br>
<a href="trogusers.html">Troggle - the users</a> - Who needs to know What and When<br>
<a href="trogusers.html">Troggle - the Users</a> - Who needs to know What and When<br>
<a href="trogdocm.html">Troggle - Fixing things</a> - users' manuals for data import<br>
<a href="trogimport.html">Troggle - Data Import</a> - reset and import data<br>
<a href="trogmanual.html">Troggle - Maintenance</a> - list of maintenance tasks<br>
@ -20,18 +20,19 @@
<a href="datamodel.html">Troggle - Data Model</a> - syntax-coloured list of classes, instance variables and foreign keys<br>
<a href="trogdesign.html">Troggle - Design Decisions for the Future</a> - open issues being worked on:<br>
<ul>
<li><a href="trogregistr.html">Troggle Login and user registration</a> - proposal to remove registration<br>
<li><a href="menudesign.html">Troggle Menu Design</a> - options for replacing the menuing system<br><br>
<li><a href="trogsimpler.html">Troggle - a kinder simpler troggle?</a> - Radost's proposals (critiqued)<br>
<li><a href="trogspeculate.html">Troggle Architecture Speculations</a> - as in April 2020<br>
<li><a href="trog2030.html">Troggle in 2025-2030</a> - architectural evolution proposal<br>
<li><a href="trogregistr.html">Troggle Login and user registration</a> - proposal to remove registration<br>
<li><a href="menudesign.html">Troggle Menu Design</a> - options for replacing the menuing system<br>
</ul>
<a href="serverconfig.html">Troggle server configuration</a> - how to get troggle running on a new machine<br>
<a href="trogdjango.html">Troggle and django</a> - Upgrading to later django versions<br>
<a href="unittests.html">Troggle unit tests</a> - test suite for programmers<br>
<a href="trogstatus.html">Troggle & Expo Systems - status update</a> - where we are now<br>
<a href="trogstatus.html">Troggle & expo systems - status update</a> - where we are now<br>
<br>
<a href="trognotes.html">Troggle Notes</a> - the beginnings of a manual<br>
<a href="trognotes.html">Troggle Notes</a> - manual notes - (to be moved/edited)<br>
<a href="trogassumptions.html">Pre-Troggle Assumptions</a> - list of assumptions<br>
<a href="archnotes.html">Archive Notes</a> - old ideas and original discussions<br>
<br>

2
handbook/troggle/trogmanual.html
View File

@ -21,6 +21,8 @@
<li><a href="trogarch.html">Troggle and database</a> architecture: how it all fits together
<li><a href="trogdesign.html">Design decisions</a> inc. troggle replacement e.g. new systems for <a href="menudesign.html">website menus</a>
<li><a href="trogdocm.html">Maintain troggle</a> itself. The code is public on repository <a href="../computing/repos.html">:troggle:</a>
<li><a href="trogdjango.html">Troggle and django</a> - Upgrading to later django versions
<li><a href="trognotes.html">Uncategorised notes</a> and past speculations
</ul>
<p>Troggle is completely unlike any other django installation: it has a database, but the database is rebuilt from files every time it starts.

13
handbook/troggle/trogspeculate.html
View File

@ -95,7 +95,7 @@ could mean that we could gain.
<p>Writing our own multi-user code would not be sensible, hence the database.
But we could move to a read-only system where the only writing happens on data-import.
Then we could use python 'pickle()' or 'json()' read-only data structures, but we
would need to create all our own indexing and cross-referencing code.
would need to create all our own indexing and cross-referencing code (which is <a href="#mud">a much bigger job</a><sup>*</sup> than you might think).
<p>There would be more lower-level code, but the
different segments of the system could be in caving-sensible modules not
django-meaningful modules. And we would not have all the extra
@ -114,6 +114,17 @@ which is a port to JavaScript of the Django templating system we use
Nunjucks including <a href="https://service-manual.nhs.uk/design-system/prototyping">
the NHS digital service</a> and Firefox.
<h3 =id="mud">* Later Note on object dependencies</h3>
<!-- Philip Sargent 29 July 2020 -->
<a href="http://picocontainer.com/inversion-of-control-history.html#timelines">
<img class="onright" src ="../computing/ioc-timeline.png" width="200px"></a>
<p>Currently every troggle code operation uses the django ORM <var>search</var> and <var>filter</var> operations on the central database to find any object it needs. If we don't have a central database then we have to use direct object references and we need to think about the design of <a href="https://medium.com/@geoffreykoh/implementing-the-factory-pattern-via-dynamic-registry-and-python-decorators-479fc1537bbe">a central registry object</a> to hold these. There is a well-studied design pattern that describes this design "<a href="http://www.laputan.org/mud/">Big Ball of Mud</a>" which and the contributing actions "Piecemeal growth" and "Sweeping it under the rug".
<p>We are always using one object, e.g. a wallet, just to get at another object, e.g. a scan of some original notes, in order to check the data we are checking, e.g. a survex file. Maintaining two-way dependencies amoung all the objects is what "foreign keys" do in a database, but the problem doesn't go away when we don't have a database: it gets slightly harder. <p>One thing that is easier with troggle is that we don't have many object lifecycle issues. Everything is created once and lasts forever. There are only a few ephemeral objects during the initial data import from files.
<h4>Wiring-up components</h4>
<p>Troggle today doesn't need anything complex, a single <a href="https://hub.packtpub.com/python-design-patterns-depth-singleton-pattern/">registry
singleton</a> would probably be fine (though hard to test), but if it evolves towards being a set of interacting services then a more sophisticated architecture would be needed.
<p>The Java community found "dependency resolution" very helpful for wiring-up loosely objects/components in the late 1990s with the "<a href="http://picocontainer.com/inversion-of-control.html">Inversion of Control</a>" technique which can be implemented in several ways, most commonly using "<a href="https://martinfowler.com/articles/injection.html">Dependency Injection</a>". But for troggle we must be careful that doing this the "right" way may make the code even more inaccessible to novice caver-programmers than django is. Which is the whole point of moving away from django. Fortunately python programmers have produced some recent guidance: <a href="https://blog.benpri.me/blog/2020/05/13/python-dependency-injection-made-simple/">Python Dependency Injection Made Simple</a> and <a href="https://python-dependency-injector.ets-labs.org/introduction/di_in_python.html">Dependency injection and inversion of control in Python</a>. We should probably use the simpler "<a href="http://picocontainer.com/constructor-injection.html">Constructor Injection</a>" variation as we need to make all our code <a href="http://picocontainer.com/mock-objects.html">more easily testable</a>. Flask uses that.
<hr />
Return to: <a href="trogdesign.html">Troggle design and future implementations</a><br />
Return to: <a href="trogintro.html">Troggle intro</a><br />