expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-25 08:41:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Django migrations and troggle laptops

Browse Source
This commit is contained in:
Philip Sargent 2021-10-24 16:31:22 +03:00
parent 9e98d24bb6
commit 1a5145169a
6 changed files with 41 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
handbook/computing/basiclaptop.html
View File

@ -19,7 +19,7 @@
<li>Edit the cave description and entrance description text for existing caves
<li>Create entirely new caves in the system by filling out online forms
</ul>
[more details on cave data and survey maintenance are in the <a href="manual.html">data maintenance manual</a>]
<p>Documentation on how to actually do these things are in the <a href="manual.html">data maintenance manual</a>.
<p>And using email to send the results to an expo nerd, you can:
<ul>
<li>Type up your logbook entry for any trip you do (surface or underground), but please use <a href="../logbooks.html">our standard format</a>
@ -34,13 +34,15 @@
<p>We are actively working on increasing the number of expo activities that can be done with just a browser and no installed software.
<h3>Which laptop do you need?</h3>
<p>If you have not actively used troggle since 2018, you are probably not aware of <a href="#any">all the things</a> you can now do with just a browser. Many of these capabilities are not new, but they weren't documented and had been forgotten over the past 10+ years. Now <a href="manual.html">these capabilities are documented</a>, though writing better documentation is an unending job.
<ul>
<li><a href="#any">Any laptop</a> with a broswer and email
<li><a href="#any">Any laptop</a> with a broswer and email.
<li>An <a href="#basic">expo basic laptop</a>
<li>A <a href="winlaptop.html">Windows expo basic laptop</a>
<li>A full <a href="yourlaptop.html">expo data maintenance computer</a> and Android phone config
<li>An <a href="../troggle/troglaptop.html">expo software development computer</a> for troggle programming
</ul>
<p>See <a href="../troggle/trognotes.html#devtree">the expertise sequence</a> which lists what you can do at each stage.
<h2 id="basic">Your own basic laptop</h2>
<p>If you are new to expo and can't do what you want with just a browser and email, then please use the <em>expo laptop</em> in the potato hut first. You don't <em>need</em> to use your own laptop - which can take several hours to configure completely.

6
handbook/troggle/trogarch.html
View File

@ -85,12 +85,6 @@ See the <a href="https://docs.djangoproject.com/en/dev/misc/design-philosophies/
<h3>Files generated by troggle</h3>
<p>There are only two places where this happens. This is where online forms are used to create cave entrance records and cave records. These are created in the database but also exported as files so that when troggle is rebuilt and data reimported the new cave data is there.
<h3>Helpful tools and scripts</h3>
<img class="onleft" width = "100px" src="https://mariadb.com/kb/static/images/logo-2018-black.95f5978ae14d.png">
<img class="onright" width = "80px" src="https://sqlite.org/images/sqlite370_banner.gif">
<p>The public server uses a <a href="https://mariadb.org/about/">MariaDB SQL database</a> and development is usually done using a single-user <a href="https://sqlite.org/about.html">sqlite database</a> which is a standard django option.
<p>
You will find it very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. A light-weight, simple db browser is <a href="https://sqlitebrowser.org/">DB Browser for SQLite</a>. Connecting directly the the MariaDB database with a control panel or <a href="https://www.mysql.com/products/workbench/">workbench</a> gives even more tools and documentation capabilities.
<hr />
See: <a href="datamodel.html">Troggle data model</a> in python code <br />

16
handbook/troggle/trogdjangup.html
View File

@ -70,7 +70,7 @@
</div>
<h4>Major, minor and releases</h4>
<p>Django release 2.2.20 is major-version 2, minor-version 2, and patch-release 20. 2.2 is the "feature release" and patch releases within each feature release are not meant to break anything. They are just to fix bugs.
<p>Things <em>will break</em> between <a href="https://docs.djangoproject.com/en/dev/internals/release-process/">feature releases</a> which come out every 8 months.
<p>Things <em>will break</em> between <a href="https://docs.djangoproject.com/en/dev/internals/release-process/">feature releases</a> which come out every 8 months. We plan on upgrading troggle on the server whenever we upgrade the server operating system, which we do only every 2-3 years between <a href="https://wiki.debian.org/LTS">Debian LTS releases</a>.
<p>You will come to rely extensively on <a href="https://docs.djangoproject.com/en/3.2/releases/">the release notes and versions documentation</a> which is maintained by django.
<p>You will also need to read the django <a href="https://docs.djangoproject.com/en/3.2/howto/upgrade-version/">guide to upgrading to newer versions</a>.
@ -86,7 +86,7 @@
<a href="https://pypi.org/project/django-extensions/">pypi.org/project/django-extensions/</a> (only available for django 2.2 and later). ]
<h3 id="dev">Troggle software development</h3>
<p>Upgrading the version of django used by troggle is a serious programming job. It is not just a matter of editing a few config files. You will need a full troggle software development environment set up on your machine including, most definitely, the capability of running the troggle test suite. See <a href="troglaptop.html">how to set up a troggle software development laptop</a>.
<p>Upgrading the version of django used by troggle is a serious programming job. It is not just a matter of editing a few config files. You will need a full troggle software development environment set up on your machine including, most definitely, the capability of running the troggle test suite. See <a href="troglaptop.html">how to set up a troggle software development laptop</a>. Note particularly that you will find sqlite database browser software very helpful and that you will need to know git.
<h3 id="djangotricks">Important Tricks</h3>
<p>There are six critical tricks that make everything much, much easier:
@ -119,6 +119,18 @@
<var>-Wall</var> is a standard python option and gives warnings of deprecated python features used by django and all the current plugins. So it tells us that django 1.11.29 is using a deprecated python language feature which will be removed from the language in python 3.9 . Python version compatibilities are documented at the top of each x.0 release note. From Django 3.0 it requires python 3.6.
<h3>The upgrade process</h3>
<h4>Django migrations - prior reading</h4>
<p>Even if you have made no changes to the data model, and thus the database scheme is unchanged, you still have to run the Django migration software between different Django Feature Releases, e.g. between 3.1.x and 3.2.y .
<p>Even if you have not made any changes, Django itself may have made substantial changes to its admin data model and so the database structure needs to be migrated to the new structure - even though it is an empty database since we recreate it all every time we do a data import.
<ul>
<li><a href="https://docs.djangoproject.com/en/3.2/topics/migrations/">Django migrations documentation</a>. Read it all even if you don't understand it all yet. You will come back to it.
<li><a href="https://realpython.com/django-migrations-a-primer/">Django Migrations: A Primer</a>. A training course in migrations. Really useful.
<li><a href="https://docs.djangoproject.com/en/3.2/ref/django-admin/#django-admin-makemigrations">command line documentation</a> for the 'makemigrations' and 'migrate' commands.
</ul>
<img class="onright" width = "80px" src="https://sqlite.org/images/sqlite370_banner.gif">
<p>Now might also be a good idea to get familiar with using a database browser (see <a href="troglaptop.html#dbtools">configuring a troggle laptop</a>) and use it with the example project described in the Primer article: which you should run through on your own machine.
<h4>Running troggle before and after a Django migration</h4>
<ol>
<li>ensure that you have the exact version of python installed on your machine as is live for troggle on the server, e.g. do <var>$ sudo apt install python3.7.5</var>.
<li>create a venv using the version of python to be used.

2
handbook/troggle/trogimport.html
View File

@ -58,7 +58,7 @@ survexblks (s) 1153.1 - 3917.0 1464.1 1252.9
tunnel (s) - - 25.5 - 23.1
scans (s) - - 52.5 - 45.9
</pre></code>
[This data is from May 2020 immediately after troggle had been ported from python2 to python3 but before the survex import was re-engineered. Since July it takes only ~80s for a full reset.]
[This data is from May 2020 immediately after troggle had been ported from python2 to python3 but before the survex import was re-engineered. It now takes only ~5 minutes for a full reset.]
<p>The 'survexblks' option loaded all the survex files recursively following the <var>*include</var>
statements. It took a long time when memory was low and the operating system had to page a lot. This has now been rewritten and the all batched within a single database transaction.
<p>(That value of 0 seconds for QMs looks suspicious..)

30
handbook/troggle/troglaptop.html
View File

@ -32,7 +32,7 @@ handbook/computing/yourlaptop.html
<li>Install in <a href="https://docs.microsoft.com/en-us/windows/wsl/install">WSL on Windows 10</a>.
<li>Install <a href="https://ubuntu.com/download/desktop">Ubuntu 20.04 LTS</a>.
</ul>
<p>and familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in <a href="serverconfig.html">The Expo Server</a>. We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on django.
<p>and familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in <a href="serverconfig.html">The Expo Server</a>. We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on Django.
<h3>WSL on Windows</h3>
<p>The standard documentation for Ubuntu or debian below all works, but you should first skim the
@ -54,7 +54,7 @@ sudo ln -s pip3 pip </code></pre>
<p>You will also definitely need sqlite3 even if you are planning to use another
database. Sqlite3 is used by the test harness system.
<p>Note that when you install survex it installs a shed load of packages that it needs.
<p>We do <em>not</em> install django at this point. We will be installing django in a separate virtual
<p>We do <em>not</em> install Django at this point. We will be installing Django in a separate virtual
environment (a 'venv'), not in the main linux system.
<h2 id="links">Creating folders and soft links</h2>
@ -82,15 +82,15 @@ ls -tlA expo</pre></code>
<img src="Django_Logo-420x180.png" align="right" hspace="20" width='210' alt='django logo'>
<p>The important point to note
here is that unless you are doing something fairly trivial, or you are a git genius,
it is sensible to set up a python virtual environments to hold duplicate copies of both troggle and django code.
it is sensible to set up a python virtual environments to hold duplicate copies of both troggle and Django code.
Then you will be able to check very quickly that your edited version of troggle runs with old, current
and pre-release versions of python and of django;
and you will more easily be able to manage problems with incompatible versions of django plugins as installing and upgrading the dependent packages is very fast.
and pre-release versions of python and of Django;
and you will more easily be able to manage problems with incompatible versions of Django plugins as installing and upgrading the dependent packages is very fast.
<h3 id="venv">Installing a venv</h3>
<p>We set up
a <var>venv</var> specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and <a href="https://docs.djangoproject.com/en/3.2/releases/2.2/">django 2.2.19</a>. See the <a href="https://docs.python.org/3.7/library/venv.html">standard python documentation on venv</a> for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it <var>python3.7 -m venv py37d22</var>.
a <var>venv</var> specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and <a href="https://docs.djangoproject.com/en/3.2/releases/2.2/">Django 2.2.19</a>. See the <a href="https://docs.python.org/3.7/library/venv.html">standard python documentation on venv</a> for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it <var>python3.7 -m venv py37d22</var>.
Note that we are creating it as a sibling folder to /expoweb/ . Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':
<pre><code>cd ~
cd ../expo
@ -99,12 +99,12 @@ cd py37d22
source bin/activate
pip list -o</pre></code>
<p>The last command lists the default packages installed in the venv. This is for comparison later, after we
have installed troggle, django and dependencies. You will get a warning that you have an out of date version of pipbut this is as we want: we are using a version of pip appropriate for the older version of python within
have installed troggle, Django and dependencies. You will get a warning that you have an out of date version of pipbut this is as we want: we are using a version of pip appropriate for the older version of python within
the venv.
<img border="1" class="onright" width="150px" src='tricky-troggle.jpg' hspace="20" alt='troggle logo'/></a>
<h3 id="venv">Installing the troggle dependencies</h3>
<p>The first time you do this on a new machine you can't complete the pip installation of django as you have not yet got the
<p>The first time you do this on a new machine you can't complete the pip installation of Django as you have not yet got the
dependencies appropriate for troggle - because you have not yet cloned the troggle repo. So the first time it is easiest to just create requirements.txt yourself with a text editor. Without using git yourself, you can get the file from the website at <a href="http://expo.survex.com/repositories/troggle/.git/tree/requirements.txt">requirements.txt</a>. If you have already cloned all the repos, then just copy it.
<pre><code>cp ../troggle/requirements.txt .</pre></code>
where <var>requirements.txt</var> (note the capitalisation of the listed packages) is:
@ -116,7 +116,7 @@ Pillow==5.4.1
pytz==2019.1
sqlparse==0.2.4
Unidecode==1.0.23</pre></code>
This will pick up the latest django 2.2.x available, but all the other dependencies are pinned. These
This will pick up the latest Django 2.2.x available, but all the other dependencies are pinned. These
dependencies are as-standard on debian Buster (10) and you will want to experiment with upgrading them
for Bullseye (v11).
<p>Once you have the file, install the listed dependencies like this:
@ -131,15 +131,15 @@ see <a href="http://expo.survex.com/prospecting_guide/">/prospecting_guide/</a>)
tinymce is the wysiwyg in-browser
editor (disabled pending reinstatement)
<p>This is also documented in the <a href="trogdjangup.html">updating django for troggle page</a>
where it describes upgrading and testing with later versions of django.
<p>This is also documented in the <a href="trogdjangup.html">updating Django for troggle page</a>
where it describes upgrading and testing with later versions of Django.
<p>If you have not used pip before, read <a href="https://linuxize.com/post/how-to-install-pip-on-ubuntu-20.04/">this</a>
<h3 id="troginstall">Installing troggle</h3>
<p>The <var>:troggle:</var> repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/django template code. This is over 600 files in over 400 folders, but only 38MB in size.
<p>The <var>:troggle:</var> repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/Django template code. This is over 600 files in over 400 folders, but only 38MB in size.
<h4>key exchange</h4>
<p>Follow this link to <a href="../computing/keyexchange.html">register a key with the expo server</a> to get git access if you have not already cloned the <var>:troggle:</var> repo.
@ -204,6 +204,12 @@ Nearly half the code deals with importing and parsing data, so you need to test
<h4>Use git to commit your edits</h4>
<p>You need to know git. Sorry, but there it is. See <a href="../computing/repos.html">our git repositories</a> and <a href="../computing/qstart-git.html">our git cheat sheet</a>.
<h3 id="dbtools">Helpful database tools and scripts</h3>
<img class="onleft" width = "100px" src="https://mariadb.com/kb/static/images/logo-2018-black.95f5978ae14d.png">
<img class="onright" width = "80px" src="https://sqlite.org/images/sqlite370_banner.gif">
<p>The public server uses a <a href="https://mariadb.org/about/">MariaDB SQL database</a> and development is usually done using a single-user <a href="https://sqlite.org/about.html">sqlite database</a> which is a standard Django option.
<p>
You will find it very, very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. This is vital when doing Django migrations between Django versions. A light-weight, simple db browser is <a href="https://sqlitebrowser.org/">DB Browser for SQLite</a>. Connecting directly the the MariaDB database with a control panel or <a href="https://www.mysql.com/products/workbench/">workbench</a> gives even more tools and documentation capabilities.
<hr />

7
handbook/troggle/trognotes.html
View File

@ -27,7 +27,7 @@
and you will also also have merely skimmed as irrelevant or trivial:
<ul>
<li><a href="serverconfig.html">Troggle server configuration</a> - troggle running on a new machine<br>
<li><a href="serverconfig.html">Troggle server configuration</a> - troggle running on a new server<br>
<li><a href="scriptsother.html">Other scripts</a>
<li><a href="scriptscurrent.html">Additional Scripts</a> - more detail<br>
<li><a href="../computing/repos.html">Version control</a> - why we have repos<br>
@ -56,10 +56,11 @@ to <a href="../computing/contribute.html#programming">get involved in the progra
<li><a href="exporttgz.html">Compressed data export</a> - how to extract cave and expo data in gzip/tar format.
</ul>
<h3>Creating your own system development machine</h3>
<p>It's a simple sequence:
<h3 id="devtree">Creating your own system development machine</h3>
<p>It's a simple sequence whcih steadily builds your expertise:
<ol>
<li>Use the online forms and reports.
<li>Read documentation to get the <a href="trogarch.html">concepts and structures</a> into your head, so that you can see how it all fits together and works.
<li>Explore the public pages of the <a href="/repositories">4 git repositories</a> including <a href="/repositories/troggle/.git/tree/">all the source code</a>
<li>Use the online forms and reports together with 'survex' and 'therion' installed on your own laptop.
<li>Get the 'expoadmin' password and explore the live system online: watch the data change in real time