diff --git a/handbook/computing/basiclaptop.html b/handbook/computing/basiclaptop.html index afbc7c59f..f52ac5b39 100644 --- a/handbook/computing/basiclaptop.html +++ b/handbook/computing/basiclaptop.html @@ -19,7 +19,7 @@
Documentation on how to actually do these things are in the data maintenance manual.
And using email to send the results to an expo nerd, you can:
We are actively working on increasing the number of expo activities that can be done with just a browser and no installed software.
If you have not actively used troggle since 2018, you are probably not aware of all the things you can now do with just a browser. Many of these capabilities are not new, but they weren't documented and had been forgotten over the past 10+ years. Now these capabilities are documented, though writing better documentation is an unending job.
See the expertise sequence which lists what you can do at each stage.
If you are new to expo and can't do what you want with just a browser and email, then please use the expo laptop in the potato hut first. You don't need to use your own laptop - which can take several hours to configure completely.
diff --git a/handbook/troggle/trogarch.html b/handbook/troggle/trogarch.html
index ede20be96..41e250f08 100644
--- a/handbook/troggle/trogarch.html
+++ b/handbook/troggle/trogarch.html
@@ -85,12 +85,6 @@ See the
-
- The public server uses a MariaDB SQL database and development is usually done using a single-user sqlite database which is a standard django option.
-
-You will find it very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. A light-weight, simple db browser is DB Browser for SQLite. Connecting directly the the MariaDB database with a control panel or workbench gives even more tools and documentation capabilities.
See: Troggle data model in python code
diff --git a/handbook/troggle/trogdjangup.html b/handbook/troggle/trogdjangup.html
index 334c36c38..39b8c1dc9 100644
--- a/handbook/troggle/trogdjangup.html
+++ b/handbook/troggle/trogdjangup.html
@@ -70,7 +70,7 @@
Django release 2.2.20 is major-version 2, minor-version 2, and patch-release 20. 2.2 is the "feature release" and patch releases within each feature release are not meant to break anything. They are just to fix bugs. -
Things will break between feature releases which come out every 8 months. +
Things will break between feature releases which come out every 8 months. We plan on upgrading troggle on the server whenever we upgrade the server operating system, which we do only every 2-3 years between Debian LTS releases.
You will come to rely extensively on the release notes and versions documentation which is maintained by django.
You will also need to read the django guide to upgrading to newer versions. @@ -86,7 +86,7 @@ pypi.org/project/django-extensions/ (only available for django 2.2 and later). ]
Upgrading the version of django used by troggle is a serious programming job. It is not just a matter of editing a few config files. You will need a full troggle software development environment set up on your machine including, most definitely, the capability of running the troggle test suite. See how to set up a troggle software development laptop. +
Upgrading the version of django used by troggle is a serious programming job. It is not just a matter of editing a few config files. You will need a full troggle software development environment set up on your machine including, most definitely, the capability of running the troggle test suite. See how to set up a troggle software development laptop. Note particularly that you will find sqlite database browser software very helpful and that you will need to know git.
There are six critical tricks that make everything much, much easier: @@ -119,6 +119,18 @@ -Wall is a standard python option and gives warnings of deprecated python features used by django and all the current plugins. So it tells us that django 1.11.29 is using a deprecated python language feature which will be removed from the language in python 3.9 . Python version compatibilities are documented at the top of each x.0 release note. From Django 3.0 it requires python 3.6.
Even if you have made no changes to the data model, and thus the database scheme is unchanged, you still have to run the Django migration software between different Django Feature Releases, e.g. between 3.1.x and 3.2.y . +
Even if you have not made any changes, Django itself may have made substantial changes to its admin data model and so the database structure needs to be migrated to the new structure - even though it is an empty database since we recreate it all every time we do a data import. +
Now might also be a good idea to get familiar with using a database browser (see configuring a troggle laptop) and use it with the example project described in the Primer article: which you should run through on your own machine. + +
The 'survexblks' option loaded all the survex files recursively following the *include statements. It took a long time when memory was low and the operating system had to page a lot. This has now been rewritten and the all batched within a single database transaction.
(That value of 0 seconds for QMs looks suspicious..) diff --git a/handbook/troggle/troglaptop.html b/handbook/troggle/troglaptop.html index 2e3987541..736ada548 100644 --- a/handbook/troggle/troglaptop.html +++ b/handbook/troggle/troglaptop.html @@ -32,7 +32,7 @@ handbook/computing/yourlaptop.html
and familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in The Expo Server. We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on django. +
and familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in The Expo Server. We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on Django.
The standard documentation for Ubuntu or debian below all works, but you should first skim the @@ -54,7 +54,7 @@ sudo ln -s pip3 pip
You will also definitely need sqlite3 even if you are planning to use another database. Sqlite3 is used by the test harness system.
Note that when you install survex it installs a shed load of packages that it needs. -
We do not install django at this point. We will be installing django in a separate virtual +
We do not install Django at this point. We will be installing Django in a separate virtual environment (a 'venv'), not in the main linux system.
The important point to note here is that unless you are doing something fairly trivial, or you are a git genius, - it is sensible to set up a python virtual environments to hold duplicate copies of both troggle and django code. + it is sensible to set up a python virtual environments to hold duplicate copies of both troggle and Django code. Then you will be able to check very quickly that your edited version of troggle runs with old, current - and pre-release versions of python and of django; - and you will more easily be able to manage problems with incompatible versions of django plugins as installing and upgrading the dependent packages is very fast. + and pre-release versions of python and of Django; + and you will more easily be able to manage problems with incompatible versions of Django plugins as installing and upgrading the dependent packages is very fast.
We set up - a venv specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and django 2.2.19. See the standard python documentation on venv for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it python3.7 -m venv py37d22. + a venv specifically for python 3.7 (which is the standard version on our server which is running unmodified Buster (debian v10) and Django 2.2.19. See the standard python documentation on venv for python 3.7.12. You can upgrade the version of python installed within pip venv but not downgrade. So get that first venv installed right by explicitly stating the python version to create it python3.7 -m venv py37d22. Note that we are creating it as a sibling folder to /expoweb/ . Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':
cd ~
cd ../expo
@@ -99,12 +99,12 @@ cd py37d22
source bin/activate
pip list -o
The last command lists the default packages installed in the venv. This is for comparison later, after we -have installed troggle, django and dependencies. You will get a warning that you have an out of date version of pipbut this is as we want: we are using a version of pip appropriate for the older version of python within +have installed troggle, Django and dependencies. You will get a warning that you have an out of date version of pipbut this is as we want: we are using a version of pip appropriate for the older version of python within the venv.
The first time you do this on a new machine you can't complete the pip installation of django as you have not yet got the +
The first time you do this on a new machine you can't complete the pip installation of Django as you have not yet got the dependencies appropriate for troggle - because you have not yet cloned the troggle repo. So the first time it is easiest to just create requirements.txt yourself with a text editor. Without using git yourself, you can get the file from the website at requirements.txt. If you have already cloned all the repos, then just copy it.
cp ../troggle/requirements.txt .
where requirements.txt (note the capitalisation of the listed packages) is:
@@ -116,7 +116,7 @@ Pillow==5.4.1
pytz==2019.1
sqlparse==0.2.4
Unidecode==1.0.23
-This will pick up the latest django 2.2.x available, but all the other dependencies are pinned. These
+This will pick up the latest Django 2.2.x available, but all the other dependencies are pinned. These
dependencies are as-standard on debian Buster (10) and you will want to experiment with upgrading them
for Bullseye (v11).
Once you have the file, install the listed dependencies like this: @@ -131,15 +131,15 @@ see /prospecting_guide/) tinymce is the wysiwyg in-browser editor (disabled pending reinstatement) -
This is also documented in the updating django for troggle page -where it describes upgrading and testing with later versions of django. +
This is also documented in the updating Django for troggle page +where it describes upgrading and testing with later versions of Django.
If you have not used pip before, read this
The :troggle: repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/django template code. This is over 600 files in over 400 folders, but only 38MB in size. +
The :troggle: repo is the python source code for troggle. This is what you will be editing. There are over 8,000 lines of python code (excluding comments and blank lines) and over 2,000 lines of HTML/Django template code. This is over 600 files in over 400 folders, but only 38MB in size.
Follow this link to register a key with the expo server to get git access if you have not already cloned the :troggle: repo. @@ -204,6 +204,12 @@ Nearly half the code deals with importing and parsing data, so you need to test
You need to know git. Sorry, but there it is. See our git repositories and our git cheat sheet. +
The public server uses a MariaDB SQL database and development is usually done using a single-user sqlite database which is a standard Django option. +
+You will find it very, very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. This is vital when doing Django migrations between Django versions. A light-weight, simple db browser is DB Browser for SQLite. Connecting directly the the MariaDB database with a control panel or workbench gives even more tools and documentation capabilities.
It's a simple sequence: +
It's a simple sequence whcih steadily builds your expertise: