expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2025-12-10 07:38:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update docs to bullseye

Browse Source
This commit is contained in:
Philip Sargent
2022-03-16 12:46:33 +00:00
parent cc6605d06b
commit 8b0056ccd5
3 changed files with 113 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
handbook/troggle/trogimport.html
View File

@@ -15,58 +15,87 @@ The python stand-alone script <var>databaseRest.py</var> imports data from files
<p>In the :troggle: directory:
<code><pre>$ python databaseReset.py
Usage is 'python databaseReset.py &lt;command&gt; [runlabel]'
Usage is 'python databaseReset.py <command> [runlabel]'
where command is:
test - testing... imports people and prints profile. Deletes nothing.
profile - print the profile from previous runs. Import nothing.
- del - deletes last entry
- del - deletes last entry
- delfirst - deletes first entry
reset - normal usage: clear database and reread everything from files - time-consuming
init - clear database (delete everything) and make empty tables. Import nothing.
caves - read in the caves
people - read in the people from folk.csv
init - initialisation. Automatic if you run reset.
caves - read in the caves (must run first after initialisation)
people - read in the people from folk.csv (must run after 'caves')
logbooks - read in the logbooks
QMs - read in the QM csv files (older caves only)
scans - the survey scans in all the wallets
scans - the survey scans in all the wallets (must run before survex)
drawings - read in the Tunnel & Therion files - which scans the survey scans too
survex - read in the survex files - all the survex blocks and entrances x/y/z
dumplogbooks - Not used. write out autologbooks (not working?)
and [runlabel] is an optional string identifying this run of the script
in the stored profiling data 'import-profile.json'
caves and logbooks must be run on an empty db before the others as they
set up db tables used by the others.
</pre></code>
<p>On a clean computer with 16GB of memory and using sqlite a complete import takes less than 2 minutes now if nothing else is running.
On the shared expo server it takes longer if the server was in use
(we have only a share of it).
<p>On a clean computer using sqlite a complete import takes less than 100 seconds now if nothing else is running.
On the shared expo server it takes 1,000s as it is a shared machine. More than half of the time on the server is reinitialising the MariaDB database.
<p>Here is an example of the output after it runs, showing which options were used recently and how long
each option took (in seconds). <code><pre>
-- troggle.sqlite django.db.backends.sqlite3
** Running job Profile
** Ended job Profile - 0.0 seconds total.
days ago -4.28 -4.13 -4.10 -3.03 -3.00
runlabel (s) svx NULL RESET svx2 RESET2
reinit (s) - 1.9 1.9 - 1.8
caves (s) - - 39.1 - 32.2
people (s) - - 35.0 - 24.4
logbooks (s) - - 86.5 - 67.3
QMs (s) - - 19.3 - 0.0
survexblks (s) 1153.1 - 3917.0 1464.1 1252.9
survexpos (s) 397.3 - 491.9 453.6 455.0
tunnel (s) - - 25.5 - 23.1
scans (s) - - 52.5 - 45.9
* importing troggle/settings.py
* importing troggle/localsettings.py
- settings on loading databaseReset.py
- settings on loading databaseReset.py
- Memory footprint before loading Django: 8.746 MB
* importing troggle/settings.py
- Memory footprint after loading Django: 31.863 MB
-- start django.db.backends.sqlite3 troggle.sqlite
** Running job 3_16_2022 to troggle.sqlite
-- Initial memory in use 31.906 MB
Reinitialising db troggle.sqlite
- deleting troggle.sqlite
- Migrating: troggle.sqlite
No changes detected in app 'core'
Operations to perform:
Apply all migrations: admin, auth, contenttypes, core, sessions
....much more output from all the import steps....
** Ended job 3_16_2022 - 89.7 seconds total.
days ago -312.64 -95.05 -95.04 -10.65 this
runlabel (s) 7th 000 wsl2_2 3_05 3_16_2022
reinit (s) 1.6 3.1 3.1 2.5 1.5 -40.2%
caves (s) 7.2 12.5 12.5 7.3 6.8 -7.2%
people (s) 9.8 11.6 11.7 9.5 9.1 -3.8%
logbooks (s) 21.2 41.0 40.9 20.2 19.9 -1.5%
QMs (s) 7.7 87.5 87.0 7.8 7.1 -8.6%
scans (s) 1.7 19.9 20.2 2.0 1.7 -14.6%
survex (s) 80.2 143.6 52.1 31.0 36.5 17.8%
drawings (s) 6.0 13.5 8.9 5.2 6.9 33.8%
</pre></code>
[This data is from May 2020 immediately after troggle had been ported from python2 to python3 but before the survex import was re-engineered. It now takes only ~5 minutes for a full reset.]
<p>The 'survexblks' option loaded all the survex files recursively following the <var>*include</var>
statements. It took a long time when memory was low and the operating system had to page a lot. This has now been rewritten and the all batched within a single database transaction.
<p>(That value of 0 seconds for QMs looks suspicious..)
[This data is from March 2022 on an 11-year old PC: Win10, WSL1+Ubuntu20.04, Intel Core i7+2600K, solid-state hard drive.]
<p>The last column shows the precentage chnage in the import runtime for each class of data. This varies quite a bit depending on
what else is running on the computer and how much has been put in virtual memory and file caches by the operating ststem.
<p>The file <var>import_profile.json</var> holds these historic times. Delete it to get
a clean slate.
<h3>Logging Import Errors</h3>
<p>Import glitches are documented on the <a href="http://expo.survex.com/dataissues">Data Issues</a> page. You should always check
this after any import. (Don't worry about the xTherion is"Un-parsed image" messages, this is work in progress.)
<p>There are detailed logs created in the <var>troggle</var> folder where you ran the import from:
<code><pre>
svxblks.log
_1623.svx
svxlinear.log
loadlogbk.log</pre></code>
<p>Severe errors are also printed to the terminal where you are running the import, so watch this. It also prints the terminal the duration of each step and the memory in use while importing the survex files.
<hr />
Return to: <a href="trogintro.html">Troggle intro</a><br />
Troggle index: