The python stand-alone script databaseRest.py imports data from files into the troggle database (sqlite or MariaDB). It is separate from the process which runs troggle and serves the data as webpages (via apache), but it is plugged in to the same hierarchy of django python files.
In the :troggle: directory:
$ python databaseReset.py
Usage is 'python databaseReset.py <command> [runlabel]'
where command is:
test - testing... imports people and prints profile. Deletes nothing.
profile - print the profile from previous runs. Import nothing.
- del - deletes last entry
- delfirst - deletes first entry
reset - normal usage: clear database and reread everything from files - time-consuming
init - clear database (delete everything) and make empty tables. Import nothing.
caves - read in the caves
people - read in the people from folk.csv
logbooks - read in the logbooks
QMs - read in the QM csv files (older caves only)
scans - the survey scans in all the wallets
drawings - read in the Tunnel & Therion files - which scans the survey scans too
survex - read in the survex files - all the survex blocks and entrances x/y/z
dumplogbooks - Not used. write out autologbooks (not working?)
and [runlabel] is an optional string identifying this run of the script
in the stored profiling data 'import-profile.json'
caves and logbooks must be run on an empty db before the others as they
set up db tables used by the others.
On a clean computer with 16GB of memory and using sqlite a complete import takes less than 2 minutes now if nothing else is running. On the shared expo server it takes longer if the server was in use (we have only a share of it).
Here is an example of the output after it runs, showing which options were used recently and how long
each option took (in seconds).
-- troggle.sqlite django.db.backends.sqlite3
** Running job Profile
** Ended job Profile - 0.0 seconds total.
days ago -4.28 -4.13 -4.10 -3.03 -3.00
runlabel (s) svx NULL RESET svx2 RESET2
reinit (s) - 1.9 1.9 - 1.8
caves (s) - - 39.1 - 32.2
people (s) - - 35.0 - 24.4
logbooks (s) - - 86.5 - 67.3
QMs (s) - - 19.3 - 0.0
survexblks (s) 1153.1 - 3917.0 1464.1 1252.9
survexpos (s) 397.3 - 491.9 453.6 455.0
tunnel (s) - - 25.5 - 23.1
scans (s) - - 52.5 - 45.9
The 'survexblks' option loaded all the survex files recursively following the *include statements. It took a long time when memory was low and the operating system had to page a lot. This has now been rewritten and the all batched within a single database transaction.
(That value of 0 seconds for QMs looks suspicious..)
The file import_profile.json holds these historic times. Delete it to get a clean slate.