expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2026-02-26 06:05:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Documenting data import

Browse Source
This commit is contained in:
Philip Sargent
2020-05-18 22:25:07 +01:00
parent 5d9854a1e7
commit 451492112b
6 changed files with 105 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
handbook/troggle/trogarch.html
View File

@@ -12,28 +12,37 @@
<h3>Troggle data architecture</h3>
<p>
The core of troggle is the data architecture: the set of tables into which all the cave survey and expo data is poured and stored. These tables are what enables us to produce a large number of different but consistent reports and views.
<style>figure {font-weight: bold; font-size: small; font-family: sans-serif;}</style>
<div style="display: flex">
<div style="flex: 50%">
<figure>
<a href="../i/troggle-tables.jpg">
<img src="../i/troggle-tables-small.jpg" /></a>
<figurecaption>
<img src="../t/troggle-tables-small.jpg" /></a>
<br><figurecaption>Tables (Objects)</figurecaption>
</figure>
</div>
<div style="flex: 50%">
<figure>
<a href="../i/trogpkg.jpg">
<img src="../t/trogpkg-small.jpg" /></a>
<br><figurecaption>Packages</figurecaption>
</figure>
</div></div>
<h3>Architecture description</h3>
<p>Read the proposal: "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis</em>. But remember that this paper is an over-ambitious proposal. Only the core data management features have been built. We have none of the person management features and only two forms in use: for entering cave and cave entrance data.
<p>Read the proposal: "<a href="/expofiles/documents/troggle/troggle_paper.pdf" download>Troggle: a novel system for cave exploration information management</a>", by Aaron Curtis</em>. But remember that this paper is an over-ambitious proposal. Only the core data management features have been built. We have none of the "person management" features, none of the "wallet progress" stuff and only two forms in use: for entering cave and cave entrance data.
<p>
ALSO there have been tables added to the core representation which are not anticipated in that document of this diagram, e.g. Scannedimage, Survexdirectory, Survexscansfolder, Survexscansingle, Tunnelfile, TunnelfileSurvexscansfolders, Survey. See <a href="datamodel.html">Troggle data model</a> python code (3 April 2020).
ALSO there have been tables added to the core representation which are not anticipated in that document of this diagram, e.g. Scannedimage, Survexdirectory, Survexscansfolder, Survexscansingle, Tunnelfile, TunnelfileSurvexscansfolders, Survey. See <a href="datamodel.html">Troggle data model</a> python code (15 May 2020).
<h3>Troggle parsers and input files</h3>
[describe which files they read and which tables they write to. Also say what error messages are likely on import and what to do about them.]
<ul>logbooks
<li>surveyscans
<li>survex files (caves)
<p>To understand how troggle imports the data from the survex files, tunnel files, logbooks etc., see the <a href="trogimport.html">troggle import (databaseReset.py)</a> documentation.
<p>The following separate import operations are managed by the import utility <a href="trogimport.html">(databaseReset.py)</a>:
<ul>
<li>expo logbooks
<li>folk (people)
<li>QMs
<li>subcaves
<li>entrances
<li>drawings (tunnel)
<li>wallets of notes & scans (surveyscans)
<li>cave survey data - survex files
<li>QMs - question marks
<li>drawings - tunnel & therion files
</ul>
@@ -42,7 +51,11 @@ ALSO there have been tables added to the core representation which are not antic
<p>There are only two places where this happens. This is where online forms are used to create cave entrance records and cave records. These are created in the database but also exported as files so that when troggle is rebuilt and data reimported the new cave data is there.
<h3>Helpful tools and scripts</h3>
[ALSO talk about useful tools, such as those which interrogate MySQL or sqlite databases directly so that one can see the internals chnage as data is imported]
<img class="onleft" width = "100px" src="https://mariadb.com/kb/static/images/logo-2018-black.95f5978ae14d.png">
<img class="onright" width = "80px" src="https://sqlite.org/images/sqlite370_banner.gif">
<p>The public server uses a <a href="https://mariadb.org/about/">MariaDB SQL database</a> and development is usually done using a single-user <a href="https://sqlite.org/about.html">sqlite database</a> which is a standard django option.
<p>
You will find it very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. A light-weight, simple db browser is <a href="https://sqlitebrowser.org/">DB Browser for SQLite</a>. Connecting directly the the MariaDB database with a control panel or <a href="https://www.mysql.com/products/workbench/">workbench</a> gives even more tools and documentation capabilities.
<hr />
Go on to: <a href="trognotes.html">Troggle uncategorised notes to be edited</a><br />