Troggle programmers guide - beginnings

This commit is contained in:
Philip Sargent
2021-04-29 17:52:28 +01:00
parent b09a66b6ff
commit 07b5b8aa8e
13 changed files with 195 additions and 93 deletions

View File

@@ -19,6 +19,7 @@
<li>Edit the cave description and entrance description text for existing caves
<li>Create entirely new caves in the system by filling out online forms
</ul>
[more details on cave data and survey maintenance are in the <a href="manual.html">data maintenance manual</a>]
<p>And using email to send the results to an expo nerd, you can:
<ul>
<li>Type up your logbook entry for any trip you do (surface or underground), but please use <a href="../logbooks.html">our standard format</a>

View File

@@ -23,9 +23,11 @@ did has been properly recorded and
<a href="/personexpedition/BeckaLawson/2009">attributed</a>, or maybe you have
seen an <a href="/survey_scans/">obvious gap in the cross-referencing</a> and would like to fix it.
<h3>Survey Data</h3>
<h3 id="surveydata">Survey Data</h3>
<p>We have a <a href="todo-data.html">Expo <em>Data</em> To-Do List</a>. If all you do is to check items on this list
and email a nerd to tell them which have already been done then that would be excellent. And if you discover any bad or missing survey data then here is where you record the problem (see "Edit this page" below).
and email a nerd to tell them which have already been done then that would be excellent.
And if you discover any bad or missing survey data then here is
where you record the problem (see "Edit this page" below).
<p>The biggest job after every expo is "tunneling": turning the typed-up survex data into drawings.
@@ -47,7 +49,9 @@ underground sketches (in the plastic survey wallets) as scanfiles into the
But the <a href="/expofiles/surveyscans/2019/walletindex.html">outstanding tasks list</a> after expo
is always much longer than we would like.
So <a href="/survexfile/291">the list of surveys attached to a cave</a> is incomplete,
as can be seen by looking at <a href="">the equivalent logbook records of trips</a> for those dates
as can be seen by looking at
<a href="/logbookentry/2019-07-13/2019.s08.plateau-fi">the equivalent logbook records of trips</a>
for those dates
(scroll down to the bottom of the expedition page for the calendar).
<p>There is always a backlog of surveyed caves that need their scanned centrelines annotating with
@@ -62,7 +66,7 @@ because the survex data uses an old format
In 1999 and earlier the scanned notes and wallets are not connected to the survex files either as you can see
in the blank "survex blocks" column in <a href="/survey_scans/">the survey scans folders</a> list.
<h3>Programming</h3>
<h3 id="programming">Programming</h3>
<p>The expo software system is a multi-decade long-term project. So it is probably unlike
any software projects you have been involved with. It is quite unlike other Django projects.

View File

@@ -12,7 +12,7 @@
<p>This software works identically on both Windows and Linux.
<p>Filezilla is an "FTP client". This means that it connects to servers using a venerable service called "file transfer protocol" i.e. FTP. It looks a bit like copying files from one folder to another on your desktop but it works between different machines.
<p>These are instructions for installing it on your own machine. But <em>none of this will work</em> until you have also done the <a href="computing/keyexchange.html">key-pair setup</a> procedure.
<p>These are instructions for installing it on your own machine. But <em>none of this will work</em> until you have also done the <a href="keyexchange.html">key-pair setup</a> procedure.
<ul>
<li> Download the software from here <a href="https://filezilla-project.org/download.php?show_all=1">Filezilla Downloads</a>. ( Obviously Linux users will use their usual package management system instead of doing this download.)
<li>Now install the software following <a href="https://wiki.filezilla-project.org/Client_Installation">the instructions here</a>.

View File

@@ -108,7 +108,7 @@ you into an emacs editing window (C-x C-C is the way to exit emacs). Instead, us
<code>git commit -m "changed topcamp phone number - myName" </code>
which submits the obligatory comment with the commit operation. You should write something informative and brief about your changes between the quotation marks and also give your full name.
</ul>
<p>We hope to make this issue go away by integrating triggers, but it is a low priority (April 2020).
<p>We hope to make this issue go away by integrating triggers, but it is a low priority (April 2021).
<hr />
<p>Go on to <a href="hbmanual2.html">Editing several pages</a><br />
Return to <a href="onlinesystems.html">Online systems overview</a>

View File

@@ -73,9 +73,6 @@ you have access to the <i>expo laptop</i>.
<li>Quick <a href="manual.html#quickstart">reminders for using rsync</a> at the command line.
</ul>
<p>Simple changes to static HTML files will take effect immediately
but changes to dynamically-generated files - cave descriptions, QM lists etc. -
will not take effect, until the troggle <a href="../troggle/trogintro.html">import/update scripts</a> are run on the server. These should <a href="../troggle/scriptsother.html">run automatically and frequently</a> but currently they are run manually by nerds as the expo server is undergoing heavy software maintenance. </p>
<hr/>

View File

@@ -9,56 +9,112 @@
<h2 id="tophead">CUCC Expedition: Data Maintenance Manual</h2>
<h1>Expo Data Maintenance Manual</h1>
<p><span style="color:red">[This page currently being restructured and edited to make it more appropriate to the task. Many fragments have been moved here but not yet properly edited together. Also there are no links to other pages which will be needed.]
</span>
<h2><a id="manual">Expo data maintenance manual</a></h2>
<h2><a id="manual">Oh No! Something has changed.</a></h2>
<p>It happens all the time,
<ul>
<li>somebody annoyingly links your new entrance to an existing cave
<li>or your new discovery is discovered to be the same as something explored in the 1980s
<li>or two people each created a new online wallet for the same trip
<li>or somebody didn't read the <a href="../survey/">survey manual</a> carefully enough (tsk.)
</ul>
<p>so now you need to fix things.
<h2>Do not despair - no nerd needed</h2>
<p>OK, <em>probably</em> what you need is written up somewhere in the 7-stage '<a href="">complete process for recording cave data</a>', possibly in one of the side-pages which you skipped through when you read it.
<p>You already have the <a href="#password">password</a>, so just using troggle web pages you can already
<ul>
<li>This page is for cavers wanting to edit any expo data on their own machine. So that's the website, survey data, or drawn up surveys</li>
<li>This page is <i>not</i> for cavers wanting to know how to type in logbooks or upload photographs or edit data on the expo laptop.
<li>There is another page with more info on <a href="#yourownlaptop">the software expo uses and setting up your own laptop</a>.
<li>Edit any existing survex files, save them, and run survex on them <em>on the server</em>
<li>Create entirely new survex files, including creating intermediate subdirectories (see below)
<li>Edit the cave description text and technical data fields for existing caves
<li>Edit the entrance description text and technical data fields for existing caves
<li>Create new entrances on existing caves
<li>Reconnect an entrance to a different cave
<li>Edit HTML to add an existing uploaded photo into the description text for an extrance or cave
<li>Create entirely new caves in the system by filling out online forms
<li>Upload scanned notes to an online wallet
<li>Edit any HTML page in the online handbook: correct errors, update phone numbers
<li>Edit the HTML of any online expo logbook
</ul>
<h3>Data maintenance task list</h3>
<p>We have an online list of outstanding data maintenance tasks. See the <a href="contribute.html#surveydata">'Survey Data' to-do</a> list
<h3><a id="usernamepassword">Getting a username, password and key</a></h3>
<p>You don't need a password to view most things, but you will need one to change them.</p>
<h3>Clever use of Survex file editor</h3>
<p>You will have seen this when following a link, e.g. from <a href="/survexfile/204">cave 204</a> to <a href="/survexfile/caves-1623/204/nearend/stitchthis.svx">stitchthis.svx</a>.
<p>But if you hand-edit the address bar in your browser to, say, <br /><a href="/survexfile/caves-1623/2020-W-01/mynewcave.svx">/survexfile/caves-1623/2020-W-01/mynewcave.svx</a> the page will load at that new address and pre-fill the editing window with default survex content. If you edit this in the webpage and then click the "Save this edited svx file' button (you will need to remove all the square brackets while you do that). This will create both the file 'mynewcave.svx' and the directory '2020-w-01' within the existing 'caves-1623' directory on the server in the <var>:loser:</var> repository directory.
<p>Note that you should have set the '*ref' field correctly to the number of the plastic wallet you are using.
<p>Use these credentials for access to the troggle site. The user is 'expo',
with a cavey:beery password. Ask someone if this isn't enough clue for you.
<b>This password is important for security</b>. The whole site <strong>will</strong> get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not unencrypted email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.
<p>You will use this when creating a new survex file from your hand-written notes (but only after you have photographed them and put them in the plastic wallet in the potato hut of course).
<p>This is the method you will use when setting <a href="https://survex.com/docs/manual/genhowto.htm">'*equate' and '*export' survey points</a> between different cave surveys to link everything together later, and for adding missing *ref references, correcting the spelling of surveyors' surnames etc.
<p>You can also 'move' survex files by the method, with careful use of cut and paste, but you cannot delete them. Also you cannot save an empty file: it has to be a valid survex format and include the required fields *begin and *end.
<p>Always click the button: 'Run cavern on this svx file' after you have edited any survex file. This will check that there is no typo and so will not crash the bulk import of 1,200 survex files when troggle starts up.
<p>You can create new directories to any depth of nesting by this method - all at once.
<p>Editing or creating new survex files means that the <var>:loser:</var> repository will need to be updated [automation of this is a pending task].
<h3>Clever use of the Cave editor</h3>
<em>[ New cleverness to be added in here]</em>
<p>Editing or creating cave description data files means that the <var>:expoweb:</var> repository will need to be updated [automation of this is a pending task].
<h3>Clever use of Entrance editor</h3>
<em>[ New cleverness to be added in here]</em>
<p>Editing or creating entrance description data files means that the <var>:expoweb:</var> repository will need to be updated [automation of this is a pending task].
<h3>Clever use of the Scan Upload form</h3>
<em>[ New cleverness to be added in here]</em>
<p>Editing or uploading scan files is in /expofiles/ and so not in a version-controlled repository [be extra careful].
<h3>Clever use of 'Edit this page'</h3>
<p>You can create new pages as well as edit existing pages. This is all documented in the <a href="/hbmanual1.html">handbook editing manual</a>
<em>[ New cleverness to be added in here]</em>
<p>Editing or creating handbook or logbook HTML files means that the <var>:expoweb:</var> repository will need to be updated [automation of this is a pending task].
<h3 id="password">The expo password</h3>
<p>The username is 'expo',
with a cavey:beery password. It is written on the notice-board inside the potato hut.
<b>This password is important</b> - keep it safe.
<p>The whole site <strong>will</strong> get hacked by spammers or worse if you are not careful with it. Use a secure method for passing it on to others that need to know (i.e not ordinary email), don't publish it anywhere, don't check it in to the data management system by accident. A lot of people use it and changing it is a pain for everyone so do take a bit of care.
</p>
<p>This password is all you need to log in to troggle and to use the troggle control panel (very few people need to do this). But if you want to update webpages (a much more common requirement) or to edit the software itself (very rare), then
you will also need to get a login (register a key with the server). See <a href="keyexchange.html">key-pair setup</a> for details.
<p>Pushing cave data to the :loser: and :drawings: <a href="../computing/repos.html">repositories</a> also needs a key. So cavers entering their cave survey data have to use a machine on which this already set up. These machines are
the <i>expo laptop</i> and the laptop '<i>aziraphale</i>' which live in the potato hut during expo. If you want to use your own laptop then
see <a href="#yourownlaptop">below</a>.
<h3><a id="cavepages">Updating cave pages</a></h3>
<span style="color:red">
<p>Public cave description pages are automatically generated by troggle from a set of
cave files in /cave_data/ and /entrance_data/. These files
are named <area>-<cavenumber>.html (where area is 1623 or 1626), e.g. /cave_data/1623-115.html
<p>
Read the survey handbook section on <a href="../survey/caveentry.html">creating a new cave</a> in the system for instructions on how to name caves and the files you use to recoird them.
<p>Cave names do not have leading zeros
They are stored by number/ID in the dataset, not by name.
<p>Caves with a provisional number consisting of a year and a serial number
should be hyphenated, thus 2002-04 not 2002_04 or any of the various other
variants
<p>Clicking on 'New cave' (at the bottom of the cave index) lets you enter a new cave. <a href="caveentry.html">Info on how to enter new caves has been split into its own page</a>.</p>
<p>This password is all you need to log in to troggle. There is also an a systems account 'expoadmin' with a different password which enables the <a href="/controlpanel">import/export control panel</a> for re-importing <em>all</em> the input data files.
<p>This may be a useful reminder of what is in a survex file <a href="/expofiles/documents/surveying/survex-guide.pdf">how to create a survex file</a>.
<h2>Beyond the easy stuff</h2>
<p>There are several standard data maintenance jobs that haven't yet got trogglized. So if:
<ul>
<li>We get a new Austrian kataster number for a cave, so we need to move and rename a lot of files
<li>A cave has got so big that we need to create sudirectories for the long and complicated cave descriptions in separate HTML files
<li>Some scanned notes have been uploaded into the wrong online wallet
</ul>
we will need to use the things* installed on the <em>expo laptop</em> or on <a href="basiclaptop.html">your own laptop</a> if you are not in the potato hut on expo. <a href="fzconfig.html">Filezilla</a> will do most of what you need for moving files.
We also have <a href="uploading.html#android">andftp Android instructions</a> for file manipulation.
<p>While Filezilla (including the digital key) is enough for moving files in <a href="/expofiles">expofiles</a>, moving files in the version-controlled <a href="repos.html">repositories</a> means you need to find someone who knows git (see <a href="qstart-git.html">git cheatsheet</a>) to clean up everything after you have finished.
<p>* footnote: the 'things' include the <a href="keyexchange.html">digital key</a> that allows the laptop to be trusted by the server, as well as various installed software.
<hr />
Return to:
<a href="../computing/onlinesystems.html">expo online systems overview</a><br />
Troggle index:
<a href="../troggle/trogindex.html">Index of all troggle documents</a><br /><hr />
</body>
</html>