New docum. and logbook/blog update

2024-11-21 23:01:55 +00:00 · 2022-12-17 18:34:39 +00:00 · 2022-12-17 18:34:39 +00:00 · 8d3f22f1ad
commit 8d3f22f1ad
parent 2dfcc42bd4
8 changed files with 2066 additions and 10 deletions
--- a/handbook/computing/blog-pages.jpg
+++ b/handbook/computing/blog-pages.jpg
--- a/handbook/computing/log-blog-parsing.html
+++ b/handbook/computing/log-blog-parsing.html
@ -0,0 +1,73 @@
+<!DOCTYPE html>
+<html>
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>CUCC Expedition Handbook: Logbook - importing Blog posts</title>
+<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
+</head>
+<body><style>body { background: #fff url(/images/style/bg-system.png) repeat-x 0 0 }</style>
+<h2 id="tophead">CUCC Expedition Handbook</h2>
+<h1>Blog Import</h1>
+
+    <!-- Yes we need some proper context-marking here, breadcrumb trails or something. 
+        Maybe a colour scheme for just this sequence of pages
+    -->
+
+
+<h2 id="import">Importing the UK Caving Blog into troggle</a></h2>
+<p>This is straightforward but a bit time-consuming. You need a 
+<a href="../troggle/troglaptop.html">Troggle software development machine</a> and be happy running
+the python programe <var>databaseReset.py</var> at the command line. (The <var>expo laptop</var> is <em>not</em> a Troggle software development machine.)
+
+<p><b>Simply:</b> we import all the logbook entries and blog posts for an expo into the database, then export them to a single file. This file is then used for future database resets.
+
+<p>This is the online <a href="https://ukcaving.com/board/index.php?threads/cucc-austria-expedition-2022-blog.29712/">UK Caving Blog for Expo 2022</a>
+
+<ol>
+<li>Use a web browser to save the UK Caving blog to a file and sub-folder holding the images.
+<li>Go to the image folder and make all photos smaller, and convert .png files to .jpg. Delete all non-image files.
+<li>Edit the troggle import parser <var>troggle/parsers/logbooks.py</var> to include a line of code to do the import.
+<li>Run <var>python databaseReset.py <b>logbooks</b></var> to import all the logbooks including the blog.
+<li>Export all the logbook entries for the year to a single file <var>logbook-new-format.html</var>. using the <var>expoadmin</var> control panel on troggle running locally on your machine.
+<li>Rename the existing <var>logbook.html</var> as <var>logbook-original.html</var> and rename <var>logbook-new-format.html</var> as <var>logbook.html</var>.
+<li>Comment out the additional line you put into <var>troggle/parsers/logbooks.py</var> to import the blog.
+<li>Re-import all the logbooks.
+<li>Tidy up oddities by hand-editing <var>logbook.html</var>: e.g. &amp;amp; incidental decodings, delete blog entry comments, fix blog post author names.
+<li>Re-import all the logbooks to check that it all looks good. (Several times in practice.)
+<li>Commit and push the changes you made to the :expoweb: and :troggle: git repos.
+<li>Log on to the server and do a complete database reset online.
+</ol>
+
+<p>It's a bit easy to get lost in this process and forget where you were, especially if you are interrupted. So
+it is handy to print out this page and tick off the steps as you do them.
+
+<p>After step 4, the blog posts appear in the list of logbook entries in the troggle Expo page for the year, correctly dated, and with titles such as "Expo - UK Caving Blog post 3". 
+
+<h3 id="gotcha">Future Gotcha</a></h3>
+<p>The UK Caving Blog regularly upgrades its software which completely changes the hidden structure of the posts. They did this sometime between the 2017 and 2018 expos. When they do it again, the function 
+<var>parser_blog(year, expedition, txt, sq="")</var> in <var>troggle/parsers/logbooks.py</var> will need to be completely re-written. It is currently 70 lines long and uses several regular expression recognizers.
+
+<h3 id="save">Saving the Blog</a></h3>
+ <img src="blog-pages.jpg" hspace="20" align="right">
+<ul>
+<li>With your browser (this example uses Chrome), go to <a href="https://ukcaving.com/board/index.php?threads/cucc-austria-expedition-2022-blog.29712/">UK Caving Blog for Expo 2022</a>.
+<li>Press <var>ctrl-S</var> and save as filename "ukcavingblog.html" in <var>:expoweb:/years/2022/</var> where <var>:expoweb:</var> is where you keep you copy of the <var>:expoweb:</var> on your <a href="../troggle/troglaptop.html">Troggle software development laptop</a>.
+<li>Now for 2022, the blog split the posts onto two pages (see image), so if that is the case with the year you are dealing with, you will need to navigate to the next page and save again, this time with the filename "ukcavingblog2.html". Our existing troggle code handles up to 4 of these, numbered sequentially.
+</ul>
+<p>Now delete all the non-image files in the "ukcavingblog_files/" and "ukcavingblog2_files/" folders.
+<p>Now use your favourite photo editor (e.g. Irfanview on Windows) or a command-line tool to resize all the photos. A maximum of 600 pixels wide or high, or 400 or 300 pixels wide if the image quality is poor. Keep the same filename then you don't have to try to edit the horrendously horrible HTML which was generated by the blog software. If there are any .png files, convert them to .jpg.
+<p>Look at all the photos in the file browser set to show thumbnails and delete all advertising logos etc., and delete the UK Caving header image which will be of random people not us.
+
+<h3 id="code1">Edit logbooks.py</a></h3>
+<h3 id="export">Exporting all entries to a new file</a></h3>
+<h3 id="code2">Edit logbooks.py again</a></h3>
+<h3 id="tidy">Tidy oddities</a></h3>
+<hr />
+<p>
+Back to <a href="logbooks-parsing.html">Logbooks Import for Nerds</a> documentation.<br>
+Back to <a href="../logbooks.html">Logbooks for Cavers</a> documentation.
+<hr />
+
+</body>
+</html>
+
--- a/handbook/computing/logbooks-parsing.html
+++ b/handbook/computing/logbooks-parsing.html
@ -23,6 +23,12 @@
 <p>Ideally this would all be done on a stand-alone laptop to get the bugs in the logbook parsing sorted out before we upload the corrected file to the server. Unfortunately this requires a full troggle software development laptop as the parser is built into troggle. The <var>expo laptop</var> in the potato hut is not set up to do this (yet - 2022).
 <p>However, the <var>expo laptop</var> (or any 'bulk update' laptop) is configured to allow an authorized user to log in to the server itself and to run the import process directly on the server.

+<h4>Importing the Blog</h4>
+<p>During expo lots of people post text and photos to the UK Caving (rope competition) website. During the winter after expo, an extra nerd task is to fold in all those entries into the main logbook so that
+the trips are indexed and we can see who was doing what where.
+<p>This is sufficiently complicated that it is documented 
+<a href="log-blog-parsing.html">in another page</a>. But read this page first.
+
 <h4>Current situation</h4>
 <p>The nerd needs to do this:
 <ol>
@ -76,6 +82,7 @@ which takes between 300s and 15 minutes on the server.
 <p>Older logbooks (prior to 2007) were stored as logbook.txt with just a bit of consistent markup to allow troggle parsing.</p>

 <p>The formatting was largely freeform, with a bit of markup ('===' around header, bars separating date, <place> - <description>, and who) which allows the troggle import script to read it correctly. The underlines show who wrote the entry. There is also a format for time-underground info so it can be automagically tabulated.</p>
+<p>There were also several previous (different) styles of using HTML. The one we are using now is the 5th variant. These older variants are steadily being reformatted into the current HTML format so that we only need to maintain the code for one parser.

 <!--
 <p>So the format should be:</p>
@ -88,8 +95,10 @@ which takes between 300s and 15 minutes on the server.
 T/U: Jess 1 hr, Emma 0.5 hr
 </code>
 -->
+<hr />
 <p>
-<a href="../logbooks.html">Back to Logbooks for Cavers</a> documentation.
+Back to <a href="../logbooks.html">Logbooks for Cavers</a> documentation.<br>
+Forward to <a href="log-blog-parsing.html">Importing the UK Caving Blog</a>. 
 <hr />

 </body>
--- a/handbook/logbooks.html
+++ b/handbook/logbooks.html
@ -117,9 +117,9 @@ copy it by email or USB stick to another laptop, edit it there and then copy it
 <p>If you are using your own laptop then you will need to either:
 <ul>
 <li>Just type up your trip as a separate file with a useful filename e.g. "logbook-myname-2018-08-03.txt", or just write it in an email, and send it to someone nerdish, or
-<li><a href="computing/basiclaptop.html">install and learn how to use</a> the version control software. 
+<li>Install and learn how to use the version control software. (This requires a <var><a href="computing//bulkupdatelaptop.html">Bulk Update Laptop</a></var>).
 And you will need to synchronise regularly (every day) to 
-ensure that the updates from all the people entering trip data are OK and don't get overwritten by ignorant use of this software.
+ensure that the updates from all the people entering trip data are OK and don't get overwritten by ignorant use of this software. Not recommended until you have been on a previous expo and have helped do the post-expo data tidy afterwards.
 </ul>

 <p>Logbooks are typed up and kept in the [expoweb]/years/[nnnn]/ directory as 'logbook.html'.</p>
@ -144,6 +144,7 @@ idea to type up <i>just your trip(s)</i> in a separate file, e.g. "logbook-mynew
 when there is more than one trip on a day.</p>
 <p>Note: <var><span style="color:red">T/U</span></var> stands for "Time Underground" in minutes and/or hours. (We do not parse or collate this information so the format and units don't matter.)
 <p>Note: the <var><span style="color:red">&lt;hr /&gt;</span></var> is significant and used in parsing, it is not just prettiness.
+<p>Note: follow this format exactly. No HTML comments or  tabs or newlines.

 <p>Note this special format <var>"<span style="color:red">Top Camp - </span>"</var> in the triptitle line:
 <code><pre>&lt;div class="triptitle"&gt;<span style="color:red">Top Camp - </span>Setting up 76 bivi&lt;/div&gt;</pre></code>
--- a/years/2007/logbook.html
+++ b/years/2007/logbook.html
@ -2855,8 +2855,9 @@ for the similarly big phreas in 1987-02 and less than 200m away.</p>

 <hr />

-<div class="triptitle">Rigging Guide for Tunnockshacht entrance</div>
+<div class="tripdate" id="t2007-08-14c">2007-08-14</div>
 <div class="trippeople"><u>Duncan Collis</u></div>
+<div class="triptitle">258 - Rigging Guide for Tunnockshacht entrance</div>
 <p>drawn by Duncan</p>

 <p><img src="logbkimg01.jpg" alt="258 rigging guide"></p>
@ -2967,8 +2968,12 @@ ready for a 7.30 start to catch our train to Italy.</p>

 <hr />

+<div class="tripdate" id="t2007-08-17B">2007-08-17</div>
+<div class="trippeople"><u>Becka Lawson</u></div>
 <div class="triptitle">204 - Razordance Rigging Guides - scanned in, presumably from a logbook</div>

+<p>[Author unknown, presumed to be Becka. These diagrams do not actually appear in the scanned logbook 
+so must have been done on separate bits of paper. We should search the wallets to see who did them.]
 <p><img src="logbkimg09.jpg" alt="204 Razordance rigging guide 1"></p>
 <p><img src="logbkimg13.jpg" alt="204 Razordance rigging guide 2"></p>
 <p><img src="logbkimg14.jpg" alt="204 Razordance rigging guide 3"></p>
--- a/years/2017/logbook.html
+++ b/years/2017/logbook.html
@ -607,7 +607,7 @@ Blog Author: kbender

 <div class="tripdate" id="t2017-07-23a">2017-07-23</div>
 <div class="trippeople"><u>Philip Withnall</u>, Nadia Raeburn, </div>
-<div class="triptitle">plateau - Prospecting - Bad Forecast (2017-PW-01) found</div>
+<div class="triptitle">2012-OK-01 - Prospecting - Bad Forecast (2017-PW-01) found</div>

 <br /><br />Due to another apocalyptic weather forecast, Phil and Nadia decided to do another day of prospecting north of Balcony rather than potentially getting marooned down a cave. We went back to a potential lead at a cave tagged 2012-OK-01, for which the existing prospecting notes were along the lines of ‘tagged, undropped, unsurveyed’. Not quite as bad as some of the notes for prospects, which were along the lines of ‘lost’. We dropped 2012-OK-01 off 3 naturals, to find a pleasant amount of nothingness with a peephole through to a depression in the plateau. Another one crossed off the list. <br /><br />We then went back to a potential lead north of 2011-01. Shining a headtorch down it showed a passage and a lot of dry dust. A handline was rigged (p8, 45° slope) gave us access to a cave. With a drafting phreatic passage leading off at 45° down at the bottom. We followed this down 30m until the slope angle increased and a rope was needed (which we didn’t have). Surveyed, photos and GPS coordinates taken. QMA! Nadia christened the cave ‘Bad Forecast’ since it turned out sunny all day.
 <div class="timeug">T/U: </div>
--- a/years/2018/logbook-new-format.html
+++ b/years/2018/logbook-new-format.html
--- a/years/2022/logbook.html
+++ b/years/2022/logbook.html
@ -95,11 +95,13 @@ rigging Therefore our rerigging plans turned into an enjoyable tourist trip and

 <hr />
 <div class="tripdate" id="t2022-07-31a">2022-07-31</div> 
-<div class="trippeople"><u>Frank</u>, Martin Green, Paul</div>
-<div class="triptitle">Balkonh&ouml;hle</div>
+<div class="trippeople"><u>Frank</u>, Martin Green, Paul Fox</div>
+<div class="triptitle">Balkonh&ouml;hle - Balkonh&ouml;hle</div>

 <p>Martin rigged 'hilti a plenty' whilst I fettled the bivi, then turned up a bit late. Headed down to the 'sponge chamber' sandy slope and missed the low crawl to 'let's be appalling', slid down the low slide onto the pitchhead which we started to rig. Whilst bored I climbed 2m above the pitchhead, gaining access into a tiny tube, 'sponge chamber2 tiny tube'. Phreatic tube trending down, wide enough at the top but slightly narrower below, basically its quite strenuous, pushed for a few meters beyond the end of the survey. The tube continues much the same but it is not draughting much. Headed out because we got cold, found the passage to 'let's be appalling'.</p>

+<p>[Frank's writeup just says "Paul". This is presumed to be Paul Fox not Paul Walko given that they were
+all caving together the previous day.]]
 <hr />
 <div class="tripdate" id="t2022-07-31b">2022-07-31</div>
 <div class="trippeople">Hannah, Harry, Chris D, <u>Anthony</u></div>
@ -199,7 +201,7 @@ Large hole, craps out.</p>
 <hr />
 <div class="tripdate" id="t2022-08-02c">2022-08-02</div> 
 <div class="trippeople"><u>Frank</u>, Paul Fox and Aaron Clair</div>
-<div class="triptitle">Balkonh&ouml;hle</div>
+<div class="triptitle">Balkonh&ouml;hle - Balkonh&ouml;hle</div>

 <p>Rigged down the pitch which links ‘Let’s be appalling’ to ‘Can’t stop now its bat country’. Surveyed the passage up dip from the bottom of the pitch, howling draught coming towards us, the passage ends in a clean washed chamber, possible continuation of the passage in the North East. There is a QMA lead from ‘Let’s be appalling’ above. I need to confirm that all the draught is coming down the shaft, or that the continuation of the lower passage is where the draught is coming from. Needs looking at.</p>
 <p>Followed the lower passage down dip to where the survey ended, ducked under the left wall and found a continuation of the passage.  Surveyed this. This ends in a pitch  into a small chamber with rifts exiting below, and a continuation of the passage across the pitch this appears to go into a larger space. Down appears to be 15m from the floor. Across is 5m or so. Need a drill and a rope. Continued down to below the ‘can’t stop now its bat country’ pitch which is not on the survey. Scan for rigging guide!!!!</p>
@ -282,7 +284,7 @@ phreatic passage remains to be attemped by a bold caver who still has the energy
 <hr />
 <div class="tripdate" id="t2022-08-04c">2022-08-04</div>
 <div class="trippeople"><u>Oakem</u>, Anthony and Alice</div>
-<div class="triptitle">Balkonh&ouml;hle</div> 
+<div class="triptitle">Balkonh&ouml;hle - Balkonh&ouml;hle</div> 

 <p>Anthony guided us to the Custard Cavalry in Southern Pitstop using Becka's description. The survey was being used as a "no pissing" sign in Fishface. We rigged a pitch down an aven and off a natural and a hilti down on the left. Ascended the rope left in and began surveying. Alice started with sketching but soon retreated to the bothy. Anthony took over. Oaken weilded the disto with great skill and precision. Overall about 130m of cave were mapped. The tunnel continued down a straight line before taking an obvious right turn (with a B/C lead on the left up a 3m climb.)</p>
 <div class="timeug">T/U: 13 hours</div>
@ -338,7 +340,7 @@ phreatic passage remains to be attemped by a bold caver who still has the energy
 <hr />
 <div class="tripdate" id="t2022-08-05b">2022-08-05</div>
 <div class="trippeople"><u>Nat D</u>, Chris D, Hannah C </div>
-<div class="triptitle">Balkonh&ouml;hle N Pitstop</div> 
+<div class="triptitle">Balkonh&ouml;hle - N Pitstop</div> 

 <p>After some last minute plan changed, Chris was able to convince me that a Balkon camp trip would be a good idea, despite the fact I'd only just arrived on expo/up the hill! We packed up the camp and assorted chattels (with Chris trying to unsuccessfully fat-shame Hannah and I into taking less food) along with rope, drill and hangers into several oversized tacklesacks. We managed to find some willing volunteers to assist with the ridiculous carry and we were off.</p>
 <p>Steady progress was made down to Pitstop, with some minor route finding issues in Upper Balkon trying to avoid the sandy crawl. Whilst Hannah and I faffed, Chris has found an excellent [], so we commenced work. A while later a cosy camp, wellstocked  with a variety of delicious foods, fluids and pits.  After a full on day, we got an early night.</p>