<!-- Yes we need some proper context-marking here, breadcrumb trails or something.
Maybe a colour scheme for just this sequence of pages
-->
<h2id="import">Importing the UK Caving Blog into troggle</a></h2>
<p>This is straightforward but a bit time-consuming. You need a
<ahref="../troggle/troglaptop.html">Troggle software development machine</a> and be happy running
the python programe <var>databaseReset.py</var> at the command line. (The <var>expo laptop</var> is <em>not</em> a Troggle software development machine.)
<p><b>Simply:</b> we import all the logbook entries and blog posts for an expo into the database, then export them to a single file. This file is then used for future database resets.
<p>This is the online <ahref="https://ukcaving.com/board/index.php?threads/cucc-austria-expedition-2022-blog.29712/">UK Caving Blog for Expo 2022</a>
<ol>
<li>Use a web browser to save the UK Caving blog to a file and sub-folder holding the images.
<li>Go to the image folder and make all photos smaller, and convert .png files to .jpg. Delete all non-image files.
<li>Edit the troggle import parser <var>troggle/parsers/logbooks.py</var> to include a line of code to do the import.
<li>Run <var>python databaseReset.py <b>logbooks</b></var> to import all the logbooks including the blog.
<li>Export all the logbook entries for the year to a single file <var>logbook-new-format.html</var>. using the <var>expoadmin</var> control panel on troggle running locally on your machine.
<li>Rename the existing <var>logbook.html</var> as <var>logbook-original.html</var> and rename <var>logbook-new-format.html</var> as <var>logbook.html</var>.
<li>Comment out the additional line you put into <var>troggle/parsers/logbooks.py</var> to import the blog.
<li>Re-import all the logbooks.
<li>Tidy up oddities by hand-editing <var>logbook.html</var>: e.g. &amp; incidental decodings, delete blog entry comments, fix blog post author names.
<li>Re-import all the logbooks to check that it all looks good. (Several times in practice.)
<li>Commit and push the changes you made to the :expoweb: and :troggle: git repos.
<li>Log on to the server and do a complete database reset online.
</ol>
<p>It's a bit easy to get lost in this process and forget where you were, especially if you are interrupted. So
it is handy to print out this page and tick off the steps as you do them.
<p>After step 4, the blog posts appear in the list of logbook entries in the troggle Expo page for the year, correctly dated, and with titles such as "Expo - UK Caving Blog post 3".
<h3id="gotcha">Future Gotcha</a></h3>
<p>The UK Caving Blog regularly upgrades its software which completely changes the hidden structure of the posts. They did this sometime between the 2017 and 2018 expos. When they do it again, the function
<var>parser_blog(year, expedition, txt, sq="")</var> in <var>troggle/parsers/logbooks.py</var> will need to be completely re-written. It is currently 70 lines long and uses several regular expression recognizers.
<h3id="save">Saving the Blog</a></h3>
<imgsrc="blog-pages.jpg"hspace="20"align="right">
<ul>
<li>With your browser (this example uses Chrome), go to <ahref="https://ukcaving.com/board/index.php?threads/cucc-austria-expedition-2022-blog.29712/">UK Caving Blog for Expo 2022</a>.
<li>Press <var>ctrl-S</var> and save as filename "ukcavingblog.html" in <var>:expoweb:/years/2022/</var> where <var>:expoweb:</var> is where you keep you copy of the <var>:expoweb:</var> on your <ahref="../troggle/troglaptop.html">Troggle software development laptop</a>.
<li>Now for 2022, the blog split the posts onto two pages (see image), so if that is the case with the year you are dealing with, you will need to navigate to the next page and save again, this time with the filename "ukcavingblog2.html". Our existing troggle code handles up to 4 of these, numbered sequentially.
</ul>
<p>Now delete all the non-image files in the "ukcavingblog_files/" and "ukcavingblog2_files/" folders.
<p>Now use your favourite photo editor (e.g. Irfanview on Windows) or a command-line tool to resize all the photos. A maximum of 600 pixels wide or high, or 400 or 300 pixels wide if the image quality is poor. Keep the same filename then you don't have to try to edit the horrendously horrible HTML which was generated by the blog software. If there are any .png files, convert them to .jpg.
<p>Look at all the photos in the file browser set to show thumbnails and delete all advertising logos etc., and delete the UK Caving header image which will be of random people not us.
<p>Run Troggle locally, and navigate your browser to
<var>http://localhost:8000/controlpanel</var>. Select the drop-down for the year you are working on (2017 in this example).
<p>You need to login as the "expoadmin" user id, not the usual "expo" id. This has a different password but you already know what it is because you set up your local copy of Troggle.
<p>There is only one export format: "HTML 2005 style". This uses the Django template
<p>All entries for the year will be exported in date order, which may not be the order they were originally written in the paper logbook.
<p>Some logbooks have "front matter": text and images which are not part of any trip entry. This front matter is copied out when the logbook.html file was most recently parsed and is copied in to the front of the generated export file. The export file is always called "logbook-new-format.html" and is located in the same folder as "logbook.html". If there is a file of that name already there it is overwritten without warning.
<p><p>Edit this bit in the obvious manner to show that you have done all the work for 2017. Don't just delete the line, make it obvious that the importing job was done:
<code><pre>BLOG_PARSER_SETTINGS = {
# "2017": ("ukcavingblog.html", "parser_blog"), # now folded in to logbooks.html
<p>With the blog, we have well known expoers labelled as unrecognized because while they posted to the blog, they were not actually on expo in that year. This is not a bug, but don't be confused by it.
<p>Somewhere the encode/decode process of exporting the content of the trip writeups is turning quote marks into question marks, and > into &gt;. Currently these are all being hand-edited to fix. The fault
is somewhere in the settings for rendering a dictionary using a Django template, and hard to find and fix.
<p>It it noticeable that a single blog post may cover several trips, and that the blog post date may be several days after the trip(s). So you need to manually find out the exact date of the trip (from the other trip records and particularly from the Bier Book) and change the date on the entry.
<p>One blog post may also need to be split into several entries - in which case be careful with the 'id=' string as this needs to be unique for each entry.
When you split a blog into different entries the quickest way to re-order everything in date-order is to export the logbook and re-import it.
<h3id="final">Finally: Separate out Training Weeks</a></h3>
<p>The very last thing to do is to edit 'logbook.html' to remove the pre-expo training events and to put them into a file 'training-weekends.html', and to edit the 'index.html' file to link to that as well as to the logbook itself. This is necessary for the pre-expo material to get indexed by the server free-text indexer and search engine.
