CUCC Expedition Handbook

Blog Import

Importing the UK Caving Blog into troggle

This is straightforward but a bit time-consuming. You need a Troggle software development machine and be happy running the python programe databaseReset.py at the command line. (The expo laptop is not a Troggle software development machine.)

Simply: we import all the logbook entries and blog posts for an expo into the database, then export them to a single file. This file is then used for future database resets.

This is the online UK Caving Blog for Expo 2022

1. Use a web browser to save the UK Caving blog to a file and sub-folder holding the images.
2. Go to the image folder and make all photos smaller, and convert .png files to .jpg. Delete all non-image files.
3. Edit the troggle import parser troggle/parsers/logbooks.py to include a line of code to do the import.
4. Run python databaseReset.py logbooks to import all the logbooks including the blog.
5. Export all the logbook entries for the year to a single file logbook-new-format.html. using the expoadmin control panel on troggle running locally on your machine.
6. Rename the existing logbook.html as logbook-original.html and rename logbook-new-format.html as logbook.html.
7. Comment out the additional line you put into troggle/parsers/logbooks.py to import the blog.
8. Re-import all the logbooks.
9. Tidy up oddities by hand-editing logbook.html: e.g. & incidental decodings, delete blog entry comments, fix blog post author names.
10. Re-import all the logbooks to check that it all looks good. (Several times in practice.)
11. Commit and push the changes you made to the :expoweb: and :troggle: git repos.
12. Log on to the server and do a complete database reset online.

It's a bit easy to get lost in this process and forget where you were, especially if you are interrupted. So it is handy to print out this page and tick off the steps as you do them.

After step 4, the blog posts appear in the list of logbook entries in the troggle Expo page for the year, correctly dated, and with titles such as "Expo - UK Caving Blog post 3".

Future Gotcha

The UK Caving Blog regularly upgrades its software which completely changes the hidden structure of the posts. They did this sometime between the 2017 and 2018 expos. When they do it again, the function parser_blog(year, expedition, txt, sq="") in troggle/parsers/logbooks.py will need to be completely re-written. It is currently 70 lines long and uses several regular expression recognizers.

Saving the Blog

• With your browser (this example uses Chrome), go to UK Caving Blog for Expo 2022.
• Press ctrl-S and save as filename "ukcavingblog.html" in :expoweb:/years/2022/ where :expoweb: is where you keep you copy of the :expoweb: on your Troggle software development laptop.
• Now for 2022, the blog split the posts onto two pages (see image), so if that is the case with the year you are dealing with, you will need to navigate to the next page and save again, this time with the filename "ukcavingblog2.html". Our existing troggle code handles up to 4 of these, numbered sequentially.

Now delete all the non-image files in the "ukcavingblog_files/" and "ukcavingblog2_files/" folders.

Now use your favourite photo editor (e.g. Irfanview on Windows) or a command-line tool to resize all the photos. A maximum of 600 pixels wide or high, or 400 or 300 pixels wide if the image quality is poor. Keep the same filename then you don't have to try to edit the horrendously horrible HTML which was generated by the blog software. If there are any .png files, convert them to .jpg.

Look at all the photos in the file browser set to show thumbnails and delete all advertising logos etc., and delete the UK Caving header image which will be of random people not us.

Edit logbooks.py

Exporting all entries to a new file

Edit logbooks.py again

Tidy oddities

Back to Logbooks Import for Nerds documentation.
Back to Logbooks for Cavers documentation.