From 595125ea3c72511eea57093d6eb3c83df8f3ae6a Mon Sep 17 00:00:00 2001 From: Philip Sargent Date: Thu, 11 Jul 2019 16:25:19 +0200 Subject: [PATCH] Fixing FTP instructions as sfpt and scp no longer work for generic users --- handbook/fzconfig.html | 13 ++- handbook/keyexchange.html | 32 ++++++++ handbook/upload-expert.html | 80 +++++++++++++++++++ handbook/uploading.html | 154 ++++++++++++++++++++---------------- handbook/zilla-uploads.xml | 13 +-- 5 files changed, 218 insertions(+), 74 deletions(-) create mode 100644 handbook/keyexchange.html create mode 100644 handbook/upload-expert.html diff --git a/handbook/fzconfig.html b/handbook/fzconfig.html index aca97488d..0e8cda093 100644 --- a/handbook/fzconfig.html +++ b/handbook/fzconfig.html @@ -7,6 +7,17 @@

CUCC Expedition Handbook

Configuring Filezilla to upload photos etc.

+ +

Installing Filezilla

+

This software works identically on both Windows and Linux. +

Filezilla is an "FTP client". This means that it connects to servers using a venerable service called "file transfer protocol" i.e. FTP. It looks a bit like copying files from one folder to another on your desktop but it works between different machines. +

+ +

Before continuing, make sure that you have the "cavey:beery" password for the expo server. You will need it.

Having installed Filezilla using the installation instructions, you now need to import a configuration file to set it up to use with the public expo server. @@ -36,7 +47,7 @@ may try to delete things on the server which you don't have on your laptop, i.e. This setting should be fixed automatically in zilla-uploads.xml but we have seen occasions where it seems to have been re-set somehow. -

  • Now you are in and can copy and move files anywhere. But please don't: stick to copying files from your machine (the left window) to the server (the right window):

    +
  • Now you are in and can copy and move files anywhere. But please stick to copying files from your machine (the left window) to the server (the right window) into the /uploads/, /photos/ or /gpslogs/ folders only:


    (Click to enlarge). diff --git a/handbook/keyexchange.html b/handbook/keyexchange.html new file mode 100644 index 000000000..7d47ef41d --- /dev/null +++ b/handbook/keyexchange.html @@ -0,0 +1,32 @@ + + + +Handbook placeholder page + + + +

    CUCC Expedition Handbook

    + +

    Placeholder

    +

    This is not the page you are looking for. +

    This will be replaced with the information you want as soon as someone gets around to writing it. Why not find out how to do this yourself ? + +

    + + + + diff --git a/handbook/upload-expert.html b/handbook/upload-expert.html new file mode 100644 index 000000000..c4bd5d7d5 --- /dev/null +++ b/handbook/upload-expert.html @@ -0,0 +1,80 @@ + + + +CUCC Expedition Handbook: Uploading files/photos + + + +

    CUCC Expedition Handbook

    +

    Uploading Photos and files - Expert Use

    + +

    Experts only

    +

    If you have been using FTP since the last century or are particularly keen on doing everything using the command-line, read on. + +

    With the new server in 2019 it is harder to configure scp or sftp (or ssh or rsync) because the server needs a key exchange +setting up first. So while scp (once working) is now +more secure than it was, we are forced to allow ordinary ftp to allow experienced but non-key-exchanged expoers' laptops or phones to upload photos +to the server. This is worse than it was as the password is now sent in clear. + +

    So it is important that the ordinary ftp access is restricted to the /uploads/ folder tree. + +See key exchange instructions for how to arrange with an admin for this to be done for your device. + + +

    Using scp - requires key exchange for the device

    + +

    Works on Windows (using winscp), Linux (using scp), and no doubt +mac and android with other tools. If you have Windows 10 and have installed bash, then you can use scp.

    + +

    If you don't have winscp installed you can get it from here: +http://winscp.net/eng/.

    + +

    quick start guide: +http://winscp.net/eng/docs/getting_started

    + +

    screenshots: +http://winscp.net/eng/docs/screenshots

    + + + +

    scp gives you an 'explorer-like' interface (although winscp can +give you a norton-commander-style 2-pane UI as well).

    + + +

    rewite this bit... +

      +
    1. Download and install Filezilla. +
    2. Or learn to use scp. +
    3. Or on a Windows machine try Winscp. +
    4. Or if you are using an Android mobile phone, follow Radost's instructions which also cover Winscp briefly. +
    + + +

    Expo laptops

    +

    The expo laptop has got the key exchange set up on it so it is configured for Filezilla to use sftp not ftp but this is invisible to normal use. + +

    Using WebDAV

    +This no longer works as we had to change the folder permissions for /uploads/. Sorry. + +

    Using rsync

    +

    No, don't use rsync for this. Really don't. It's too liable to delete everything or to overwrite files which are not changed at all because of the incompatibilities between Linux and Windows filename conventions (uppercase and lowercase are automagically converted and rsync gets it wrong). + +

    Regenerating photos albums

    +

    ssh in to the server, cd to /expofiles/ and +

    +make photos
+
    +will regenerate it all using the installation of bins on the server +using the config in /home/expo/config/bins/binsrc + +
    + + \ No newline at end of file diff --git a/handbook/uploading.html b/handbook/uploading.html index 5d726e37a..5ded230c3 100644 --- a/handbook/uploading.html +++ b/handbook/uploading.html @@ -9,66 +9,62 @@

    Uploading Photos

    The end-result you are trying to achieve

    -What you are trying to do is to get your happy holiday snaps appear properly indexed with all the others from the previous decades of expo history. You can see them all here:
    properly indexed with all the others from the previous decades of expo history. You can see them all here:
    http://expo.survex.com/photos/
    which is the end result. But all you have to do is to upload the photos to the right place. A hidden script does the hard work to make it all look nice after the upload.

    If you are a beginner you can use the initial simple method below (using /uploads/ ) -for the photos you have taken of cave entrances for cave survey and prospecting purposes. But please -rename the filenames of the photos intelligently, e.g. "big-hole-near-path-to-fgh.jpg", or -"2018-ad-07-entrance3.jpg" (rather than "DSC31415926.jpg"), and explain to an admin/nerd what you have done. -Please use lower-case for all filenames. +for the photos you have taken of cave entrances for cave survey and prospecting purposes.

    If you are looking for how to upload a GPS track, those instructions have been moved to here. + +

    Names for your photo files and folders

    +

    Please use lower-case for all filenames. +

    Please +rename the filenames of the photos intelligently to e.g. "big-hole-near-path-to-fgh.jpg", or +"2018-ad-07-entrance3.jpg" rather than "DSC31415926.jpg" or whatever your camera app produces. Please explain by email to an admin/nerd what you have done, where you have put them and how you have renamed files. + +

    Please don't upload lots of near-duplicate photos: cull them first to just the good ones. We don't want any that are out of focus either. + +

    When copying many files put them into a new folder which identifies you personally. +When you copy the files create your own folder with your name like this: 

    /YourName/
    and put your files into that folder. "YourName" should have no spaces, hyphens or underline characters (CamelCase). +

    Simple instructions

      -
    1. Email a photo or two to someone who knows how to do it.
      -(If you are doing more than a few photos, email will be clunky, so use another method). -
    2. Use the Expo laptop in the tatty hut. Get someone to show you how to use FTP. +
    3. If you only have a photo or two just email them to someone who knows how to do it.
      +If you are doing more than a few photos, email will be clunky, so do this: +
    4. Copy the photo files from your phone to the expo laptop in the tatty hut using a USB cable.
      See phone app if you have problems. +
    5. Rename the filenames of the photos to something descriptive (see para above). +
    6. After copying your files to the expo laptop into the /uploads/ folder tell a nerd that you have done it.
    -

    Beginners should always put all their files into the folder /home/expo/expofiles/uploads/ and ask an admin to move them to the right place. -

    Now go to using Filezilla initially - still using the Expo laptop. +

    Complete beginners should always put all their files into their own folder within the folder /home/expo/expofiles/uploads/ on the expo laptop and ask an experienced user to move them to the right place later. -

    More complex instructions

    -

    Using your own laptop on expo, or after you return from expo: -

      -
    1. Download and install Filezilla. -
    2. Or learn to use scp. -
    3. Or on a Windows machine try Winscp. -
    4. Or if you are using an Android mobile phone, follow Radost's instructions which also cover Winscp briefly. -
    +

    If you want to use your photos in a blog (see below) tell an experienced user which and they will copy a lower-resolution version to the appropriate place. -

    You do need to know the expo password.

    +

    That's it, you're done. -

    Installing Filezilla

    -

    This software works identically on both Windows and Linux. -

    Filezilla is an "FTP client". This means that it connects to servers using a venerable service called "file transfer protocol" i.e. FTP. It looks a bit like copying files from one folder to another on your desktop but it works between different machines. -

    +

    Using uploaded photos in blogs

    +

    The UK Caving blog requires that +photos in it are actually hosted elsewhere. This needs to be a permanent place so +that the blogs can be read in years to come. Imgur, Flikr or Instagram are OK though not ideal (expo has its own Instagram account) but +your personal GoogleDocs or GooglePhotos stash definitely is not. +Preferably +they should be put on the expo server e.g. 2018 blog photos are permanently archived on expo.survex.com/expofiles/photos/2018/BlogPostPhotos/ in the published resolution. -

    Using Filezilla initially

    -

    The expo website has a big section under 'home/expo/expofiles/' that is not under version control. This is dangerous as there is no backup. If you overwrite some important files with your holiday snaps then we are in big trouble. This is where we store big files that we don't want to keep multiple versions of which is why it is not under verson control. - -

    So beginners should always put all their files into the folder -

    /home/expo/expofiles/uploads/
    and then ask an admin to move them to the right place. The configuration which you just did (if you followed the instructions) will set you up pointing at the correct folder automatically. -

    -To make the admin's life easier, create your own folder in 

    /home/expo/expofiles/uploads/
    with your name like this: 
    /home/expo/expofiles/uploads/YourName/
    and put your files into that folder. ("YourName" should have no spaces, hyphens or underline characters.) So that the admin knpows who is responsible. And for goodness sake please don't upload lots of duplicate photos: cull them first to just the good ones.

    Experienced users

    -

    OK the admin/nerd you have been asking to move the files to the right place is getting fed up and now wants you to put the uploaded photos in the right place yourself. This is where they go: +

    OK the admin/nerd you have been asking to move the files to the right place on the expo laptop is getting fed up and now wants you to put the uploaded photos in the right place yourself and not just in /uploads/. This is where they go: 

    -directory: /home/expo/expofiles/photos/2018/YourName/
+directory: /home/expo/expofiles/photos/2019/YourName/
    or, for GPS logs (GPX files): 
    -directory: /home/expo/expofiles/gpslogs/2018/YourName/
+directory: /home/expo/expofiles/gpslogs/2019/YourName/
    +

    Obviously replace /2019/ with the current year. This is the expo year the photos are taken, so if you are uploading a previous years' photos which you never got around to uploading you will use /2018/ or whatever.

    Obviously replace 'YourName' with your actual name (no spaces!). It is important that you get this right as this specific way of writing your name is standardised across the website @@ -79,45 +75,69 @@ only lower-case letters, no spaces, and hyphens (not underlines) when creating f

    Use the year that the photo was taken or the GPStrack logged.

    + +

    Uploading files from the expo laptop

    +

    Always learn how to use the system first on the expo laptop before trying to do it with your own machine. +

    OK the files have been put into the correct place on the laptop by an experienced user and now you need to copy them to the server. Don't worry about deleting them from the expo laptop after copying as we can do that clean-up at the end of expo. In fact don't delete them from /uploads/ on the expo laptop. + +

    The expo server has a big section under 'home/expo/expofiles/' that is not under version control. This is dangerous as there is no backup. If you overwrite some important files with holiday snaps then we are in big trouble. + +

    This is where we store big files that we don't want to keep multiple versions of which is why it is not under verson control. + +

    This is why we do not want beginners directly uploading photos to the server but to just put them on the expo laptop instead. + +

    So assuming that you have the photos on the laptop in directory: +

    +/home/expo/expofiles/photos/2019/YourName/
+
    +you are going to be copying them to directory: +
    +/home/expo/expofiles/photos/2019/YourName/
+
    +on the server. + +

    + +

    +

    Now you are in and can copy and move files anywhere. But please stick to copying files from your machine (the left window) to the server (the right window) into the /uploads/, /photos/ or /gpslogs/ folders only:

    + +
    (Click to enlarge). + +

    For installing and pre-configuring Filezilla on a new machine see FileZilla install instructions which will set you up pointing at the correct folder automatically. + +

    Automated magic

    Note that uploading photos does not automatically update the view at http://expo.survex.com/photos/ immediately. An update script needs to be run. This should run automatically once/day around midnight UTC (2017 and earlier) or a couple of minutes after you do the upload to the right place (2018 if Wookey gets this sorted out in time) but may be broken. Prod a web admin if nothing is updated by the next morning..

    -

    Experts only

    -

    If you have been using FTP since the last century or are particularly keen on doing everything using the command-line, read on. + +

    Expert instructions

    +

    These have been moved to a separate page: expert upload instructions. -

    Using scp

    +

    Using your own laptop

    +

    To use your own laptop on expo, or after you return from expo, you need need to use FTP. So become an experienced user first. -

    Works on Windows (using winscp), Linux (using scp), and no doubt -mac and android with other tools. If you have Windows 10 and have installed bash, then you can use scp.

    +

    You will need to know the expo password.

    -

    If you don't have winscp installed you can get it from here: -http://winscp.net/eng/.

    +

    To installand configure Filezilla on your machine see FileZilla install instructions which will set you up pointing at the correct folder automatically. -

    quick start guide: -http://winscp.net/eng/docs/getting_started

    -

    screenshots: -http://winscp.net/eng/docs/screenshots

    - - - -

    scp gives you an 'explorer-like' interface (although winscp can -give you a norton-commander-style 2-pane UI as well).

    - -

    Using WebDAV

    -This no longer works as we had to change the folder permissions for /uploads/. Sorry. - -

    Using rsync

    -

    No, don't use rsync for this. Really don't. It's too liable to delete everything or to overwrite files which are not changed at all because of the incompatibilities between Linux and Windows filename conventions (uppercase and lowercase are automagically converted and rsync gets it wrong). +

    Getting files from phones

    +

    to be written. test andftp app and document properly...

    diff --git a/handbook/zilla-uploads.xml b/handbook/zilla-uploads.xml index 82088162b..78374ea91 100644 --- a/handbook/zilla-uploads.xml +++ b/handbook/zilla-uploads.xml @@ -1,5 +1,5 @@ - + expo.survex.com @@ -14,17 +14,18 @@ 0 Auto 0 - Expo-uploads + expo-uploads 0 /home/ 1 0 4 home 4 expo 9 expofiles 7 uploads 0 1 - Expo-uploadsExpo-uploads - Expo-uploads - /home/ - 1 0 4 home 4 expo 9 expofiles 7 uploads + + expo-uploads + ~ + 1 0 7 uploads + 1