diff --git a/handbook/computing/winlaptop.html b/handbook/computing/winlaptop.html index 38730ce13..93063d313 100644 --- a/handbook/computing/winlaptop.html +++ b/handbook/computing/winlaptop.html @@ -22,9 +22,10 @@ - sFTP or scp for more than a handful of files
  • Things that are really, really hard
    - using rsync
    -- large-scale updating of several folders at once on expofiles without overwriting other people's work (which means using rsync) +- large-scale updating of several folders at once on expofiles without overwriting other people's work (which means using rsync)
    +- well not "hard" exactly, but complicated with lot of steps that are easy to get wrong and with poor feedback as to whether you have done each step correctlyS
  • Things for the bold
    -- using WSL and WSL2, Linux virtual machines and Docker containers +- using WSL2, Linux virtual machines and Docker containers

    Things that already work well

    @@ -38,7 +39,7 @@

    Things that cause problems

    Filenames

    -

    Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "CON") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware. +

    Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "CON") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware.

    Symbolic links and sFTP

    Linux people like to use links. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks. @@ -52,10 +53,10 @@ is a link to the file -There are two types of linux links: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux soft links. Fortunately we don't seem to have any hard links anywhere. +There are two types of linux links: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux hard links. Fortunately we don't seem to have any hard links anywhere. -

    What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't. +

    What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this,as it pftp (PuTTY) working in eith sFTP or scp mode.. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.

    So the ordinary user won't notice any problems, but the nerds behind the scenes start to cuss and shout and generally carry-on in an expletive-heavy manner. @@ -78,20 +79,63 @@ it downloads a copy of the contents of essentials.gpx and not a link. -

    Things that are really, really hard

    +

    Things that are really quite involved

    The core problem is integrating the PuTTy key management software (pagent.exe) with a terminal window. We need a terminal window to run rsync as none of the packaged software (Filezilla, PuTTY) includes an rsync client. -

    We don't currently (December 2019) have a working recipe to set this up. Hopefully we will have it sorted in a month or two. +

    The solution we have now is to use WSL1 and to create another key, distinct from the PuTTY one, and to upload that key to the expo server. Because this is treating WSL as if it were a different machine requiring its own key quite separate from the Windows key, we expect this to continue to work when WSL2 becomes the default behaviour on Windows10. +

    So on a machine with WSL enabled, create an ordinary cmd window and get intot he WSL environment using the wsl command:
    + +D:\CUCC-Expo\expoweb\ wsl + +
    +which puts you into the WSL environment with a new command prompt, e.g.
    + + +philip@Muscogee:/mnt/c/Users/Philip$ + +

    and now you can setup keys in the same way as you would on a Linux machine using ssh-keygen: +

    +$ ssh-keygen -C "philip@muscogee-wsl"
+Generating public/private rsa key pair.
+Enter file in which to save the key (/home/philip/.ssh/id_rsa): id_rsa_wsl
+Enter passphrase (empty for no passphrase):
+Enter same passphrase again:
+Your identification has been saved in id_rsa_wsl.
+Your public key has been saved in id_rsa_wsl.pub.
+The key fingerprint is:
+SHA256:ySs0YD5IG2ZD50+riUDHWosNq+WJdqpkDlINXh709r0 philip@muscogee-wsl
+The key's randomart image is:
++---[RSA 2048]----+
+|  . o            |
+| ..+ .           |
+| oB+* +          |
+|.=O%.* + o       |
+|.+*o= = S .      |
+|.* o = . . .     |
+|=++.o . . E      |
+|B o    .         |
+|oo               |
++----[SHA256]-----+
+$
+
+
    +The generated key is in the current directory and you need to move them to ~/.ssh/ as is standard on Linux (which is not at all the same place that PuTTy uses to keep keys on Windows). +

    Now you have to complete the key exchange process with the new key "id_ras_wsl.pub". But you don't need anyone else's help this time as you can use PuTTy to ssh into the server and copy your key to the right place yourself. +

    +Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as: +

    +rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles
    +

    "Are you feeling lucky, punk"

    So here is the current wild frontier. Currently these are the ways to get a terminal window which might work:

    -WSL: the Windows Subsystem for Linux. The first release of this didn't do the ssh key exchange process easily: "fairly annoying because of how out-to-lunch SSH Agent is". -

    WSL introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file permissions that "mirror" the NTFS permissions but can get out of sync. All sorts of fun results: "With network file systems, DrvFs does not set the correct Linux permissions bits on a file; instead, all files are reported with full access (0777) and the only way to determine if you can actually access the file is by attempting to open it." +WSL: the Windows Subsystem for Linux, variant 1. The first versions of WSL1 didn't do the ssh key exchange process easily: "fairly annoying because of how out-to-lunch SSH Agent is" but it works now. +

    WSL1 also introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file permissions that "mirror" the NTFS permissions but can get out of sync. All sorts of fun results: "With network file systems, DrvFs does not set the correct Linux permissions bits on a file; instead, all files are reported with full access (0777) and the only way to determine if you can actually access the file is by attempting to open it.". This will be fixed by WSL2 which will have an entirely separate filesystem, a Virtual Hardware Disk (VHD). Which will introduce a quite different set of interesting problems. +

    If you are disturbed by the instructions to produce an entirely different key for WSL1 to use when your PC already has a perfectly good PuTTy key installed on the server, then you are right. It is inelegant. But it works, the instructions are shorter and there are fewer things that go wrong. If you are terribly offended by that then you can set your PC up to use one key shared between WSL and normal-Windows as described in this October 2019 article. (Don't set up a password on the key because then you don't need to install keychain.) But beware, this sort of thing goes out of date quite rapidly and WSL2 is looming.

    Links to useful articles to help you work this out for yourself:

    -

    Scatch list of things that go wrong when trying to sort this out +

    When things go wrong when trying to sort this out, you may find these pages useful. I did. file-system-improvements-to-the-windows-subsystem-for-linux/ id-rsa-pub-file-ssh-error-invalid-format key-load-public-invalid-format