CUCC Expedition Handbook - Computers

Setting up a Windows expo laptop

First read the generic instructions for all the software installations you will need:
Setting up a basic Expo laptop
Setting up a machine for Expo data maintenance

It used to be quite simple: software equivalents existed for Windows for everything we needed. Since Spring 2018, we unfortunately have to use ssh keys. This has simply added an extra step for most things, but has made one thing (rsync) really quite hard to get working.

What makes this all much more tiresome is that Microsoft are introducing some cool new ways of interacting with Linux systems but these are new and need some effort to get configured correctly. See the end of this page for more details.

• Things that already work well with a Windows laptop
  - Editing the handbook webpages, typing up SVX files and transcribing the logbook,
  - Anything where the file-transfer to the expo server is via the git version control software
• Things that cause problems
  - filenames and unintentional duplication because links are not understood by Windows,
  - sFTP or scp for more than a handful of files
• Things that are really, really hard
  - using rsync
  - large-scale updating of several folders at once on expofiles without overwriting other people's work (which means using rsync)
  - well not "hard" exactly, but complicated with lot of steps that are easy to get wrong and with poor feedback as to whether you have done each step correctly.
• The easier way to do it
  - using WSL: Windows System for Linux

Things that already work well

Anything where the file upload and download is done via the verson control client software works really well.

Key management using PuTTy

Most Windows software that we recommend "just works" if you have set up PuTTy and have done the key-pair setup and are running a local ssh agent (pagent) automatically at boot up on your laptop.

Some software, such as the commercial (but free) GitKraken, requires that you click a checkbox to say that you are "using local SSH agent" rather than specifying ssh private keys explicitly (File->Preferences->Authentication in GitKraken).

Some software (such as Filezilla) defaults to using the local agent and it "just works".

Brendan wrote a guide to using putty and git for expo on a Windows machine. It's worth reading and it has lots of screenshots: Idiots guide to accessing expo git.pdf.

Things that cause problems

Filenames

Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "CON") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware.

Symbolic links and sFTP

Linux people like to use links. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks.

The links you are most likely to come across are that what looks like /home/expo/expoweb is really just a link to the folder /home/expo/repositories/git/expoweb,
and that expoweb/essentials.gpx is a link to the file /home/expo/expofiles/gpslogs/essentials/essentials2019.gpx

But that example is in the :expoweb: repository, so you won't be using sFTP to download it. Instead you will be using the version control software which handles it without problems. But we use it as an example of what to look out for when using sFTP.

There are two types of linux links: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux hard links. Fortunately we don't seem to have any hard links anywhere.

What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this,as it pftp (PuTTy) working in eith sFTP or scp mode.. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.

So the ordinary user won't notice any problems, but the nerds behind the scenes start to cuss and shout and generally carry-on in an expletive-heavy manner.

What is even more irritating is that Filezilla displays an link (essentials.gpx in the image) with a little arrow - it knows perfectly well that it is a symbolic link (although it does display it with a folder icon) - but it downloads it as a file copy.

Symbolic links: the solution for Windows

• When using the git version control systems the download of a link works fine. But be careful not to edit the link file downloaded (it is just a text file holding the path of the file holding the actual contents) because then the version control client would upload it to the server and overwrite the link on the server with something that isn't a link. It also won't work as a Windows shortcut, but at least the default behaviour isn't actively dangerous.
• When using sFTP, manually check whether any files you are copying from the server are links - look at the symbol in Filezilla.
• Be careful not to copy any links using sFTP and instead recreate them manually on the Windows filesystem using right-click "Create shortcut".
• You will have to find out what to make the shortcut link to by logging in to the server (using a PuTTy ssh logon) and doing "ls -l" in the folder where the link is.
• But later, when re-uploading edited files from Windows to the server, Filezilla will see the Windows shortcut as a ".lnk" file which it will upload, but which will mean nothing to the linux server receiving it. So you would have to manually recreate the symbolic link by logging into the server using ssh and using the "ln -s" command. Yuk.
  
• Even if you use scp instead of sFTP, it does the same stupid thing when copying from a linux filesystem to a Windows filesystem. The PuTTy package includes pscp.exe but even if you force it to use the scp protocol like this: pscp -scp expo@expo.survex.com:expoweb/essentials.gpx . it downloads a copy of the contents of essentials.gpx and not a link.
• A possible fix in the future might be to keep all your expo files in a separate partition of your hard disc which is formatted with a linux filesystem (such as ext4) and run the ext2ntfs driver to mount this fielsystem read-write from Windows. Probably not a good idea as the driver is a bit flaky in read-write mode and you could lose everything.

Things that are really quite involved

The core problem is integrating the PuTTy key management software (pagent.exe) with a terminal window. We need a terminal window to run rsync as none of the packaged software (Filezilla, PuTTy) includes an rsync client.

The solution we have now for rsync is to use WSL and to create another key, distinct from the PuTTy one, and to upload that key to the expo server. Because this is treating WSL as if it were a different machine requiring its own key quite separate from the Windows key. (This works the same way in WSL1 and WSL2)..

So on a machine with WSL enabled, create an ordinary cmd window and get into the WSL environment using the wsl command:
D:\CUCC-Expo\expoweb\ wsl
which puts you into the WSL environment with a new command prompt, e.g.
philip@Muscogee:/mnt/c/Users/Philip$

and now you can setup keys in the same way as you would on a Linux machine using ssh-keygen:

$ ssh-keygen -C "philip@muscogee-wsl"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/philip/.ssh/id_rsa): id_rsa_wsl
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa_wsl.
Your public key has been saved in id_rsa_wsl.pub.
The key fingerprint is:
SHA256:ySs0YD5IG2ZD50+riUDHWosNq+WJdqpkDlINXh709r0 philip@muscogee-wsl
The key's randomart image is:
+---[RSA 2048]----+
|  . o            |
| ..+ .           |
| oB+* +          |
|.=O%.* + o       |
|.+*o= = S .      |
|.* o = . . .     |
|=++.o . . E      |
|B o    .         |
|oo               |
+----[SHA256]-----+
$

Now you have to complete the key-pair setup with the new key "id_ras_wsl.pub". But you don't need anyone else's help this time as you can use PuTTy to ssh into the server and copy your key to the right place yourself.

Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as:

rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles

All of which is really a bit ridiculous if all you want to do is to use rsync on a Windows machine to sort out some photos.

"Are you feeling lucky, punk"

So here is the current wild frontier. Currently these are the ways to get a terminal window which you might expect to work:

• cmd window - the old faithful going all the way back to MS-DOS. But no path to an rsync.exe executable as standard.
• PowerShell terminal window - nope, no rsync.
• bash window - installed by default when you install gitforwindows. Unfortunately while this MINGW32 setup includes a command-line git executable it doesn't include rsync.
• cygwin - a cmd terminal where you have downloaded and installed Cygwin. Yes, if you have selected the rsync package you will be able to run the rysnc executable, but it won't have access to the cyptographic key so it can't connect to the expo server. Please feel free to work out how to make this work. A more recent, graphical variant is Swan.
• Windows Subsystem for Linux aka WSL available on all Windows10 machines since November 2019.
  • WSL Install it like this. This does what we need. This works using a key generated by its own version of ssh-keygen if you follow the instructions above about putting it in the right place.
  • WSL: Converting Windows paths to Linux paths and vice-versa.
  • In WSL you can choose whether to store the files in the WSL view of NTFS, e.g. /mnt/c/expo/ or in an ext4 filesystem e.g. \\wsl$\Ubuntu-20.04\home\expo . Our experience is with /mnt/c/expo/ .
  • WSL now installs as WSL2 by default, but older machines (mostly laptops) may not have the Hyper-V Virtualization hardware and may have to run WSL1. This is fine: the behaviour is identical so far as an expo laptop is concerned. Even if you have the hardware, you may have to enable it in your BIOS and install the Windows Feature to do it. Instructions here and here.
• A full virtual Linux machine running using a hypervisor such as VirtualBox which has its own virtual Linux filesystem. This setup has the advantage that you don't have to partitition your hard drive but the disadvantage that you may not have easy access to the files from Windows.
• Canonical Multipass - a VM alternative (Ubuntu only).

WSL tricks and tips

WSL unfortunately introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file permissions that are fundamentally different things and are never in sync in any sense. All sorts of fun results. The alternative is to store all your data in the \\wsl$\ ext4 filesystem (WSL2 only, not WSL1).

If you are disturbed by the instructions to produce an entirely different key for WSL to use when your PC already has a perfectly good PuTTy key installed on the server, then you are right. It is inelegant. But it works, the instructions are shorter and there are fewer things that go wrong. If you are terribly offended by that then you can set your PC up to use one key shared between WSL and normal-Windows as described in this October 2019 article. (Don't set up a password on the key because then you don't need to install keychain.) But beware, this sort of thing goes out of date quite rapidly.

Links to useful articles to help you work this out for yourself:

• Cool WSL tricks - running Windows commands from WSL environment and running Linux commands from Windows terminal.
• deep integration - Don't use gitforwindows, install the linux git client in WSL2
• using-github-credentials-in-wsl2 - How to use gitforwindows and WSL to connect to GitHub.
• WSL & Visual Studio Code

When things go wrong when trying to sort this out, you may find these pages useful. I did. rsync - failed to set permission - operation not permitted file-system-improvements-to-the-windows-subsystem-for-linux/ id-rsa-pub-file-ssh-error-invalid-format key-load-public-invalid-format ssh-error-loading-key-id-rsa-invalid-format chmod-chown-wsl-improvements mount-usb-drives-windows-subsystem-for-linux VS Code and WSL ssh command line

Installing and Configuring the rest of the software you need on Windows

Now return to the data maintenance laptop page to configure all the rest of the software you need.