CUCC Expedition Handbook - Windows Bulk Updates

A Windows Bulk Update laptop

First read the generic instructions for all the software installations you will need:
Setting up a basic Expo laptop
Setting up a machine for Expo survey processing

Software

Visual Studio Code editor

If you use VS Code here are some relevant extras. Not really for beginners but here are instructions for configuring it for python. On Windows you run VS Code as a Windows app but it communicates directly ("remotely") with the WSL Linux environment.

You will definitely want the "Git History" extension and probably the "GitLens" extension too.

It is entirely feasible (and on a slow machine, recommended) to use VS Code only for git management, and using a faster Windows-native editor, such as Notepad++, for actually editing code and text.

Configuration

Idiots guide to setting up git for expo - PDF - Brendan's guide. Uses PuTTy and GitKraken on WIndows.

You need to register a key with the expo server to get upload (i.e. read/write) access. Do this first, Without it none of git, scp, ftp or rsync will work. You can do this entirely on your own if you have access to the expo laptop to upload and install the public key generated by your laptop.

On a Windows machine you will need to configure pageant (the putty authentication agent) to run at startup to load your key. Note that you are loading your private key, the .ppk file, into pageant and that this key never leaves your laptop.

Full illustrated instructions:

The above gets the command-line PuTTY tools (ssd, sftp, pscp) running, but doesn't get rsync working. You might like to try this (untested).

When using Windows please, please be excessively careful when naming files and survex names and be exceptionally careful when using rsync.

Things that already work well

Anything where the file upload and download is done via the verson control client software works really well.

Key management using PuTTy

Most Windows software that we recommend "just works" if you have set up PuTTy and have done the key-pair setup and are running a local ssh agent (pagent) automatically at boot up on your laptop.

Some software, such as the commercial (but free) GitKraken, requires that you click a checkbox to say that you are "using local SSH agent" rather than specifying ssh private keys explicitly (File->Preferences->Authentication in GitKraken).

Some software (such as Filezilla) defaults to using the local agent and it "just works".

Brendan wrote a guide to using putty and git for expo on a Windows machine. It's worth reading and it has lots of screenshots: Idiots guide to accessing expo git.pdf.

Read the online instructions about using OpenSSH in Windows as an alternative. (Then write them up and edit this handbook to be more useful.)

Things that cause problems

Filenames

Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "CON") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware.

Symbolic links and sFTP

Linux people like to use links. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks.

The links you are most likely to come across are that what looks like /home/expo/expoweb is really just a link to the folder /home/expo/repositories/git/expoweb,
and that expoweb/essentials.gpx is a link to the file /home/expo/expofiles/gpslogs/essentials/essentials2019.gpx

But that example is in the :expoweb: repository, so you won't be using sFTP to download it. Instead you will be using the version control software which handles it without problems. But we use it as an example of what to look out for when using sFTP.

There are two types of linux links: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux hard links. Fortunately we don't seem to have any hard links anywhere.

What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this,as it pftp (PuTTy) working in eith sFTP or scp mode.. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.

So the ordinary user won't notice any problems, but the nerds behind the scenes start to cuss and shout and generally carry-on in an expletive-heavy manner.

What is even more irritating is that Filezilla displays an link (essentials.gpx in the image) with a little arrow - it knows perfectly well that it is a symbolic link (although it does display it with a folder icon) - but it downloads it as a file copy.

Symbolic links: the solution for Windows

Things that are really quite involved

The core problem is integrating the PuTTy key management software (pagent.exe) with a terminal window. We need a terminal window to run rsync as none of the packaged software (Filezilla, PuTTy) includes an rsync client.

[Well, strickly speaking we need a Windows rsync executable, and this does exist, and a command-line ssh, which the 'OpenSSH in Windows' system provides. But we haven't properly explored this set of mechanisms yet.]

The solution we have now for rsync is to use WSL and to create another key, distinct from the PuTTy one, and to upload that key to the expo server. Because this is treating WSL as if it were a different machine requiring its own key quite separate from the Windows key. (This works the same way in WSL1 and WSL2 - but see "Do you feel lucky, punk" below...)

[We are not the only people to find this nasty and irritating. Other hacks are to use the obsolescent cygwin rsync or the rsync packaged within bash within more recent versions of 'Git for Windows' (which is built on MYSYS2. If you are already familiar with any of these, then use them and not WSL.]

WSL - Windows Subsystem for Linux

[On 23 November 2022 a new way of installing and using WSL was announced, see Windows Subsystem for Linux now packaged as a Microsoft Store app so all documentation on this expo site needs to be revised.]

So on a machine with WSL enabled, create an ordinary cmd window and get into the WSL environment using the wsl command:
D:\CUCC-Expo\expoweb\ wsl
which puts you into the WSL environment with a new command prompt, e.g.
philip@Muscogee:/mnt/c/Users/Philip$

and now you can setup keys in the same way as you would on a Linux machine using ssh-keygen: 

$ ssh-keygen -C "philip@muscogee-wsl"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/philip/.ssh/id_rsa): id_rsa_wsl
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa_wsl.
Your public key has been saved in id_rsa_wsl.pub.
The key fingerprint is:
SHA256:ySs0YD5IG2ZD50+riUDHWosNq+WJdqpkDlINXh709r0 philip@muscogee-wsl
The key's randomart image is:
+---[RSA 2048]----+
|  . o            |
| ..+ .           |
| oB+* +          |
|.=O%.* + o       |
|.+*o= = S .      |
|.* o = . . .     |
|=++.o . . E      |
|B o    .         |
|oo               |
+----[SHA256]-----+
$
The generated key is in the current directory and you need to move them to ~/.ssh/ as is standard on Linux (which is not at all the same place that PuTTy uses to keep keys on Windows).

Now you have to complete the key-pair setup with the new key "id_ras_wsl.pub". But you don't need anyone else's help this time as you can use PuTTy to ssh into the server and copy your key to the right place yourself: it is a "Second Machine".

Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as: 

rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles

But this only works with WSL1 not WSL2 ! Under WSL1 it doesn't matter if your /expofiles/ is mounted on the Linux filesystem, typically under /home/expo/expofiles, or whether it it in the NTFS filesystem mount for Linux as, e.g. /mnt/d/expofiles. Under WSL2 /mnt/d/expofiles fails, either with a permissions problem or, if you try sudo rsync.. , with a ssh authentication failure (why? Given that we explicitly sate that we want to be user expo@ ?). Ghastly anyway. See next section, where the same problem appears on WSL1.

All of which is really a bit ridiculous if all you want to do is to use rsync on a Windows machine to sort out some photos.

Permissions, permissions..

Having happily used WSL1 in this manner for a couple of years, it was a rude awakening to try this with a new laptop with a small hard-drive, so all the expo code was mounted on a plugged-in SD card. The problem of permissions that seemed to be a WSL2 issue reappeared with a vengence on WSL1.

The recommended WSL procedure (Jan.2018) said to mount the drive using a new 'metadata' setting: 

sudo umount /mnt/d
sudo mount -t drvfs D: /mnt/d -o metadata,uid=1000,gid=1000,umask=22,fmask=111

This is probably why there is a difference between automatically-mounted drives (e.g. C:, /mnt/c) and plug-in drives: "By default, WSL will set the uid and gid to the default user with drives that are auto-mounted during instance start. If you mount manually, you will have to set these explicitly (the default user that gets created when WSL is first installed has a uid=1000 and gid=1000)."

And of course you will need to arrange that this happens automatically whenever you start WSL, so you will need to set /etc/fstab, so ensure that the relevant line says: 

D:           /mnt/d         drvfs   metadata,uid=1000,gid=1000,umask=22,fmask=11     0    0

See also File Permissions for WSL (Dec. 2021).

"Are you feeling lucky, punk"

So here is the current wild frontier. Currently these are the ways to get a terminal window which you might expect to work:

Go back to: Basic laptop
Go back to: Survey laptop