CUCC Expedition Handbook - Computers/WSL

WSL on a Windows laptop

First read the generic instructions for all the software installations you will need:
Setting up a basic Expo laptop
Setting up a Survey machine for cave survey production
Setting up a machine for expo bulk data rearrangements
Setting up a Windows machine for expo bulk data rearrangements

This is how to install and use WSL and by default we use Ubuntu24.04. We assume that you have a relatively recent machine. If not, see the bottom of this page about WSL1, which is different and old.

You will be doing all your work in the \\wsl.localhost\Ubuntu\home\username\expo\filesystem, not in C:/Users/username/expo not even if you have mounted it as /mnt/c/username/expo. If you try that you will hit file permissions problems and it will also be quite slow (but see WSL1 below).

WSL - Creating another ssh key

If you have PuTTY installed and working, but you now want to use WSL rsync, you need to set up a key again within the WSL environment.

[ Now that Microsoft allows installation of OpenSSL using standard Windows tools on Windows 10 and 11, we need someone to revise all the Windows/troggle documentation for a PuTty-free route. Volunteers? ]

So on a machine with WSL enabled, click on "Ubuntu" in the start menu system under "Apps"
which puts you into the WSL environment with a new command prompt, e.g.
username@machinename:~$ where 'username' is your logged-in user name.

and now you can setup keys in the same way as you would on a Linux machine using ssh-keygen:

$ ssh-keygen -C "username@machinename-wsl"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa): id_rsa_wsl
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa_wsl.
Your public key has been saved in id_rsa_wsl.pub.
The key fingerprint is:
SHA256:ySs0YD5IG2ZD50+riUDHWosNq+WJdqpkDlINXh709r0 philip@muscogee-wsl
The key's randomart image is:
+---[RSA 2048]----+
|  . o            |
| ..+ .           |
| oB+* +          |
|.=O%.* + o       |
|.+*o= = S .      |
|.* o = . . .     |
|=++.o . . E      |
|B o    .         |
|oo               |
+----[SHA256]-----+
$

The generated key is in the current directory and you need to move them to ~/.ssh/ as is standard on Linux (which is not at all the same place that puTTY uses to keep keys on Windows).

Now you have to complete the key-pair setup with the new key "id_ras_wsl.pub". But you don't need anyone else's help this time as you can use puTTY to ssh into the server and copy your key to the right place yourself: it is a "Second Machine".

Now finally you can use all the usual command line tools at your wsl command line to communicate with the server with ssh, scp, rsync, such as:

rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles

If you are disturbed by the instructions to produce an entirely different SSL key for WSL to use when your PC already has a perfectly good PuTTy key installed on the server, then you are right. It is inelegant. But it works, the instructions are shorter and there are fewer things that go wrong. If you are terribly offended by that then you can set your PC up to use one key shared between WSL and normal-Windows as described in this October 2019 article. (Don't set up a password on the key because then you don't need to install keychain.) But beware, this sort of thing goes out of date quite rapidly.

Python on WSL

WSL installs python by default, but it may be a rather old python (depending on the history of your laptop, phase of moon etc.) and if you want development speed you want python3.13. [NB check what version of python is running on the server. It was 3.11.2 in December 2024.].

We used to have lengthy instructions fo installing python using "deadsnakes" but since December 2024 we just use the uv utility for everything.

WSL1 Historic problems

WSL2 is now noticeably even slower for NTFS files mounted as /mnt/c or /mnt/d, so now it is only really useable if you move all the troggle code into your Linux filespace on \\wsl.localhost\Ubuntu-22.04\home which means that those files will only be accesible from Linux. Buggeration.

WSL1 and WSL2

WSL now installs as WSL2 by default, but older machines (mostly laptops) may not have the Hyper-V Virtualization hardware and may have to run WSL1. This may be fine: the behaviour is nearly identical so far as an expo laptop is concerned, where you just want to use rsync and scp. An old 2011-era PC had been doing troggle development happily using WSL1. However time is clearly running out for WSL1, and the file handling is now very noticably slower. If your laptop won't run WSL2 then you are advised to get rid of Windows completely and install Xubuntu as a dual boot system.

For some laptops though, WSL1 does not install properly and you have to use WSL2. The symptom is two error messages when you run dmesg in the Linux bash terminal window. However, if all you want to use it for is rsync, this may still be fine.

WSL2 - things to watch for

On some Windows 10 laptops WSL1 fails to install properly and you have to use WSL2.

You should check that the configuration files do what you want.

When running a full troggle development laptop (see separate troggle documentation) you will want to use python venv. This barfs untidily if you have the code on NTFS, e.g. if mounted on /mnt/c/. You almost certainly have to move all your code to the internal network share e.g. \\wsl$\Ubuntu\home\expo\troggle\ which is how it looks when you are browing from Windows. (No, don't try to be cute and keep it on /mnt/c/ and just put a soft link in /home/expo/. That doesn't work either.) This means that the code is actually living on a Linux ext4 filesystem hidden away on your disc where you can only see it using the 'network' method \\wsl$\Ubuntu-20.04\home\expo\troggle\. This means that file access is noticably faster too. You will need to make sure that your backup/archive methods access this filesystem though.

NB rsync doesn't work with NTFS partitions the way that WSL1 does.

Read WSL & Visual Studio Code and go back and read the bits about VS Code running remotely in the ※Windows data maintenance laptop page.

WSL1 gotchas and tips

WSL using files on NTFS unfortunately introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file permissions that are fundamentally different things and are never in sync in any sense. All sorts of fun results. The alternative is to store all your data in the \\wsl$\ ext4 filesystem (WSL2 only, not WSL1).

Permissions, permissions..

Having happily used WSL1 in this manner for a couple of years, it was a rude awakening to try this with a new laptop with a small hard-drive, so all the expo code was re-mounted on a plugged-in SD card. The problem of permissions that seemed to be a WSL2 issue reappeared with a vengence on WSL1.

The recommended WSL procedure (Jan.2021) said to mount the drive using a new 'metadata' setting:

sudo umount /mnt/d
sudo mount -t drvfs D: /mnt/d -o metadata,uid=1000,gid=1000,umask=22,fmask=111

This is probably why there is a difference between automatically-mounted drives (e.g. C:, /mnt/c) and plug-in drives: "By default, WSL will set the uid and gid to the default user with drives that are auto-mounted during instance start. If you mount manually, you will have to set these explicitly (the default user that gets created when WSL is first installed has a uid=1000 and gid=1000)."

And of course you will need to arrange that this happens automatically whenever you start WSL, so you will need to set /etc/fstab, so ensure that the relevant line says:

D:           /mnt/d         drvfs   metadata,uid=1000,gid=1000,umask=22,fmask=11     0    0

See also File Permissions for WSL (Dec. 2021).

"Are you feeling lucky, punk"

When things go wrong when trying to sort this out, you may find these pages useful. I did. rsync - failed to set permission - operation not permitted file-system-improvements-to-the-windows-subsystem-for-linux/ id-rsa-pub-file-ssh-error-invalid-format key-load-public-invalid-format ssh-error-loading-key-id-rsa-invalid-format chmod-chown-wsl-improvements mount-usb-drives-windows-subsystem-for-linux ssh command line


Go back to: ※ Basic laptop
Go back to: ⁂ Survey laptop
Go back to: ☯ Windows Bulk Update laptop
Go back to: ⚒ Troggle development laptop