expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-25 16:52:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

Revised all WSL documentaiton

Browse Source
This commit is contained in:
Philip Sargent 2022-12-22 15:10:14 +00:00
parent 01a821002e
commit 1be6e5cbfe
4 changed files with 213 additions and 176 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
handbook/computing/bulkupdatelaptop.html
View File

@ -8,7 +8,7 @@
<body>
<h2 id="tophead">CUCC Expedition Handbook - Bulk Update laptop</h2>
<h1>Cave data - Bulk Updates</h1>
<h1>Cave data - Bulk Update Laptop</h1>
<p>You will already have explored what your laptop can do in the
<a href="basiclaptop.html">basic Expo laptop</a> guide,
@ -62,11 +62,7 @@ raw survey notes (surveyscans), scanned cave centre-line plots and everything el
<p>You need to <a href="keyexchange.html">register a key with the expo server</a> to get upload (i.e. read/write) access. Do this first, Without it none of git, scp, ftp or rsync will work.
You can do this entirely on your own if you have access to the <i>expo laptop</i> to upload and install the public key generated by your laptop.</p>
<h4>QMs and scripts</h4>
<p>You may not need a full troggle software development machine if you are only fixing a small script. If you are, you need
<ul>
<li>python - maybe - but only if you are working on the <a href="../troggle/scriptsother.html">scripts in expoweb/noinfo/</a> such as <var>make-folklist.py</var> or some of the <a href="../troggle/scriptsqms.html">standalone QM scripts</a> - but not on troggle itself - then please use the as close to the same setup as you can to <a href="../troggle/troglaptop.html">the setup we use for troggle</a>. <img src="https://www.python.org/favicon.ico" width=64 hspace="20" align="right">
</ul>
<h3>Your own Bulk Update laptop</h3>
@ -99,6 +95,29 @@ You can do this entirely on your own if you have access to the <i>expo laptop</i
<p>Read the <a href="winlaptop.html">Detailed Windows Configuration Instructions</a> for foconfiguring a Windows Bulk Update machine.
<h4>QMs and scripts</h4>
<p>You may not need a full troggle software development machine if you are only fixing a small script. If you are, you need
<ul>
<li>python - maybe - but only if you are working on the <a href="../troggle/scriptsother.html">scripts in expoweb/noinfo/</a> such as <var>make-folklist.py</var> or some of the <a href="../troggle/scriptsqms.html">standalone QM scripts</a> - but not on troggle itself - then please use the as close to the same setup as you can to <a href="../troggle/troglaptop.html">the setup we use for troggle</a>. <img src="https://www.python.org/favicon.ico" width=64 hspace="20" align="right">
</ul>
<h3>The Expo Laptop(s)</h3>
<p>[These notes are to remind nerds when configuring or updating a replacement <var>expo laptop</var>.]
<p>The <var>expo laptop</var> is configured as a "Bulk Update" laptop (including al the "Survey laptop" tools), but has a few extras.
<ul>
<li>Create a user 'expo'
<li>Add user 'expo' to <a href="https://www.how2shout.com/linux/2-ways-to-add-users-to-sudoers-group-in-debian-11/">the sudoers list</a> or alternatively <a href="https://vitux.com/how-to-make-a-user-an-administrator-in-debian/">use the gui</a> to do the same thing.
<li>Log in as user 'expo' again (so that ~/ = /home/expo/)
<li>Set up the expo Gmail account in Firefox
<li>Install Firezilla FTP software and <a href="fzconfig.html">download zilla-uploads.xml</a> to configure it
<li>Copy ~/.ssh/* from the previous install or from the other <var>expo laptop</var>
<li>Copy ~/expofiles from from the previous install or from the other <var>expo laptop</var>
<li>Run <a href="qstart-rsync.html">rsync</a> to ensure /expofiles is correctly identical to the server
<li>Configure git as user 'expo laptop' (or 'azirophale' or 'crowley') - see below, with expo Gmail address
<li><a href="qstart-git.html">git clone</a> the four repositories
<li>Check that you have a GPS trck editor installed, e.g. GPSprune
</ul>
<h2 id="git">Configuring git</h2>
<p>On a new machine you need to configure your git identity:

247
handbook/computing/winlaptop.html
View File

@ -14,47 +14,55 @@
<br>
<a href="basiclaptop.html">Setting up a basic Expo laptop</a>
<br>
<a href="surveylaptop.html">Setting up a machine for Expo survey processing</a>
<a href="surveylaptop.html">Setting up a 'survey laptop' for Expo survey processing</a>
</p>
<p>Unlike a Linux machine, on Windows there are two distinct levels of 'bulk update laptop' capability:
<ol style="list-style-type:decimal-leading-zero; line-height:130%">
<li> <a href="#keys">Keys and key management</a>, simple file transfer, <em>scp, ftp</em>
<li> <a href="#rsync">Synchronisation of entire file heirarchies</a>: installation and use of <em>rsync</em>
</ol>
<h2 id="keys">Level 1 - Windows Bulk Update</h2>
<ul>
<li><a href="#well">Things that already work well with a Windows laptop</a><br />
<li><a href="#well">Things that already work well with any Windows laptop</a><br />
- Editing the handbook webpages, typing up SVX files and transcribing the logbook, <br />
- Anything where the file-transfer to the expo server is via the git version control software
<li><a href="#problems">Things that cause problems</a><br />
- filenames and unintentional duplication because links are not understood by Windows,<br />
- sFTP or scp for more than a handful of files
<li><a href="#hard">Things that are really, really hard</a><br />
- using rsync <br />
- large-scale updating of several folders at once on expofiles without overwriting other people's work (which means using rsync)<br />
- well not "hard" exactly, but complicated with lot of steps that are easy to get wrong and with poor feedback as to whether you have done each step correctly.
<li><a href="#bold">The easier way to do it</a><br />
- using WSL: Windows System for Linux
<li>See our own expo page about <a href="../computing/wsllaptop.html">setting up a WSL Expo Laptop</a>
<li><a href="#problems">Things that are really hard</a><br />
- large-scale updating of several folders at once on expofiles without overwriting other people's work <br />
- well not "hard" exactly, but complicated with lot of steps that are easy to get wrong and with poor feedback as to whether you have done each step correctly.
<br>- That is what you ned a <a href="#rsync">Level 2 machine</a> for.
</ul>
<li><a href="bulkupdatelaptop.html#win">Possibly an even easier way to do it</a><br />
- using OpenSSH for Windows</ul>
<h3>Software</h3>
<ul>
<li><a href="../putty/putty.html">PuTTY</a>.You need this to generate and to use ssh keys on Windows. Otherwise none of git, scp, ftp or rsync will work. It includes command line tools ssh, scp
<li>Key management, either PuTTY or OpenSSH. You need one of these to generate and to use ssh keys on Windows. Otherwise none of git, scp, ftp (or rsync) will work.
<ul>
<li><a href="https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview">OpenSSH for Windows</a> is new (fully working since 2021). <br>
- Use the Microsoft documented install method which puts the executables in <var>windows\system32\openssh\</var>.
<br>
- This is new we are still updating other troggle documentation to refer to it
<br>
- We haven't got documentation for using this with Filezilla yet. Please write it if you do this.
<br> It integrates well with Github for Windows. <br>
<li><a href="../putty/putty.html">PuTTY</a> It includes command line tools ssh, scp
(pscp) and sFTP (psftp). The <a href="https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html">PuTTY</a> installation includes puttygen and pagent which you also need. Version 0.77 was released on 2022-05-27.
<li><a href="https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview">OpenSSH for Windows</a> is new since 2018. It is a
set of command line tools which are an alternative to PuTTY. It also includes scp and sftp. It integrates well with Github for Windows and enables you to
use rsync over ssh. NB use this Microsoft documented install method which puts the executables in <var>windows\system32\openssh\</var>.
[This is so new that none of the other troggle documentation refers to it: all those other pages need to be rewritten to use this
instead of or as an alternative to PuTty/Pageant.]
<li><a href="https://desktop.github.com/">GitHub Desktop for Windows</a> - yes this works with the expo server git even though we don't use GitHub itself.
</ul>
<li><a href="https://desktop.github.com/">GitHub Desktop for Windows</a> - yes this works with the expo server git even though we don't use GitHub itself for expo software.
<li><a href="https://code.visualstudio.com/">VS Code</a> is a free (but not FOSS) editor with in-built git capability and plug-ins ("Git History",
"GitLens")which render git branches graphically. Also available for Linux.
"GitLens") which render git branches graphically. Also available for Linux.
<li><a href="https://gitforwindows.org/">Git for Windows</a> - not as simple to use as it looks
<li><a href="https://www.gitkraken.com/">GitKraken</a> - Very pretty GUI interface to git, also Linux version.
Times-out for our server unless you buy it but free for university people.
<li><a href="https://winscp.net/eng/download.php">WinScp</a> can be used as an alternative to Filezilla if you like (like Filezilla, it uses PuTTY ssh keys)
<li><a href="https://winscp.net/eng/download.php">WinScp</a> can be used as an alternative to Filezilla if you like (like Filezilla, it uses PuTTY ssh keys). It has a <a href="https://winscp.net/eng/docs/screenshots">GUI</a> similar to Filezilla
<li><a href="https://www.java.com/en/">java</a> - needed for GPSprune and CaveConverter. Has to be installed separately on a Windows machine.
<li><a href="https://www.java.com/en/">java</a> - needed for GPSprune and CaveConverter. Has to be <a href="https://www.java.com/en/download/windows_manual.jsp?host=java.com&locale=en-GB">installed separately</a> on a Windows machine.
<li><a href="https://notepad-plus-plus.org/">Notepad++</a> or any other syntax-highlighting code editor for HTML and python. We have a syntax-highlighter
to colourize .svx files, download it: <a href="/site_media/survex.xml">survex.xml</a> (no dark-mode yet though).
@ -74,14 +82,15 @@ to colourize .svx files, download it: <a href="/site_media/survex.xml">survex.xm
<h2><a name="configuration">Configuration</a></h2>
<p>Brendan wrote a guide to using puTTY and git for expo on a Windows machine. It's worth reading and it has lots of screenshots: <a href="/expofiles/documents/idiots-guide-expo-git.pdf">Idiots guide to accessing expo git.pdf</a>.
<p><a href="/expofiles/documents/Idiots guide to accessing expo git.pdf">Idiots guide to setting up git for expo</a>
- PDF - Brendan's guide. Uses PuTTy and GitKraken on WIndows.
<p>You need to <a href="keyexchange.html">register a key with the expo server</a> to get upload (i.e. read/write) access. Do this first, Without it none of git, scp, ftp or rsync will work.
<h4>Key management and configuration</h4>
<p>You need to <a href="keyexchange.html">register a key with the expo server</a> to get upload (i.e. read/write) access. Do this first, Without it none of git, scp, ftp (or <a href="#rsync">rsync</a>) will work.
You can do this entirely on your own if you have access to the <i>expo laptop</i> to upload and install the public key generated by your laptop.</p>
<p>On a Windows machine you will need to configure pageant (the putty authentication agent)
<p>On a Windows machine with PuTTY you will need to configure pageant (the puTTY authentication agent)
to <a href="https://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html">run at startup to load your key</a>. Note that you are loading your <em>private</em> key, the .ppk file, into pageant and that this key never leaves your laptop.</p>
@ -89,31 +98,39 @@ to <a href="https://blog.shvetsov.com/2010/03/making-pageant-automatically-load-
<ul>
<li><a href="../putty/putty.html">Installing PuTTy on Windows</a>.</li>
<li><a href="../putty/putty.html">Installing puTTY on Windows</a>.</li>
<li>Read the online instructions about using <a href="https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement">OpenSSH key management</a> (several webpages). (Then write them up and edit this handbook to be more useful.)
<li><a href="fzconfig.html">Installing Filezilla</a>.</li>
</ul>
<p>The above gets the command-line PuTTY tools (ssd, sftp, pscp) running, but doesn't get rsync working. You might like to try <a href="https://stackoverflow.com/questions/23517023/rsync-from-windows-to-linux-using-puttys-pagent-authentication">this</a> (untested).</p>
<p>The above gets the command-line puTTY tools (ssd, sftp, pscp) running.
<p>When using Windows please, please be <a href="http://expo.survex.com/handbook/survey/getin.htm#filenames">excessively careful when naming files and survex names</a> and be <a href="qstart-rsync.html">exceptionally careful when using rsync</a>.
<p>Most Windows software that we recommend "just works" if you have set up puTTY and have done the <a href="keyexchange.html">key-pair setup</a> and are running a local ssh agent (pagent) automatically at boot up on your laptop.
<p>Some software (such as Filezilla) defaults to using pageant and it "just works".
<h3 id="well">Things that already work well</h3>
<p>Anything where the file upload and download is done via the verson control client software works really well.
<p>When using Windows please, please be <a href="http://expo.survex.com/handbook/survey/getin.htm#filenames">excessively careful when naming files and survex names</a>.
<h4>Configuring Filezilla or equivalents</h4>
<ul>
<li><a href="fzconfig.html">Installing Filezilla</a>.</li>
</ul>
<p>Filezilla defaults to using pageant and it "just works" if you used PuTTy.
<h4>Configuring git</h4>
<p><a href="/expofiles/documents/Idiots guide to accessing expo git.pdf">Idiots guide to setting up git for expo</a>
- PDF - Brendan's guide. Uses puTTY and GitKraken on WIndows. Read this, but most people will need to use VS Code instead of Git Kraken.
<h4>Key management using PuTTy</h4>
<p>Most Windows software that we recommend "just works" if you have set up PuTTy and have done the <a href="keyexchange.html">key-pair setup</a> and are running a local ssh agent (pagent) automatically at boot up on your laptop.
<p>Some software, such as the commercial (but free) GitKraken, requires that you click a checkbox to say that you are "using local SSH agent" rather than specifying ssh private keys explicitly (File->Preferences->Authentication in GitKraken).
<p>Some software (such as Filezilla) defaults to using the local agent and it "just works".
<p>Brendan wrote a guide to using putty and git for expo on a Windows machine. It's worth reading and it has lots of screenshots: <a href="/expofiles/documents/idiots-guide-expo-git.pdf">Idiots guide to accessing expo git.pdf</a>.
<p>Read the online instructions about using <a href="https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview">OpenSSH in Windows</a> as an alternative. (Then write them up and edit this handbook to be more useful.)
<h3 id="problems">Things that cause problems</h3>
<h4>Filenames</h4>
<p>Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "<a href="https://answers.microsoft.com/en-us/windows/forum/windows_7-files/why-cant-i-create-a-folder-with-name/23c86662-4988-4c7d-9c2d-3e33d4413de3">CON</a>") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware.
<h4>Symbolic links and sFTP</h4>
<h4>Symbolic links and file-transfer</h4>
<p>Linux people like to use <em>links</em>. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks.
<p>
The links you are most likely to come across are that what looks like
@ -128,7 +145,7 @@ is a link to the file <span style="font-family:monospace; size=x-small; backgrou
There are two types of linux links: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux hard links. Fortunately we don't <em>seem</em> to have any hard links anywhere.
<p>What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this,as it pftp (PuTTy) working in eith sFTP or scp mode.. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.
<p>What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this,as it pftp (puTTY) working in eith sFTP or scp mode.. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.
<p>So the ordinary user won't notice any problems, but the nerds behind the scenes start to cuss and shout and generally carry-on in an expletive-heavy manner.
@ -139,10 +156,10 @@ There are two types of linux links: hard links and symbolic links. Symbolic link
<li>When using the git version control systems the download of a link works fine. But be careful not to edit the link file downloaded (it is just a text file holding the path of the file holding the actual contents) because then the version control client would upload it to the server and overwrite the link on the server with something that isn't a link. It also won't work as a Windows shortcut, but at least the default behaviour isn't actively dangerous.
<li>When using sFTP, manually check whether any files you are copying from the server are links - look at the symbol in Filezilla.
<li>Be careful not to copy any links using sFTP and instead recreate them manually on the Windows filesystem using right-click "Create shortcut".
<li>You will have to find out what to make the shortcut link to by logging in to the server (using a PuTTy ssh logon) and doing <span style="font-family:monospace; size=x-small; background-color: lightgray">"ls -l"</span> in the folder where the link is.
<li>You will have to find out what to make the shortcut link to by logging in to the server (using a puTTY ssh logon) and doing <span style="font-family:monospace; size=x-small; background-color: lightgray">"ls -l"</span> in the folder where the link is.
<li>But later, when re-uploading edited files from Windows to the server, Filezilla will see the Windows shortcut as a ".lnk" file which it will upload, but which will mean nothing to the linux server receiving it. So you would have to manually recreate the symbolic link by logging into the server using ssh and using the "ln -s" command. Yuk.
<br />
<li>Even if you use scp instead of sFTP, it does the same stupid thing when copying from a linux filesystem to a Windows filesystem. The PuTTy package includes <span style="font-family:monospace; size=x-small; background-color: lightgray">pscp.exe</span> but even if you force it to use the scp protocol like this:
<li>Even if you use scp instead of sFTP, it does the same stupid thing when copying from a linux filesystem to a Windows filesystem. The puTTY package includes <span style="font-family:monospace; size=x-small; background-color: lightgray">pscp.exe</span> but even if you force it to use the scp protocol like this:
<tt>
pscp -scp expo@expo.survex.com:expoweb/essentials.gpx .
</tt>
@ -150,137 +167,55 @@ it downloads a <em>copy</em> of the contents of essentials.gpx and not a link.
<li>A possible fix in the future might be to keep all your expo files in a separate partition of your hard disc which is formatted with a linux filesystem (such as ext4) and run the <a href="https://www.ext2fsd.com/">ext2ntfs</a> driver to mount this fielsystem read-write from Windows. Probably not a good idea as the driver is a bit flaky in read-write mode and you could lose everything.
</ul>
<h3 id="hard">Things that are really quite involved</h3>
<p>The core problem is integrating the PuTTy key management software (pagent.exe) with a terminal window. We need a terminal window to run rsync as
none of the packaged software (Filezilla, PuTTy) includes an rsync client.
<p>[Well, strickly speaking we need a Windows rsync executable, and this does exist, and a command-line ssh, which the 'OpenSSH in Windows' system
provides. But we haven't properly explored this set of mechanisms yet.]
<p>The solution we have now for rsync is to use WSL and to create another key, distinct from the PuTTy one, and to upload that key to the expo
server. Because this is treating WSL as if it were a different machine requiring its own key quite separate from the Windows key. (This works the
same way in WSL1 and WSL2 - but see "Do you feel lucky, punk" below...)
<p>[We are not the only people to find this <a href="https://www.ubackup.com/windows-10/rsync-windows-10-1021.html">nasty and irritating</a>. Other
hacks are to use the obsolescent cygwin rsync or the rsync packaged within bash within more recent versions of 'Git for Windows' (which is built on <a href="https://www.msys2.org/docs/what-is-msys2/">MYSYS2</a>.
If you are already familiar with any of these, then use them and not WSL.]
<h4>WSL - Windows Subsystem for Linux</h4>
<p>[On 23 November 2022 a new way of installing and using WSL was announced, see <a href="https://www.theregister.com/2022/11/23/wsl_microsoft_store_default_version/">
Windows Subsystem for Linux now packaged as a Microsoft Store app</a> so all documentation on this expo site needs to be revised.]
<p>So on a machine with WSL enabled, create an ordinary cmd window and get into the WSL environment using the wsl command:<br />
<span style="font-family:monospace; size=x-small; background-color: lightgray">
D:\CUCC-Expo\expoweb\ <font color=red>wsl</font>
</span>
<br />
which puts you into the WSL environment with a new command prompt, e.g.<br />
<span style="font-family:monospace; size=x-small; background-color: lightgray">
<span style="color: green">
philip@Muscogee</span><span style="color: yellow">:/mnt/c/Users/Philip</span>$
</span>
<p>and now you can setup keys in the same way as you would on a Linux machine using ssh-keygen:
<pre>
<tt>$ <b>ssh-keygen -C "philip@muscogee-wsl"</b>
Generating public/private rsa key pair.
Enter file in which to save the key (/home/philip/.ssh/id_rsa): <b>id_rsa_wsl</b>
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in <em>id_rsa_wsl</em>.
Your public key has been saved in <em>id_rsa_wsl.pub</em>.
The key fingerprint is:
SHA256:ySs0YD5IG2ZD50+riUDHWosNq+WJdqpkDlINXh709r0 philip@muscogee-wsl
The key's randomart image is:
+---[RSA 2048]----+
| . o |
| ..+ . |
| oB+* + |
|.=O%.* + o |
|.+*o= = S . |
|.* o = . . . |
|=++.o . . E |
|B o . |
|oo |
+----[SHA256]-----+
$
</tt>
</pre>
The generated key is in the current directory and you need to move them to ~/.ssh/ as is standard on Linux (which is not at all the same place that PuTTy uses to keep keys on Windows).
<p>Now you have to complete the <a href="keyexchange.html">key-pair setup</a> with the new key "id_ras_wsl.pub". But you don't need anyone else's help this time as you can use PuTTy to ssh into the server and copy your key to the right place yourself: it is a <a href="keyexchange.html#secondmachine">"Second Machine"</a>.
<p>
Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as:
<pre>
<tt>rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles</tt></pre>
<p>But this only works with WSL1 not WSL2 ! Under WSL1 it doesn't matter if your /expofiles/ is mounted on the Linux filesystem,
typically under /home/expo/expofiles, or whether it it in the NTFS filesystem mount for Linux as,
e.g. /mnt/d/expofiles.
Under WSL2 /mnt/d/expofiles fails, either with a permissions problem or, if you try sudo rsync.. ,
with a ssh authentication failure (why? Given that we explicitly sate that we want to be user expo@ ?). Ghastly anyway. See next section, where the
same problem appears on WSL1.
<p>All of which is really a bit ridiculous if all you want to do is to use rsync on a Windows machine to sort out some photos.
<h3 id="permissions">Permissions, permissions..</h3>
<p>Having happily used WSL1 in this manner for a couple of years, it was a rude awakening to try this with a new laptop with a small
hard-drive, so all the expo code was mounted on a plugged-in SD card. The problem of permissions that seemed to be a WSL2 issue reappeared with
a vengence on WSL1.
<p>The recommended WSL procedure (Jan.2018) said to mount the drive using
<a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">a new 'metadata' setting</a>:
<pre><code>sudo umount /mnt/d
sudo mount -t drvfs D: /mnt/d -o metadata,uid=1000,gid=1000,umask=22,fmask=111</code></pre>
<p>This is probably why there is a difference between automatically-mounted drives (e.g. C:, /mnt/c) and plug-in drives:
"<em>By default, WSL will set the uid and gid to the default user with drives that are auto-mounted during instance start. If you mount manually, you will
have to set these explicitly (the default user that gets created when WSL is first installed has a uid=1000 and gid=1000).</em>"
<p>And of course you will need to arrange that this happens automatically whenever you start WSL, so you will need to set
<a href="https://help.ubuntu.com/community/Fstab"><var>/etc/fstab</var></a>, so ensure that the relevant line says:
<pre><code>D: /mnt/d drvfs metadata,uid=1000,gid=1000,umask=22,fmask=11 0 0</code></pre>
<p>See also <a href="https://docs.microsoft.com/en-us/windows/wsl/file-permissions">File Permissions for WSL</a> (Dec. 2021).
<h2 id="rsync">Level 2 - Windows Bulk Synchronisation with rsync</h2>
<h3 id="bold">"Are you feeling lucky, punk"</h3>
<p>So here is the current wild frontier. Currently these are the ways to get a terminal window which you might expect to work:
<p>When using Windows please, please
<ul>
<li>cmd window - the old faithful going all the way back to MS-DOS. But no path to an rsync.exe executable as standard.
<li>PowerShell terminal window - nope, no rsync.
<li>bash window - installed by default when you install <A href="https://gitforwindows.org/">gitforwindows</a>. Unfortunately while this <a
href="https://en.wikipedia.org/wiki/MinGW">MINGW32</a> setup includes a command-line git executable it doesn't include rsync. (Well you have
<li>be <a href="../survey/getin.htm#filenames">excessively careful when naming files and survex names</a> and
<li>be <a href="qstart-rsync.html">exceptionally careful when using rsync</a>.
</ul>
<p><a href="https://rsync.samba.org/">rsync</a> is a Linux command-line application, so we need a terminal window to run it and a Windows client executable.
None of the packaged software (Filezilla, puTTY) includes a Windows rsync client.
<p>Windows comes with a terminal window (cmd.exe) so we 'just' need a native Windows rsync executable <var>rsync.exe</var>.
<ul>
<li>The only direct port of rsync to Windows is Cygwin: <a href="https://stackoverflow.com/questions/23517023/rsync-from-windows-to-linux-using-puttys-pagent-authentication">this</a> . <br>- You just need the following files: rsync.exe, cygwin1.dll and cygz.dll .
Read the <a href="https://ccm.net/computing/windows/3877-rsync-under-windows/">instructions</a><br>
- You will also have to install OpenSSH and configure the Cygwin system to use the OpenSSH ssh key.
<li>bash window - a different terminal window. This is installed by default when you install <A href="https://gitforwindows.org/">gitforwindows</a>. Unfortunately while this <a
href="https://en.wikipedia.org/wiki/MinGW">MINGW32</a> setup includes a command-line git executable it doesn't include rsync: you have
to excavate it yourself, see <a href="https://prasaz.medium.com/add-rsync-to-windows-git-bash-f42736bae1b3">a 2020 script</a>, alternatively <a
href="https://blog.tiger-workshop.com/add-rsync-to-git-bash-for-windows/">a 2017 recipe</a> or <a
href="https://gist.github.com/radleta/0b337a2b14f761951cf2aab0578512b9">recent experiments</a>.
This technique seems too fragile for us to bother with, but you might be lucky.)
<li>cygwin - a cmd terminal where you have downloaded and installed <a href="https://www.cygwin.com/">Cygwin</a>. Yes, if you have selected the
rsync package you will be able to run the rysnc executable, but it won't have access to the cyptographic key so it can't connect to the expo server (but
you can make it work by using it with ssh from OpenSSH).
Please feel free to work out how to make this work. A more recent, graphical variant is <a
href="https://hackaday.com/2017/03/29/swan-better-linux-on-windows/">Swan</a>.
<li><img src="wsl.jpg" align="right" hspace="10"><a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux">Windows Subsystem for Linux</a> aka
WSL available on all Windows10 machines since November 2019.
<li>Or you can use WSL: Windows System for Linux<img src="wsl.jpg" align="right" hspace="10"><a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux"></a>
<ul>
<li>[On 23 November 2022 a new way of installing and using WSL was announced, see <a href="https://www.theregister.com/2022/11/23/wsl_microsoft_store_default_version/">
Windows Subsystem for Linux now packaged as a Microsoft Store app</a> so all documentation on this expo site needs to be revised.]
<li>WSL <a href="https://docs.microsoft.com/en-us/windows/wsl/install">Install it like this</a>. This does what we need. This works using a key generated
by its own version of ssh-keygen if you follow the instructions above about putting it in the right place.
<li>WSL: <a href="https://hackaday.com/2019/12/23/linux-fu-wsl-tricks-blur-the-windows-linux-line/"> Converting Windows paths to Linux paths and
vice-versa</a>.
<li>In WSL you can choose whether to store the files in the WSL view of NTFS, e.g. /mnt/c/expo/ or in an ext4 filesystem e.g.
\\wsl$\Ubuntu-20.04\home\expo . Our experience is with WSL1 with the files on NFTS e.g. /mnt/d/expo/ .
<li>Not all machines are compatible with WSL1 though, see <a href="wsllaptop.html">WSL in more depth</a> and WSL2 differences.
Even if you have the hardware, you may have to enable it in
your BIOS and install the Windows Feature to do it. Instructions <a href="https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10">here</a>
and <a href="https://www.windowscentral.com/how-install-wsl2-windows-10">here</a>.
<li> This effectively installs a whole copy of Linux
<li>Unfortunately this does not integrate with PuTTY <em>at all</em>
<li>You will need to do the key management setup <em>again</em>, within the WSL environment.
<li> You want WSL1 not WSL2. WSL1 is 20x faster for Windows NTFS filesystems.
<li> WSL on anything other than drive C:\ may involve you with <a href="wsllaptop.html#bold">hard to diagnose file permissions problems</a>
<li>See our own expo page about <a href="../computing/wsllaptop.html">setting up a WSL Expo Laptop</a>.
</ul>
<li>A full <a href="https://www.brianlinkletter.com/installing-debian-linux-in-a-virtualbox-virtual-machine/">virtual Linux machine</a> running
using a hypervisor such as <a href="https://www.virtualbox.org/">VirtualBox</a> which has its own virtual Linux filesystem. This setup has the
advantage that you don't have to partitition your hard drive but the disadvantage that you may not have easy access to the files from Windows (a feature
it shares with WSL2).
<li><a href="https://github.com/canonical/multipass">Canonical Multipass</a> - a VM alternative (Ubuntu only).
disadvantage that you may not have easy access to the files from Windows.
</ul>
<p>If you just want rsync, then please try the Cygwin method first.
If you struggle with Cygwin, then you are unlikely to get any of the other methods to work either. So use Filezilla or WinScp.
<p>If you don't like Cygwin, and want to use a bash terminal environment instead of Windows cmd.exe, then try the MINGW32 method.
<p>Only install WSL if you are pretty sure that you want to use WSL for things other than rsync.
<hr />
Go back to: <a href="basiclaptop.html">Basic laptop</a><br />
Go back to: <a href="surveylaptop.html">Survey laptop</a><br />
Go on to: <a href="wsllaptop.html">Windows WSL laptop</a><br />
<hr /></body>
</html>

98
handbook/computing/wsllaptop.html
View File

@ -20,7 +20,8 @@
<br>
<a href="winlaptop.html">Setting up a Windows machine for expo bulk data rearrangements</a>
</p>
On 23 November 2022 a new way of installing and using WSL was announced, see <a href="https://www.theregister.com/2022/11/23/wsl_microsoft_store_default_version/">
Windows Subsystem for Linux now packaged as a Microsoft Store app</a> so some of the documentation on this expo site needs to be revised. WSL2 is now noticeably even slower for NTFS files.
<h3>WSL1 and WSL2</h3>
<p>WSL now installs as WSL2 by default, but older machines (mostly laptops) may not have the Hyper-V Virtualization hardware and may have to run WSL1.
This is fine: the behaviour is nearly identical so far as an expo laptop is concerned, where you just want to use rsync and scp. An old 2011-era PC has
@ -33,13 +34,13 @@ the Linux bash terminal window. However, if all you want to use it for is rsync,
<p>On some Windows 10 laptops WSL1 fails to install properly and you have to use WSL2.
<p>When running a full troggle development laptop (see separate <a href="../troggle/troglaptop.html">troggle documentation</a>)
you will want to use <var>python venv</var>. This barfs untidily if you have the code on NTFS, e.g. if mounted on <var>/mnt/c/</var>. You almost
certainly have to move all your code to the internal network share e.g. <var>\\wsl$\Ubuntu-20.04\home\expo\troggle\</var> which
certainly have to move all your code to the internal network share e.g. <var>\\wsl$\Ubuntu\home\expo\troggle\</var> which
is how it looks when you are browing from Windows.
(No, don't try to be cute and keep it on <var>/mnt/c/</var> and just put a soft link in <var>/home/expo/</var>. That doesn't work either.)
This means that the code is actually living on a Linux <var>ext4</var> filesystem hidden away on your disc where you can only see it using the
'network' method <var>\\wsl$\Ubuntu-20.04\home\expo\troggle\</var>. This means that file access is somewhat faster too but you probably won't notice.
'network' method <var>\\wsl$\Ubuntu-20.04\home\expo\troggle\</var>. This means that file access is somewhat faster too but you probably won't notice. You will need to make sure that your backup/archive methods access this filesystem though.
<p>rsync doesn't work with NTFS partitions the way that WSL1 does. See <winlaptop.html>Windows laptop</a> for a bit of detail. [Work in progress.]
<p>rsync doesn't work with NTFS partitions the way that WSL1 does.
<p>Read <a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">WSL & Visual Studio Code</a> and go back and read the bits about VS Code
running remotely in the &#8251;Windows data maintenance laptop page.
@ -54,9 +55,89 @@ href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">
in the \\wsl$\ ext4 filesystem (WSL2 only, not WSL1).
<p>If you are disturbed by the instructions to produce an entirely different key for WSL to use when your PC already has a perfectly good PuTTy key installed on the server, then you are right. It is inelegant. But it works, the instructions are shorter and there are fewer things that go wrong. If you are terribly offended by that then you can set your PC up to use one key shared between WSL and normal-Windows as described in <a href="https://devblogs.microsoft.com/commandline/sharing-ssh-keys-between-windows-and-wsl-2/">this October 2019 article</a>. (Don't set up a password on the key because then you don't need to install keychain.) But beware, this sort of thing goes out of date quite rapidly.
<h4>WSL - Creating another ssh key</h4>
<p>If you have PuTTY installed and working, but you now want to use WSL rsync, you need to set up a key again within the WSL environment. So on a machine with WSL enabled, create an ordinary cmd window and get into the WSL environment using the wsl command:<br />
<span style="font-family:monospace; size=x-small; background-color: lightgray">
D:\CUCC-Expo\expoweb\ <font color=red>wsl</font>
</span>
<br />
which puts you into the WSL environment with a new command prompt, e.g.<br />
<span style="font-family:monospace; size=x-small; background-color: lightgray">
<span style="color: green">
philip@Muscogee</span><span style="color: yellow">:/mnt/c/Users/Philip</span>$
</span>
<p>and now you can setup keys in the same way as you would on a Linux machine using ssh-keygen:
<pre>
<tt>$ <b>ssh-keygen -C "philip@muscogee-wsl"</b>
Generating public/private rsa key pair.
Enter file in which to save the key (/home/philip/.ssh/id_rsa): <b>id_rsa_wsl</b>
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in <em>id_rsa_wsl</em>.
Your public key has been saved in <em>id_rsa_wsl.pub</em>.
The key fingerprint is:
SHA256:ySs0YD5IG2ZD50+riUDHWosNq+WJdqpkDlINXh709r0 philip@muscogee-wsl
The key's randomart image is:
+---[RSA 2048]----+
| . o |
| ..+ . |
| oB+* + |
|.=O%.* + o |
|.+*o= = S . |
|.* o = . . . |
|=++.o . . E |
|B o . |
|oo |
+----[SHA256]-----+
$
</tt>
</pre>
The generated key is in the current directory and you need to move them to ~/.ssh/ as is standard on Linux (which is not at all the same place that puTTY uses to keep keys on Windows).
<p>Now you have to complete the <a href="keyexchange.html">key-pair setup</a> with the new key "id_ras_wsl.pub". But you don't need anyone else's help this time as you can use puTTY to ssh into the server and copy your key to the right place yourself: it is a <a href="keyexchange.html#secondmachine">"Second Machine"</a>.
<p>
Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as:
<pre>
<tt>rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles</tt></pre>
<p>But this only works with WSL1 not WSL2 ! Under WSL1 it doesn't matter if your /expofiles/ is mounted on the Linux filesystem,
typically under /home/expo/expofiles, or whether it it in the NTFS filesystem mount for Linux as,
e.g. /mnt/d/expofiles.
but under WSL2 rsync to /mnt/d/expofiles fails, either with a permissions problem or, if you try sudo rsync.. ,
with a ssh authentication failure (why? Given that we explicitly state that we want to be user expo@ ?). Ghastly anyway. See next section, where the
same problem appears on WSL1.
<p>All of which is really a bit ridiculous if all you want to do is to use rsync on a Windows machine to sort out some photos.
<h3 id="permissions">Permissions, permissions..</h3>
<p>Having happily used WSL1 in this manner for a couple of years, it was a rude awakening to try this with a new laptop with a small
hard-drive, so all the expo code was re-mounted on a plugged-in SD card. The problem of permissions that seemed to be a WSL2 issue reappeared with
a vengence on WSL1.
<p>The recommended WSL procedure (Jan.2021) said to mount the drive using
<a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">a new 'metadata' setting</a>:
<pre><code>sudo umount /mnt/d
sudo mount -t drvfs D: /mnt/d -o metadata,uid=1000,gid=1000,umask=22,fmask=111</code></pre>
<p>This is probably why there is a difference between automatically-mounted drives (e.g. C:, /mnt/c) and plug-in drives:
"<em>By default, WSL will set the uid and gid to the default user with drives that are auto-mounted during instance start. If you mount manually, you will
have to set these explicitly (the default user that gets created when WSL is first installed has a uid=1000 and gid=1000).</em>"
<p>And of course you will need to arrange that this happens automatically whenever you start WSL, so you will need to set
<a href="https://help.ubuntu.com/community/Fstab"><var>/etc/fstab</var></a>, so ensure that the relevant line says:
<pre><code>D: /mnt/d drvfs metadata,uid=1000,gid=1000,umask=22,fmask=11 0 0</code></pre>
<p>See also <a href="https://docs.microsoft.com/en-us/windows/wsl/file-permissions">File Permissions for WSL</a> (Dec. 2021).
<h3 id="bold">"Are you feeling lucky, punk"</h3>
<p>Links to useful articles to help you work this out for yourself:
<ul>
<li>WSL: <a href="https://hackaday.com/2019/12/23/linux-fu-wsl-tricks-blur-the-windows-linux-line/"> Converting Windows paths to Linux paths and
vice-versa</a>.
<li>Even if you have the hardware, you may have to enable it in
your BIOS and install the Windows Feature to do it. Instructions <a href="https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10">here</a>
and <a href="https://www.windowscentral.com/how-install-wsl2-windows-10">here</a>.
<li><a href="https://www.hanselman.com/blog/CoolWSLWindowsSubsystemForLinuxTipsAndTricksYouOrIDidntKnowWerePossible.aspx">Cool WSL tricks</a> - running Windows commands from WSL environment and running Linux commands from Windows terminal.
<li><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">deep integration</a> - Don't use gitforwindows, install the linux git client in WSL2
@ -74,11 +155,10 @@ in the \\wsl$\ ext4 filesystem (WSL2 only, not WSL1).
<tt><a href="https://www.scivision.dev/mount-usb-drives-windows-subsystem-for-linux/">mount-usb-drives-windows-subsystem-for-linux</a></tt>
<tt><a href="https://man.openbsd.org/ssh">ssh command line</a></tt>
<h3>Installing and Configuring the rest of the software you need on Windows</h3>
<p>Now return to &#8251;<a href="winlaptop.html">the Windows data maintenance laptop</a> page to configure all the rest of the software you need.&#8258;
<hr />
Go back to: <a href="basiclaptop.html">Basic laptop</a><br />
Go back to: <a href="surveylaptop.html">Survey laptop</a><br />
Go back to: &#8251;<a href="basiclaptop.html">Basic laptop</a><br />
Go back to: &#8258;<a href="surveylaptop.html">Survey laptop</a><br />
<hr /></body>
</html>

11
handbook/survey/getin.htm
View File

@ -65,7 +65,8 @@ template in the past, try not to do this yourself.
<li>Use Unix line endings (i.e. \n not \r\n).
<li>Use UTF-8 character encoding.
<li>NO UMLAUTS.
<li>NO UMLAUTS. Use <em>&amp;uml;</em> for u-umlaut and <em>&amp;oml;</em> for o-umlaut in text. But don't use these
in filenames.
<li>Cave numbers are written without any leading zeros at all (in filenames,
survey names, or anything else!).
@ -76,8 +77,8 @@ template in the past, try not to do this yourself.
<li>No full stop (.) characters in station names.
<li>All filenames to be in lower case. Capital letters in filenames can still (in 2018)
cause hassle when moving between Linux and Windows.
<li>All filenames to be in lower case. Capital letters in filenames can still (in 2022)
cause hassle when moving between Linux and Windows. (This includes suffix letters for cave entrances: it is "87a" NOT "87A".)
<!-- I think we can dispense with this now - DL
<li>Filenames: Because Survex is used on different sorts of computer, it is
@ -94,8 +95,10 @@ characters with no extension.-->
<li>No filenames starting with "-" or "!".
<li>No colons in filenames.
<li>Caves with a provisional number consisting of a year and a serial number
should be <em>hyphenated</em>, thus 2018-ad-01 not 2018_01 or any of the various other
should be <em>hyphenated</em>, thus 2018-ad-01 not with underscore 2018<b>_</b>01 or any of the various other
variants.
</ul>