expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Setting up a WIndows laptop for expo use - unsolved configuration issues

Browse Source
This commit is contained in:
Philip Sargent 2019-12-28 00:23:21 +00:00
parent f83edf27a0
commit fa64c485d7
4 changed files with 104 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
handbook/computing/fzlink.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

114
handbook/computing/winlaptop.html
View File

@ -6,41 +6,133 @@
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook - Your Windows laptop</h2>
<h2 id="tophead">CUCC Expedition Handbook - Computers</h2>
<h1>Setting up an Expo laptop - using Microsoft Windows</h1>
<p>A work in progress.
<p>It used to be quite simple - software equivalents existed for Windows for everything we needed. The new (since Spring 2018) requirement to make ssh keys obligatory has simply added an extra step for most things, but has made one thing (rsync) really quite hard to get working.
<p>It used to be quite simple: software equivalents existed for Windows for everything we needed. Since Spring 2018, we unfortunately have to use ssh keys. This has simply added an extra step for most things, but has made one thing (rsync) really quite hard to get working.
<p>What makes this all much more tiresome is that Microsoft are introducing some cool new ways of interacting with Linux systems but these are new and need some effort to get configured correctly. See the end of this page for more details.
<ul>
<li><a href="#well">Things that already work well with a Windows laptop</a><br>
- Editing the handbook webpages, typing up SVX files and transcribing the logbook, <br>
- Anything where the file-transfer to the expo surver is via the version control software (TortoiseHg or Git-for-Windows)
<li><a href="#problems">Things that cause problems</a><br>
- filenames and unintentional duplication because hardlinks are not understood by Windows,<br>
- filenames and unintentional duplication because links are not understood by Windows,<br>
- sFTP or scp for more than a handful of files
<li><a href="#hard">Things that are really, really hard</a><br>
- using rsync <br>
- large-scale updating of several folders at once on expofiles without overwriting other people's work (which means using rsync)
<li><a href="#bold">Things for the bold</a><br>
- using WSL and WSL2, Linux virtual machines and Docker containers
</ul>
<h3 id="well">Things that already work well</h3>
<p>Anytihng to
<p>Anything where the file upload and download is done via the verson control client software works really well.
<h4>Key management using PuTTy</h4>
<p>Most Windows software that we recommend "just works" if you have set up PuTTy and have done the <a href="keyexchange.html">cryptographic key exchange</a> and are running a local ssh agent (pagent) automatically at boot up on your laptop.
<p>Some software, such as the commercial (but free) GitKraken, requires that you click a checkbox to say that you are "using local SSH agent" rather than specifying ssh private keys explicitly (File->Preferences->Authentication in GitKraken).
<p>Some software (such as FIlezilla) defaults to using the local agent and it "just works".
<h3 id="problems">Things that cause problems</h3>
<p>Linux people like to use hard-links: where there is really only one file, but it is referred to by different names. The ones you are most likely to come across are that what looks like
/home/expo/expoweb is really just a link to /home/expo/repositories/hg/expoweb, and that
expoweb/essentials.gpx
is a link to /home/expo/expofiles/gpslogs/essentials/essentials2019.gpx
<h4>Filenames</h4>
<p>Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "CON") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware.
<h4>Symbolic links and sFTP</h4>
<p>Linux people like to use <em>links</em>. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks.
<p>
The links you are most likely to come across are that what looks like
<span style="font-family:monospace; size=x-small; background-color: lightgray">/home/expo/expoweb</span> is really just a link to the folder <span style="font-family:monospace; size=x-small; background-color: lightgray">/home/expo/repositories/hg/expoweb</span>, <br>
and that
<span style="font-family:monospace; size=x-small; background-color: lightgray">expoweb/essentials.gpx</span>
is a link to the file <span style="font-family:monospace; size=x-small; background-color: lightgray">/home/expo/expofiles/gpslogs/essentials/essentials2019.gpx</span>
<p>But that example is in the <span style="font-family:monospace; size=x-small; background-color: lightgray">::expoweb::</span> repository, so you won't be using sFTP to download it. Instead you will be using the version control software which handles it without problems. But we use it as an example of what to look out for when using sFTP.
<p>
<img src="fzlink.jpg" align="right">
There are two types of <a href="">linux links</a>: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux soft links. Fortunately we don't <em>seem</em> to have any hard links anywhere.
<p>What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.
<p>So the ordinary user won't notice any problems, but the nerds behind the scenes start to cuss and shout and generally carry-on in an expletive-heavy manner.
<p>What is even more irritating is that Filezilla displays an link (<span style="font-family:monospace; size=x-small; background-color: lightgray">essentials.gpx</span> in the image) with a little arrow - it knows perfectly well that it is a symbolic link (although it does display it with a folder icon) - but it downloads it as a file copy.
<h4>Symbolic links: the solution for Windows</h4>
<ul>
<li>When using the version control systems TortoiseHg or Git-for-Windows the download of a link works fine. But be careful not to edit the link file downloaded (it is just a text file holding the path of the file holding the actual contents) because then the version control client would upload it to the server and overwrite the link on the server with something that isn't a link. It also won't work as a Windows shortcut, but at least the default behaviour isn't actively dangerous.
<li>When using sFTP, manually check whether any files you are copying from the server are links - look at the symbol in Filezilla.
<li>Be careful not to copy any links using sFTP and instead recreate them manually on the Windows filesystem using right-click "Create shortcut".
<li>You will have to find out what to make the shortcut link to by logging in to the server (using a PuTTy ssh logon) and doing <span style="font-family:monospace; size=x-small; background-color: lightgray">"ls -l"</span> in the folder where the link is.
<li>But later, when re-uploading edited files from Windows to the server, Filezilla will see the Windows shortcut as a ".lnk" file which it will upload, but which will mean nothing to the linux server receiving it. So you would have to manually recreate the symbolic link by logging into the server using ssh and using the "ln -s" command. Yuk.
<br>
<li>Even if you use scp instead of sFTP, it does the same stupid thing when copying from a linux filesystem to a Windows filesystem. The PuTTy package includes <span style="font-family:monospace; size=x-small; background-color: lightgray">pscp.exe</span> but even if you force it to use the scp protocol like this:
<tt>
pscp -scp expo@expo.survex.com:expoweb/essentials.gpx .
</tt>
it downloads a <em>copy</em> of the contents of essentials.gpx and not a link.
<li>A possible fix in the future might be to keep all your expo files in a separate partition of your hard disc which is formatted with a linux filesystem (such as ext4) and run the <a href="https://www.ext2fsd.com/">ext2ntfs</a> driver to mount this fielsystem read-write from Windows. Probably not a good idea as the driver is a bit flaky in read-write mode and you could lose everything.
</ul>
<h3 id="hard">Things that are really, really hard</h3>
<p>The core problem is integrating the PuTTy key management software (pagent.exe) with a terminal window.
<p>The core problem is integrating the PuTTy key management software (pagent.exe) with a terminal window. We need a terminal window to run rsync as none of the packaged software (Filezilla, PuTTY) includes an rsync client.
<p>We don't currently (December 2019) have a working recipe to set this up. Hopefully we will have it sorted in a month or two.
<h3 id="bold">"Are you feeling lucky, punk"</h3>
<p>So here is the current wild frontier. Currently these are the ways to get a terminal window which might work:
<ul>
<li>cmd window - the old faithful going all the way back to MS-DOS. But no path to an rsync.exe executable as standard.
<li>bash window - installed by default when you install <A href="https://gitforwindows.org/">gitforwindows</a>. Unfortunately while this <a href="https://en.wikipedia.org/wiki/MinGW">MINGW32</a> setup includes a comamnd-line git executable it doesn't include rsync.
<li>cygwin - a cmd terminal where you have downloaded and installed <a href="https://www.cygwin.com/">Cygwin</a>. Yes, if you have selected the rsync package you will be able to run the rysnc executable, but it won't have access to the cyptographic key so it can't connect to the expo server. Please feel free to work out how to make this work. A more recent, grpahical variant is <a href="https://hackaday.com/2017/03/29/swan-better-linux-on-windows/">Swan</a>.
<li>PowerShell terminal window - nope, no rsync.
<li><a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux">Windows Subsystem for Linux</a> aka WSL1 available on all Windows10 machines since the November 2019 update.
<ul>
<li> <a href="https://code.visualstudio.com/remote-tutorials/wsl/enable-wsl">Install it like this</a>. This should do all we want. This should work, either by using the key generated by PuTTy or by using a key generated by its own version of ssh-keygen, but somehow it doesn't find the key. But fixing this is probably not a good use of time (though the fix may be trivial) because it will shortly be made obsolete by:
<li><a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux#WSL_2">WSL2</a> - the new wonderful system which will solve all our problems, make the tea and carry our gear up to topcamp. Allegedly.
<li>The <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">2019 WSL2 release</a> includes a selection of complete Linux kernels. If you want to use this, then please do - and then write the handbook documentation too. But beware that it has <a href="https://docs.microsoft.com/en-us/windows/wsl/wsl2-index">two different modes</a> which behave differently.
</ul>
<li><a href="https://www.theregister.co.uk/2019/12/18/multipass/">Canonical Multipass</a> - a completely different alternative to WSL: more isolation, more understandable behaviour (?)
<li>A full <a href="https://www.brianlinkletter.com/installing-debian-linux-in-a-virtualbox-virtual-machine/">virtual Linux machine</a> running using a hypervisor such as <a href="https://www.virtualbox.org/">VirtualBox</a> which has its own virtual Linux filesystem. This setup has the advantage that you don't have to partitition your hard drive but the disadvantage that you can't get at any of the files from Windows itself except via a network protocol \\wsl$. Which may be no hardship.
</ul>
<p><img src="wsl.jpg" align="right" hspace="10">
WSL: the Windows Subsystem for Linux. The first release of this didn't do the ssh key exchange process easily: <a href="https://blog.anaisbetts.org/using-github-credentials-in-wsl2/">"fairly annoying because of how out-to-lunch SSH Agent is"</a>.
<p>WSL introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file permissions that "mirror" the NTFS permissions but can get out of sync. <a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">All sorts of fun</a> results: <em>"With network file systems, DrvFs does not set the correct Linux permissions bits on a file; instead, all files are reported with full access (0777) and the only way to determine if you can actually access the file is by attempting to open it."</em>
<p>Links to useful articles to help you work this out for yourself:
<ul>
<li><a href="https://www.hanselman.com/blog/CoolWSLWindowsSubsystemForLinuxTipsAndTricksYouOrIDidntKnowWerePossible.aspx">Cool WSL tricks</a> - running Windows commands from WSL environment and running Linux commands from Windows terminal.
<li><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">deep integration</a> - Don't use gitforwindows, install the linux git client in WSL2
<li><a href="https://hackaday.com/2019/12/23/linux-fu-wsl-tricks-blur-the-windows-linux-line/">Converting Windows paths to Linux paths and vice-versa.
<li><a href="https://blog.anaisbetts.org/using-github-credentials-in-wsl2/">using-github-credentials-in-wsl2</a> - How to use gitforwindows and WSL to connect to GitHub.
<li><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">WSL2 & Visual Studio Code</a>
</ul>
<p>Scatch list of things that go wrong when trying to sort this out
<tt><a href="https://blogs.msdn.microsoft.com/wsl/2017/04/18/file-system-improvements-to-the-windows-subsystem-for-linux/">file-system-improvements-to-the-windows-subsystem-for-linux/</a></tt>
<tt><a href="https://stackoverflow.com/questions/48328446/id-rsa-pub-file-ssh-error-invalid-format">id-rsa-pub-file-ssh-error-invalid-format</a></tt>
<tt><a href="https://stackoverflow.com/questions/42863913/key-load-public-invalid-format">key-load-public-invalid-format</a></tt>
<tt><a href="https://superuser.com/questions/1370877/ssh-error-loading-key-id-rsa-invalid-format">ssh-error-loading-key-id-rsa-invalid-format</a></tt>
<tt><a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">chmod-chown-wsl-improvements</a></tt>
<tt><a href="https://www.scivision.dev/mount-usb-drives-windows-subsystem-for-linux/">mount-usb-drives-windows-subsystem-for-linux</a></tt>
<tt><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">VS Code and WSL</a></tt>
<tt><a href=">https://man.openbsd.org/ssh">ssh command line</a></tt>
<h3>Installing and Configuring the rest of the software you need on Windows</h3>
<p>Now return to <a href="yourlaptop.html">the Your Laptop</a> page to configure all the rest of the software you need.
<hr />
<p>WSL: the Windows Subsystem for Linux. The first release of this didn't do the ssh key exchange process easily. The 2019 WSL2 release includes a complete Linux kernel. If you want to use this, then please do - and then write the handbook documentation too. But beware that it has two different modes which behave differently.
<div id="menu">
<ul id="links">
<li><a href="../index.htm">Handbook</a>
<li><a href="yourlaptop.html">Your laptop</a>
<li><a href="../../infodx.htm">Main index</a></li>
</ul>

BIN
handbook/computing/wsl.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.8 KiB

2
handbook/computing/yourlaptop.html
View File

@ -20,7 +20,7 @@ The <i>expo laptop</i> uses <a href="https://www.debian.org/intro/about">Debian<
<li><a href="winlaptop.html">A Windows laptop for expo</a> - a page of more detailed instructions for those who are definitely going to be using a Windows PC or laptop.
<li>WSL: the Windows Subsystem for Linux. The first release of this didn't do the ssh key exchange process easily. The <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">2019 WSL2 release</a> includes a complete Linux kernel. If you want to use this, then please do - and then write the handbook documentation too. But beware that it has <a href="https://docs.microsoft.com/en-us/windows/wsl/wsl2-index">two different modes</a> which behave differently.
<li>Mac users will need to use the Linux documentation as a guide and work it all out for themselves.