expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-22 07:11:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'expoweb' of ssh://expo.survex.com/home/expo/expoweb into expoweb

Browse Source
This commit is contained in:
echarlie 2022-04-05 21:07:57 -04:00
commit 7a6ca1d9c8
4 changed files with 119 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
handbook/computing/qstart-git.html
View File

@ -23,7 +23,11 @@
<dl> <dl>
<dt>loser (The survey data) - <em>this will fail until it is moved from hg to git</em></dt><dd>This is all the survex files. You need this if you are rearranging and editing a lot of survey files.<tt>git clone ssh://expo@expo.survex.com:loser</tt> (read/write)</dd> <dt>loser (The survey data) - <em>this will fail until it is moved from hg to git</em></dt><dd>This is all the survex files. You need this if you are rearranging and editing a lot of survey files.<tt>git clone ssh://expo@expo.survex.com/home/expo/loser</tt> (read/write)<br> or
<tt>hg clone ssh://expo@expo.survex.com//home/expo/loser</tt> Note the double slash with hg !<br>
For both of these, you will be doing it from your folder which will contain the new <var>/loser/</var> folder.
</dd>
<dt>drawings</dt><dd>These are the therion and tunnel cave vector drawing files. You need this if you are 'tunneling' a lot of cave surveys.<tt>git clone ssh://expo@expo.survex.com/home/expo/drawings</tt> (read/write)</dd> <dt>drawings</dt><dd>These are the therion and tunnel cave vector drawing files. You need this if you are 'tunneling' a lot of cave surveys.<tt>git clone ssh://expo@expo.survex.com/home/expo/drawings</tt> (read/write)</dd>
<dt>expoweb</dt><dd>These are all webpages, mostly the expo handbook. You need this only if you are editing a lot of handbook pages.<tt>git clone ssh://expo@expo.survex.com/home/expo/expoweb</tt> (read/write)</dd> <dt>expoweb</dt><dd>These are all webpages, mostly the expo handbook. You need this only if you are editing a lot of handbook pages.<tt>git clone ssh://expo@expo.survex.com/home/expo/expoweb</tt> (read/write)</dd>

66
handbook/computing/winlaptop.html
View File

@ -94,9 +94,10 @@ none of the packaged software (Filezilla, PuTTy) includes an rsync client.
provides. But we haven't properly explored this set of mechanisms yet.] provides. But we haven't properly explored this set of mechanisms yet.]
<p>The solution we have now for rsync is to use WSL and to create another key, distinct from the PuTTy one, and to upload that key to the expo <p>The solution we have now for rsync is to use WSL and to create another key, distinct from the PuTTy one, and to upload that key to the expo
server. Because this is treating WSL as if it were a different machine requiring its own key quite separate from the Windows key. (This works the server. Because this is treating WSL as if it were a different machine requiring its own key quite separate from the Windows key. (This works the
same way in WSL1 and WSL2).. same way in WSL1 and WSL2 - but see "Do you feel lucky, punk" below...)
<p>[We are not the only people to find this <a href="https://www.ubackup.com/windows-10/rsync-windows-10-1021.html">nasty and irritating</a>. Other <p>[We are not the only people to find this <a href="https://www.ubackup.com/windows-10/rsync-windows-10-1021.html">nasty and irritating</a>. Other
hacks are to use the obsolescent cygwin rsync or the rsync packaged within bash within more recent versions of 'Git for Windows' (which is built on <a href="https://www.msys2.org/docs/what-is-msys2/">MYSYS2</a>. If you are already familiar with any of these, then use them and not WSL.] hacks are to use the obsolescent cygwin rsync or the rsync packaged within bash within more recent versions of 'Git for Windows' (which is built on <a href="https://www.msys2.org/docs/what-is-msys2/">MYSYS2</a>.
If you are already familiar with any of these, then use them and not WSL.]
<p>So on a machine with WSL enabled, create an ordinary cmd window and get into the WSL environment using the wsl command:<br /> <p>So on a machine with WSL enabled, create an ordinary cmd window and get into the WSL environment using the wsl command:<br />
<span style="font-family:monospace; size=x-small; background-color: lightgray"> <span style="font-family:monospace; size=x-small; background-color: lightgray">
D:\CUCC-Expo\expoweb\ <font color=red>wsl</font> D:\CUCC-Expo\expoweb\ <font color=red>wsl</font>
@ -139,6 +140,7 @@ The generated key is in the current directory and you need to move them to ~/.ss
Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as: Now finally you can use all the usual command line tools at yor wsl command line to communicate with the server with ssh, scp, rsync, such as:
<pre> <pre>
<tt>rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles</tt></pre> <tt>rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ expofiles</tt></pre>
<p>But this only works with WSL1 not WSL2 ! Under WSL1 it doesn't matter if your /expofiles/ is mounted on the Linux filesystem, typically under /home/expo/expofiles, or whether it it in the NTFS filesystem mount for Linux as, e.g. /mnt/d/expofiles. Under WSL2 /mnt/d/expofiles fails, either with a permissions problem or, if you try sudo rsync.. , with a ssh authentication failure (why? Given that we explicitly sate that we want to be user expo@ ?). Ghastly anyway.
<p>All of which is really a bit ridiculous if all you want to do is to use rsync on a Windows machine to sort out some photos. <p>All of which is really a bit ridiculous if all you want to do is to use rsync on a Windows machine to sort out some photos.
@ -148,18 +150,29 @@ Now finally you can use all the usual command line tools at yor wsl command line
<li>cmd window - the old faithful going all the way back to MS-DOS. But no path to an rsync.exe executable as standard. <li>cmd window - the old faithful going all the way back to MS-DOS. But no path to an rsync.exe executable as standard.
<li>PowerShell terminal window - nope, no rsync. <li>PowerShell terminal window - nope, no rsync.
<li>bash window - installed by default when you install <A href="https://gitforwindows.org/">gitforwindows</a>. Unfortunately while this <a <li>bash window - installed by default when you install <A href="https://gitforwindows.org/">gitforwindows</a>. Unfortunately while this <a
href="https://en.wikipedia.org/wiki/MinGW">MINGW32</a> setup includes a command-line git executable it doesn't include rsync. (Well it does, but you have to excavate it yourself.) href="https://en.wikipedia.org/wiki/MinGW">MINGW32</a> setup includes a command-line git executable it doesn't include rsync. (Well you have
to excavate it yourself, see <a href="https://prasaz.medium.com/add-rsync-to-windows-git-bash-f42736bae1b3">a 2020 script</a>, alternatively <a
href="https://blog.tiger-workshop.com/add-rsync-to-git-bash-for-windows/">a 2017 recipe</a> or <a
href="https://gist.github.com/radleta/0b337a2b14f761951cf2aab0578512b9">recent experiments</a>.
This technique seems too fragile for us to bother with, but you might be lucky.)
<li>cygwin - a cmd terminal where you have downloaded and installed <a href="https://www.cygwin.com/">Cygwin</a>. Yes, if you have selected the <li>cygwin - a cmd terminal where you have downloaded and installed <a href="https://www.cygwin.com/">Cygwin</a>. Yes, if you have selected the
rsync package you will be able to run the rysnc executable, but it won't have access to the cyptographic key so it can't connect to the expo server (but you can make it work by using it with ssh from OpenSSH). rsync package you will be able to run the rysnc executable, but it won't have access to the cyptographic key so it can't connect to the expo server (but
you can make it work by using it with ssh from OpenSSH).
Please feel free to work out how to make this work. A more recent, graphical variant is <a Please feel free to work out how to make this work. A more recent, graphical variant is <a
href="https://hackaday.com/2017/03/29/swan-better-linux-on-windows/">Swan</a>. href="https://hackaday.com/2017/03/29/swan-better-linux-on-windows/">Swan</a>.
<li><img src="wsl.jpg" align="right" hspace="10"><a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux">Windows Subsystem for Linux</a> aka WSL available on all Windows10 machines since November 2019. <li><img src="wsl.jpg" align="right" hspace="10"><a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux">Windows Subsystem for Linux</a> aka
WSL available on all Windows10 machines since November 2019.
<ul> <ul>
<li>WSL <a href="https://docs.microsoft.com/en-us/windows/wsl/install">Install it like this</a>. This does what we need. This works using a key generated by its own version of ssh-keygen if you follow the instructions above about putting it in the right place. <li>WSL <a href="https://docs.microsoft.com/en-us/windows/wsl/install">Install it like this</a>. This does what we need. This works using a key generated
<li>WSL: <a href="https://hackaday.com/2019/12/23/linux-fu-wsl-tricks-blur-the-windows-linux-line/"> Converting Windows paths to Linux paths and vice-versa</a>. by its own version of ssh-keygen if you follow the instructions above about putting it in the right place.
<li>In WSL you can choose whether to store the files in the WSL view of NTFS, e.g. /mnt/c/expo/ or in an ext4 filesystem e.g. \\wsl$\Ubuntu-20.04\home\expo . Our experience is with /mnt/c/expo/ . <li>WSL: <a href="https://hackaday.com/2019/12/23/linux-fu-wsl-tricks-blur-the-windows-linux-line/"> Converting Windows paths to Linux paths and
<li>WSL now installs as WSL2 by default, but older machines (mostly laptops) may not have the Hyper-V Virtualization hardware and may have to run WSL1. vice-versa</a>.
This is fine: the behaviour is nearly identical so far as an expo laptop is concerned. Even if you have the hardware, you may have to enable it in <li>In WSL you can choose whether to store the files in the WSL view of NTFS, e.g. /mnt/c/expo/ or in an ext4 filesystem e.g.
\\wsl$\Ubuntu-20.04\home\expo . Our experience is with WSL1 with the files on NFTS e.g. /mnt/d/expo/ .
<li>Not all machines are compatible with WSL1 though, see <a href="wsllaptop.html">WSL in more depth</a> and WSL2 differences.
Even if you have the hardware, you may have to enable it in
your BIOS and install the Windows Feature to do it. Instructions <a href="https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10">here</a> your BIOS and install the Windows Feature to do it. Instructions <a href="https://www.omgubuntu.co.uk/how-to-install-wsl2-on-windows-10">here</a>
and <a href="https://www.windowscentral.com/how-install-wsl2-windows-10">here</a>. and <a href="https://www.windowscentral.com/how-install-wsl2-windows-10">here</a>.
@ -167,41 +180,14 @@ and <a href="https://www.windowscentral.com/how-install-wsl2-windows-10">here</a
</ul> </ul>
<li>A full <a href="https://www.brianlinkletter.com/installing-debian-linux-in-a-virtualbox-virtual-machine/">virtual Linux machine</a> running <li>A full <a href="https://www.brianlinkletter.com/installing-debian-linux-in-a-virtualbox-virtual-machine/">virtual Linux machine</a> running
using a hypervisor such as <a href="https://www.virtualbox.org/">VirtualBox</a> which has its own virtual Linux filesystem. This setup has the using a hypervisor such as <a href="https://www.virtualbox.org/">VirtualBox</a> which has its own virtual Linux filesystem. This setup has the
advantage that you don't have to partitition your hard drive but the disadvantage that you may not have easy access to the files from Windows (a feature it shares with WSL2). advantage that you don't have to partitition your hard drive but the disadvantage that you may not have easy access to the files from Windows (a feature
it shares with WSL2).
<li><a href="https://github.com/canonical/multipass">Canonical Multipass</a> - a VM alternative (Ubuntu only). <li><a href="https://github.com/canonical/multipass">Canonical Multipass</a> - a VM alternative (Ubuntu only).
</ul> </ul>
<h4>WSL tricks and tips</h4>
<p>WSL unfortunately introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file permissions that are fundamentally different things and are never in sync in any sense. <a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">All sorts of fun</a> results. The alternative is to store all your data in the \\wsl$\ ext4 filesystem (WSL2 only, not WSL1).
<p>If you are disturbed by the instructions to produce an entirely different key for WSL to use when your PC already has a perfectly good PuTTy key installed on the server, then you are right. It is inelegant. But it works, the instructions are shorter and there are fewer things that go wrong. If you are terribly offended by that then you can set your PC up to use one key shared between WSL and normal-Windows as described in <a href="https://devblogs.microsoft.com/commandline/sharing-ssh-keys-between-windows-and-wsl-2/">this October 2019 article</a>. (Don't set up a password on the key because then you don't need to install keychain.) But beware, this sort of thing goes out of date quite rapidly.
<p>Links to useful articles to help you work this out for yourself:
<ul>
<li><a href="https://www.hanselman.com/blog/CoolWSLWindowsSubsystemForLinuxTipsAndTricksYouOrIDidntKnowWerePossible.aspx">Cool WSL tricks</a> - running Windows commands from WSL environment and running Linux commands from Windows terminal.
<li><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">deep integration</a> - Don't use gitforwindows, install the linux git client in WSL2
<li><a href="https://blog.anaisbetts.org/using-github-credentials-in-wsl2/">using-github-credentials-in-wsl2</a> - How to use gitforwindows and WSL to connect to GitHub.
<li><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">WSL & Visual Studio Code</a>
</ul>
<p>When things go wrong when trying to sort this out, you may find these pages useful. I did.
<tt><a href="https://serverfault.com/questions/423075/rsync-failed-to-set-permission-operation-not-permitted">rsync - failed to set permission - operation not permitted</a></tt>
<tt><a href="https://blogs.msdn.microsoft.com/wsl/2017/04/18/file-system-improvements-to-the-windows-subsystem-for-linux/">file-system-improvements-to-the-windows-subsystem-for-linux/</a></tt>
<tt><a href="https://stackoverflow.com/questions/48328446/id-rsa-pub-file-ssh-error-invalid-format">id-rsa-pub-file-ssh-error-invalid-format</a></tt>
<tt><a href="https://stackoverflow.com/questions/42863913/key-load-public-invalid-format">key-load-public-invalid-format</a></tt>
<tt><a href="https://superuser.com/questions/1370877/ssh-error-loading-key-id-rsa-invalid-format">ssh-error-loading-key-id-rsa-invalid-format</a></tt>
<tt><a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">chmod-chown-wsl-improvements</a></tt>
<tt><a href="https://www.scivision.dev/mount-usb-drives-windows-subsystem-for-linux/">mount-usb-drives-windows-subsystem-for-linux</a></tt>
<tt><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">VS Code and WSL</a></tt>
<tt><a href="https://man.openbsd.org/ssh">ssh command line</a></tt>
<h3>Installing and Configuring the rest of the software you need on Windows</h3> <h3>Installing and Configuring the rest of the software you need on Windows</h3>
<p>Now return to <a href="yourlaptop.html">the data maintenance laptop</a> page to configure all the rest of the software you need. <p>Now return to <a href="yourlaptop.html">the data maintenance laptop</a> page to configure all the rest of the software you need.
<hr /> <hr /></body>
</body>
</html> </html>

79
handbook/computing/wsllaptop.html Normal file
View File

@ -0,0 +1,79 @@
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>CUCC Expedition Handbook: Windows expo laptop</title>
<link rel="stylesheet" type="text/css" href="../../css/main2.css" />
</head>
<body>
<h2 id="tophead">CUCC Expedition Handbook - Computers/WSL</h2>
<h1>WSL on a Windows laptop</h1>
<p>First read the generic instructions for all the software installations you will need:
<br>
<a href="basiclaptop.html">Setting up a basic Expo laptop</a>
<br>
<a href="yourlaptop.html">Setting up a machine for Expo data maintenance</a>
<br>
<a href="winlaptop.html">Setting up a Windows machine for Expo data maintenance</a>
</p>
<h3>WSL1 and WSL2</h3>
<p>WSL now installs as WSL2 by default, but older machines (mostly laptops) may not have the Hyper-V Virtualization hardware and may have to run WSL1.
This is fine: the behaviour is nearly identical so far as an expo laptop is concerned, where you just want to use rsync and scp. An old 2011-era PC has
been doing troggle development happily using WSL1.
<p>For some laptops though, WSL1 does not install properly and you have to use WSL2. The symptom is two error messages when you run <var>dmesg</var> in
the Linux bash terminal window. However, if all you want to use it for is rsync, this may still be fine.
<h3>WSL2 - things to watch for</h3>
<p>On some Windows 10 laptops WSL1 fails to install properly and you have to use WSL2.
<p>When running a full troggle development laptop (see separate <a href="../troggle/troglaptop.html">troggle documentation</a>)
you will want to use <var>python venv</var>. This barfs untidily if you have the code on NTFS, e.g. if mounted on <var>/mnt/c/</var>. You almost
certainly have to move all your code to the internal network share e.g. <var>\\wsl$\Ubuntu-20.04\home\expo\troggle\</var> which
is how it looks when you are browing from Windows.
(No, don't try to be cute and keep it on <var>/mnt/c/</var> and just put a soft link in <var>/home/expo/</var>. That doesn't work either.)
This means that the code is actually living on a Linux <var>ext4</var> filesystem hidden away on your disc where you can only see it using the
'network' method <var>\\wsl$\Ubuntu-20.04\home\expo\troggle\</var>. This means that file access is somewhat faster too but you probably won't notice.
<p>rsync doesn't work with NTFS partitions the way that WSL1 does. See <winlaptop.html>Windows laptop</a> for a bit of detail. [Work in progress.]
<p>Read <a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">WSL & Visual Studio Code</a> and go back and read the bits about VS Code
running remotely in the &#8251;Windows data maintenance laptop page.
<h3>WSL1 gotchas and tips</h3>
<p>WSL using files on NTFS unfortunately introduces a wonderful new problem of file permissions. Every file on the Windows filesystem NTFS has a set of
permissions managed by the filesystem. Every NTFS file that WSL knows about (if mounted with -o metadata) acquires a completely parallel set of file
permissions that are fundamentally different things and are never in sync in any sense. <a
href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">All sorts of fun</a> results. The alternative is to store all your data
in the \\wsl$\ ext4 filesystem (WSL2 only, not WSL1).
<p>If you are disturbed by the instructions to produce an entirely different key for WSL to use when your PC already has a perfectly good PuTTy key installed on the server, then you are right. It is inelegant. But it works, the instructions are shorter and there are fewer things that go wrong. If you are terribly offended by that then you can set your PC up to use one key shared between WSL and normal-Windows as described in <a href="https://devblogs.microsoft.com/commandline/sharing-ssh-keys-between-windows-and-wsl-2/">this October 2019 article</a>. (Don't set up a password on the key because then you don't need to install keychain.) But beware, this sort of thing goes out of date quite rapidly.
<p>Links to useful articles to help you work this out for yourself:
<ul>
<li><a href="https://www.hanselman.com/blog/CoolWSLWindowsSubsystemForLinuxTipsAndTricksYouOrIDidntKnowWerePossible.aspx">Cool WSL tricks</a> - running Windows commands from WSL environment and running Linux commands from Windows terminal.
<li><a href="https://code.visualstudio.com/blogs/2019/09/03/wsl2">deep integration</a> - Don't use gitforwindows, install the linux git client in WSL2
<li><a href="https://blog.anaisbetts.org/using-github-credentials-in-wsl2/">using-github-credentials-in-wsl2</a> - How to use gitforwindows and WSL to connect to GitHub.
</ul>
<p>When things go wrong when trying to sort this out, you may find these pages useful. I did.
<tt><a href="https://serverfault.com/questions/423075/rsync-failed-to-set-permission-operation-not-permitted">rsync - failed to set permission - operation not permitted</a></tt>
<tt><a href="https://blogs.msdn.microsoft.com/wsl/2017/04/18/file-system-improvements-to-the-windows-subsystem-for-linux/">file-system-improvements-to-the-windows-subsystem-for-linux/</a></tt>
<tt><a href="https://stackoverflow.com/questions/48328446/id-rsa-pub-file-ssh-error-invalid-format">id-rsa-pub-file-ssh-error-invalid-format</a></tt>
<tt><a href="https://stackoverflow.com/questions/42863913/key-load-public-invalid-format">key-load-public-invalid-format</a></tt>
<tt><a href="https://superuser.com/questions/1370877/ssh-error-loading-key-id-rsa-invalid-format">ssh-error-loading-key-id-rsa-invalid-format</a></tt>
<tt><a href="https://devblogs.microsoft.com/commandline/chmod-chown-wsl-improvements/">chmod-chown-wsl-improvements</a></tt>
<tt><a href="https://www.scivision.dev/mount-usb-drives-windows-subsystem-for-linux/">mount-usb-drives-windows-subsystem-for-linux</a></tt>
<tt><a href="https://man.openbsd.org/ssh">ssh command line</a></tt>
<h3>Installing and Configuring the rest of the software you need on Windows</h3>
<p>Now return to &#8251;<a href="winlaptop.html">the Windows data maintenance laptop</a> page to configure all the rest of the software you need.&#8258;
<hr /></body>
</html>

10
handbook/computing/yourlaptop.html
View File

@ -134,7 +134,15 @@ but all the recommended software here is open source (and please don't install p
</ul> </ul>
<h4>Visual Studio Code editor</h4> <h4>Visual Studio Code editor</h4>
<p>A short note about the phenomenon of VS code is in order. Not really for beginners but here are <a href="https://docs.microsoft.com/en-gb/learn/modules/python-install-vscode/">instructions for configuring it for python</a>. In case you didn't know, by 2019 over <a href="https://insights.stackoverflow.com/survey/2019#technology">half</a> of all software developers used this editor for their Linux and Windows work and it is undoubtedly more now. <p>A short note about the phenomenon of VS code is in order.
Not really for beginners but here are <a href="https://docs.microsoft.com/en-gb/learn/modules/python-install-vscode/">instructions for
configuring it for python</a>.
In case you didn't know, by 2019 over <a href="https://insights.stackoverflow.com/survey/2019#technology">half</a> of
all software developers used this editor for their Linux and Windows work and it is undoubtedly more now. On Windows you run VS Code as
a Windows app but it communicates directly ("remotely") with the WSL Linux environment.
<p>You will definitely want the
<a href="https://marketplace.visualstudio.com/items?itemName=donjayamanne.githistory">"Git History" extension</a>
and probably the "Git Lens" extension too.