<li><ahref="#rsync">Synchronisation of entire file heirarchies</a>: installation and use of <em>rsync</em><br/>
TLDR; While it can be done, just don't do this in Windows. It's too fiddly and prone to error. If you really want to synchronise a large file heirarchy, use a Linux machine.
- well not "hard" exactly, but complicated with lot of steps that are easy to get wrong and with poor feedback as to whether you have done each step correctly.
<li>Key management, either PuTTY or OpenSSH. You need one of these to generate and to use ssh keys on Windows. Otherwise none of git, scp, ftp (or rsync) will work.
<ul>
<li><ahref="https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview">OpenSSH for Windows</a> is new (fully working since 2021). <br>
- Use the Microsoft documented install method which puts the executables in <var>windows\system32\openssh\</var>.
<br> See also <ahref="https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview">https://learn.microsoft.com/... openssh_overview</a>
- We need a volunteer to produce documentation on installing and using <ahref="https://medium.com/swlh/installing-openssl-on-windows-10-and-updating-path-80992e26f6a1">OpenSSL/Windows</a>. Currently all our Windows documn. used PuTty and all our Linux documentaiotn uses OpenSSL. See also <ahref="https://think.unblog.ch/en/how-to-install-openssl-on-windows-10-11/">how-to-install-openssl-on-windows-10-11/</a>.
(pscp) and sFTP (psftp). The <ahref="https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html">PuTTY</a> installation includes puttygen and pagent which you also need. Version 0.77 was released on 2022-05-27.
<li><ahref="https://desktop.github.com/">GitHub Desktop for Windows</a> - yes this works with the expo server git even though we don't use GitHub itself for expo software.
<li><ahref="https://winscp.net/eng/download.php">WinScp</a> can be used as an alternative to Filezilla if you like (like Filezilla, it uses PuTTY ssh keys). It has a <ahref="https://winscp.net/eng/docs/screenshots">GUI</a> similar to Filezilla
<li><ahref="https://www.java.com/en/">java</a> - needed for GPSprune and CaveConverter. Has to be <ahref="https://www.java.com/en/download/windows_manual.jsp?host=java.com&locale=en-GB">installed separately</a> on a Windows machine.
<li><ahref="https://notepad-plus-plus.org/">Notepad++</a> or any other syntax-highlighting code editor for HTML and python. We have a syntax-highlighter
to colourize .svx files, download it: <ahref="/site_media/survex.xml">survex.xml</a> (no dark-mode yet though).
</ul>
<h4>Visual Studio Code editor</h4>
<p>If you use VS Code here are some relevant extras.
Not really for beginners but here are <ahref="https://docs.microsoft.com/en-gb/learn/modules/python-install-vscode/">instructions for
configuring it for python</a>.
On Windows you run VS Code as a Windows app but it communicates directly ("remotely") with the WSL Linux environment.
<p>Brendan wrote a guide to using puTTY and git for expo on a Windows machine. It's worth reading and it has lots of screenshots: <ahref="/expofiles/documents/idiots-guide-expo-git.pdf">Idiots guide to accessing expo git.pdf</a>.
<p>You need to <ahref="keyexchange.html">register a key with the expo server</a> to get upload (i.e. read/write) access. Do this first, Without it none of git, scp, ftp (or <ahref="#rsync">rsync</a>) will work.
to <ahref="https://blog.shvetsov.com/2010/03/making-pageant-automatically-load-keys.html">run at startup to load your key</a>. Note that you are loading your <em>private</em> key, the .ppk file, into pageant and that this key never leaves your laptop.</p>
<li><ahref="../putty/putty.html">Installing puTTY on Windows</a>.</li>
<li>Read the online instructions about using <ahref="https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement">OpenSSH key management</a> (several webpages). (Then write them up and edit this handbook to be more useful.)
<p>Most Windows software that we recommend "just works" if you have set up puTTY and have done the <ahref="keyexchange.html">key-pair setup</a> and are running a local ssh agent (pagent) automatically at boot up on your laptop.
<p>Some software (such as Filezilla) defaults to using pageant and it "just works".
<p>When using Windows please, please be <ahref="http://expo.survex.com/handbook/survey/getin.htm#filenames">excessively careful when naming files and survex names</a>.
<p>Some software, such as the commercial (but free) GitKraken, requires that you click a checkbox to say that you are "using local SSH agent" rather than specifying ssh private keys explicitly (File->Preferences->Authentication in GitKraken).
<p>Linux allows characters in filenames which Windows doesn't. There are also apparently normal filenames which Windows rejects (such as "<ahref="https://answers.microsoft.com/en-us/windows/forum/windows_7-files/why-cant-i-create-a-folder-with-name/23c86662-4988-4c7d-9c2d-3e33d4413de3">CON</a>") for historical reasons. Linux filenames are case-senstitive and Windows filenames aren't: beware.
<p>Linux people like to use <em>links</em>. This is where there is really only one file, but it is referred to by different names. This is particularly useful when a file is moved, but you want people who have got the old location to still be able to find it. This happens quite a lot when updating handbooks.
<p>
The links you are most likely to come across are that what looks like
<spanstyle="font-family:monospace; size=x-small; background-color: lightgray">/home/expo/expoweb</span> is really just a link to the folder <spanstyle="font-family:monospace; size=x-small; background-color: lightgray">/home/expo/repositories/git/expoweb</span>, <br/>
is a link to the file <spanstyle="font-family:monospace; size=x-small; background-color: lightgray">/home/expo/expofiles/gpslogs/essentials/essentials2019.gpx</span>
<p>But that example is in the <spanstyle="font-family:monospace; size=x-small; background-color: lightgray">:expoweb:</span> repository, so you won't be using sFTP to download it. Instead you will be using the version control software which handles it without problems. But we use it as an example of what to look out for when using sFTP.
There are two types of linux links: hard links and symbolic links. Symbolic links are much the same thing as Window's "Shortcuts" but there is no equivalent on Windows to Linux hard links. Fortunately we don't <em>seem</em> to have any hard links anywhere.
<p>What really makes things unpleasant is that sFTP software won't tell you when it comes across a link and will just do something stupid. Our recommended sFTP software - Filezilla - is guilty of this,as it pftp (puTTY) working in eith sFTP or scp mode.. So what happens is that when you download a load of files onto your laptop using Filezilla it will simply turn every link it finds into a complete copy of the file. Then when you upload those files to the server, the copied file overwrites the link. So the server now has two files with the same content - which is a maintenance nightmare. This is painfully stupid because if it is a symbolic link there is no reason why Filezilla couldn't just create a Windows Shortcut which would do exactly the same thing. But it doesn't.
<p>So the ordinary user won't notice any problems, but the nerds behind the scenes start to cuss and shout and generally carry-on in an expletive-heavy manner.
<p>What is even more irritating is that Filezilla displays an link (<spanstyle="font-family:monospace; size=x-small; background-color: lightgray">essentials.gpx</span> in the image) with a little arrow - it knows perfectly well that it is a symbolic link (although it does display it with a folder icon) - but it downloads it as a file copy.
<li>When using the git version control systems the download of a link works fine. But be careful not to edit the link file downloaded (it is just a text file holding the path of the file holding the actual contents) because then the version control client would upload it to the server and overwrite the link on the server with something that isn't a link. It also won't work as a Windows shortcut, but at least the default behaviour isn't actively dangerous.
<li>You will have to find out what to make the shortcut link to by logging in to the server (using a puTTY ssh logon) and doing <spanstyle="font-family:monospace; size=x-small; background-color: lightgray">"ls -l"</span> in the folder where the link is.
<li>But later, when re-uploading edited files from Windows to the server, Filezilla will see the Windows shortcut as a ".lnk" file which it will upload, but which will mean nothing to the linux server receiving it. So you would have to manually recreate the symbolic link by logging into the server using ssh and using the "ln -s" command. Yuk.
<li>Even if you use scp instead of sFTP, it does the same stupid thing when copying from a linux filesystem to a Windows filesystem. The puTTY package includes <spanstyle="font-family:monospace; size=x-small; background-color: lightgray">pscp.exe</span> but even if you force it to use the scp protocol like this:
it downloads a <em>copy</em> of the contents of essentials.gpx and not a link.
<li>A possible fix in the future might be to keep all your expo files in a separate partition of your hard disc which is formatted with a linux filesystem (such as ext4) and run the <ahref="https://www.ext2fsd.com/">ext2ntfs</a> driver to mount this fielsystem read-write from Windows. Probably not a good idea as the driver is a bit flaky in read-write mode and you could lose everything.
<li>be <ahref="../survey/getin.htm#filenames">excessively careful when naming files and survex names</a> and
<li>be <ahref="qstart-rsync.html">exceptionally careful when using rsync</a>.
</ul>
<p><ahref="https://rsync.samba.org/">rsync</a> is a Linux command-line application, so we need a terminal window to run it and a Windows client executable.
<li>Install <ahref="https://www.itefix.net/cwrsync">cwRsync</a> which comes with <em>no documentation</em> so you have to already know how to use rsync in some detail.
<li><ahref="https://stackoverflow.com/questions/23517023/rsync-from-windows-to-linux-using-puttys-pagent-authentication">Cygwin</a> . <br>- You just need the following files: rsync.exe, cygwin1.dll and cygz.dll .
<li>Install <ahref="https://gitforwindows.org/">Git for Windows</a> which includes an rsync executable, but you have to use its own bash terminal window. This may be something you don't want to do. This bash window is installed by default when you install <Ahref="https://gitforwindows.org/">gitforwindows</a>. Unfortunately while this <a
<li>Or you can use WSL: Windows System for Linux<imgsrc="wsl.jpg"align="right"hspace="10"><ahref="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux"></a>
<ul>
<li> This effectively installs a whole copy of Linux
<li>Unfortunately this does not integrate with PuTTY <em>at all</em>
<li>You will need to do the key management setup <em>again</em>, within the WSL environment.
