expo/expoweb
1
0
Fork 0
mirror of https://expo.survex.com/repositories/expoweb/.git/ synced 2024-11-25 08:41:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

rsync troubles and updates

Browse Source
This commit is contained in:
Philip Sargent 2021-12-19 17:22:50 +00:00
parent bbeb02e1cc
commit f826f7a946
3 changed files with 17 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
handbook/computing/keyexchange.html
View File

@ -97,8 +97,11 @@ give at <a href="https://www.ssh.com/academy/ssh/agent">www.ssh.com/academy/ssh/
and if the process is not running, start it like this
<code>
<font color="darkred">$</font> eval `ssh-agent`
<font color="darkred">$</font> ssh-add ~/.ssh/<b>id_rsa_wsl</b>
<font color="darkred">$</font> ssh-add -l
</code>
Check that the key is loaded by running <var>ssh-add -l</var> and then test that you can login again.
<var>ssh-add -l</var> checks that the key is loaded and the process is running. Now try to login again
with <var>ssh expo@expo.survex.com</var>.
<br><br></li>
</ol>

8
handbook/computing/qstart-rsync.html
View File

@ -26,11 +26,11 @@ uploading photographs: <a href="uploading.html">uploading.html</a>.
<p> To sync all
the files from the server to your local expofiles directory on your laptop:</p>
<p><tt>rsync -nazv --delete-after --prune-empty-dirs expo@expo.survex.com:expofiles/ /home/expo/expofiles</tt></p>
<p><code>rsync -nazv --delete-after --exclude="thumbs/" --exclude="*.???.xml" --exclude="*.jpeg.xml" expo@expo.survex.com:expofiles/ /home/expo/expofiles</code></p>
<p>To sync the local expofiles directory back to the server after you have edited updates (e.g. scanned some hand-drawn surveys into expofiles/surveyscans/ (but only if your machine runs Linux):</p>
<p><tt>rsync -nazv /home/expo/expofiles/surveyscans/2019/ expo@expo.survex.com:expofiles/surveyscans/2019</tt></p>
<p><code>rsync -nazv --delete-after /home/expo/expofiles/surveyscans/2019/ expo@expo.survex.com:expofiles/surveyscans/2019</code></p>
then CHECK that the list of files it produces matches the ones you absolutely intend to delete forever! ONLY THEN do it without the "-n" option. "-n" is the same as "--dry-run" which shows you the overwriting changes but doesn't actually do them.
@ -38,14 +38,14 @@ then CHECK that the list of files it produces matches the ones you absolutely in
<ul>
<li>do a dry-run of rsync from the server to your laptop immediately before you do an upload to the server
<li>use --delete-after --prune-empty-dirs when downloading, but never when uploading
<li>use --delete-after
<li>work at the minimum scope of folders you need, e.g. within expofiles/photos/ or expofiles/surveyscans/ not for the whole of expofiles all at once.
<li>take exagerated care with the placement of the final slash in directory parameters to the rsync. Get it wrong and you duplicate things instead of updating them and it takes ages to sort out.
</ul>
<p>(do be <b>incredibly</b> careful not to delete piles of stuff then rsync back, or to get the directory level of the command wrong - as it'll all get deleted on the server too, and we may not have backups!). It's <b>absolutely vital</b> to use rsync --dry-run --delete-after first to check what would be deleted.
<p>If you are using rsync from a Windows machine you will <em>not</em> get all the files as some filenames are incompatible with Windows. What will happen is that rsync will invisibly change the names as it downloads them from the Linux expo server to your Windows machine, but then it forgets what it has done and tries to re-upload all the renamed files to the server even if you have touched none of them. Now there won't be any problems with simple filenames using all lowercase letters and no funny characters, but we have nothing in place to stop anyone creating such a filename somewhere in that 40GB or of detecting the problem at the time. So don't do it. If you have a Windows machine use Filezilla not rsync.
<p>If you are using rsync from an NTFS folder on a Windows machine (even if you are using WSL to do it) you will <em>not</em> get all the files for certain as some Linux filenames are incompatible with Windows. What will happen is that rsync will invisibly change the names as it downloads them from the Linux expo server to your Windows machine, but then it forgets what it has done and tries to re-upload all the renamed files to the server even if you have touched none of them. This pollutes the server. Now there won't be any problems with simple filenames using all lowercase letters and no funny characters, but we have nothing in place to stop anyone creating an incompatible filename of that sort somewhere in that 40GB or of detecting the problem at the time. So don't do it. Be extra, extra careful and religiously use the -n (DRY RUN) setting and manually check all changes before running rsync without -n.
<p>(We may also have an issue with rsync not using the appropriate user:group attributes for files pushed back to the server. This may not cause any problems, but watch out for it.)</p>
</dl>

9
handbook/computing/todo.html
View File

@ -170,6 +170,15 @@ Click here if no todo list appears here.</div></a>
<li>re-enable CaveView
</details>
<details><summary><!--2021-12-18 psargent--> Scan Upload sets ownership of new files - rsync fails thereafter</summary>
Because the upload form (in troggle) is operating with webserver permisions as user 'www-data' it can't create files
with ownership 'expo'. This stops rsync working (which uses exo@expo.survex.com username), so the form should set permsisions as <br>
-rw-rw-r--<br>
instead of<br>
-rw-r--r--<br>
which it does at the moment.
</details>
<details><summary><!--2021-11-04 psargent--> Scan Upload gives permissions error on localhost and, INITIALLY, on expo.survex.com .</summary>
<li>http://expo.survex.com/scanupload/2022:00