updating wallets scripts (expofiles copy not git)

handbook/troggle/scriptscurrent.html

@@ -33,7 +33,7 @@ p {
 <p>It coordinates producing the 3d surveys used in the cave description pages, updates the area pages, runs the folk script, runs the QM list generation within each of the cave pages that needs it, runs svxtrace, and reports on everything using "bigbro" which we don't have any other reference to. (Generation of the .3d files as required is now done by troggle.)
 
 <h4 id="wallets">Wallets</h4>
-<p><a href="../survey/onlinewallet.html">Online wallets</a> are initially maintained using the <a href="/expofiles/surveyscans/wallets.py">wallets.py</a> script, but troggle also directly imports all the expofiles/surveyscans/ directories of scanned survey notes and produces <a href="/survey_scans/">reports</a> on then.
+<p><a href="../survey/onlinewallet.html">Online wallets</a> are initially maintained using the <a href="/expofiles/surveyscans/wallets.py">wallets.py</a> script, but troggle also directly imports all the expofiles/surveyscans/ directories of scanned survey notes and produces <a href="/survey_scans/">reports</a> on them. There are several bash and python scripts in the  <a href="/expofiles/surveyscans/">surveyscans</a> directory to create wallets for the coming year, and to re-run the wallet processing on all  past years (for when we improve the script). For 2021 we have converted wallets.py to python3, so be careful of older versions which are python2.
 
 
 <h4 id="folk">Folk</a></h4>

handbook/troggle/scriptsother.html

@@ -22,7 +22,7 @@
 <li><a href="scriptscurrent.html#latex">bierbook.tex</a> LaTeX script for generating the bierbook - a new list of names and dates each year
 <li><a href="scriptscurrent.html#latex">seshbook.tex</a> LaTeX script for generating the seshbook - works from the same list of names
 <li><a href="scriptscurrent.html#latex">therionpage.tex</a> LaTeX script and makefile for generating therion-style protractors</li><br />
-<li><a href="scriptscurrent.html#latex">wallets.py</a> generates statuspages and to-do list pages for survey data production.
+<li><a href="scriptscurrent.html#wallets">wallets.py</a> generates statuspages and to-do list pages for survey data production.
 <li><a href="scriptsqms.html">svx2qm.py</a> extracts QMs from the survex files (and  <a href="scriptsqms.html">find-dead-qms.py</a>)
 <li><a href="scriptsqms.html">tablize-qms.pl</a> turns the list of QMs extracted into an HTML file 
 <li><a href="scriptscurrent.html#surface">make_svx.sh</a> generates surface Survex tracks

noinfo/walletscripts/allwallets.sh

@@ -1,6 +1,6 @@
 #/bin/sh
 
-for i in 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019; do 
+for i in 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2021; do 
 #for i in 2016 2017 2018 2019; do 
  echo $i
  cp -p wallets.py $i

noinfo/walletscripts/mkdirs.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# Make the first set of directories for the coming year
+# run from /surveyscans/ and tell it the year, e.g.
+# $ ./mkdirs.sh 2021
+
+if [$1  -eq ""]; then echo -e "mkdirs [year]\nProvide a year as the argument."; exit; fi
+for i in {100..135}; do 
+ ds=${i:1:3}
+ echo mkdir $1/$1"#"$ds
+ mkdir $1/$1"#"$ds
+done

noinfo/walletscripts/wallets.py

@@ -1,32 +1,52 @@
 #!/usr/bin/env python 
-import sys, os, operator, urllib, json, re, time
+import sys, os, operator, urllib.request, urllib.parse, urllib.error, json, re, time
 from datetime import datetime
+from functools import reduce
+from pathlib import Path
 
 # 2017 originally by Martin Green
 # 2018-08-27 edited Philip Sargent
-# 2019-03-02 extended to take command line argument of loser_dir and set mod time of index.html to be same as json file
+# 2019-03-02 extended to take command line argument of loser_dir and set mod time of index.html 
+#   to be same as json file
 # 2019-12-17 extra output of links to troggle-generated trip data
-# 2019-12-31 bits to make website link-checker not barf so much. Added endswith() to .startswith() for notes, elev, plan filenames
+# 2019-12-31 bits to make website link-checker not barf so much. Added endswith() to .startswith() 
+#   for notes, elev, plan filenames
 # 2020-01-21 Now we are using Windows10-WSL1, +links to expedition logbook on every generated page
-# 2020-03-15 Adding timestamp to visible outputs, changing name of produced files to walletindex.html so that contents can be browsed
+# 2020-03-15 Adding timestamp to visible outputs, changing name of produced files to walletindex.html 
+#   so that contents can be browsed
 # 2020-03-15 Added "ignore" to the <year>#00 folder containing scraps - then removed as we do
 # want it to appear in the reports under "UNKNOWN"
+# 2021-04-24 Converted from python2 to python3 - god almighty did I really once think this was an 
+#   acceptable python layout?
+
+'''This stand-alone programe processes all the wallet folders for one year and produces the 
+list of actions that need to be done. 
+
+It produces 
+- an overall summary page for all the wallets in this year
+- a summary page for each wallet
+- a page specific to each person.listing what they need to do across all wallets
+
+It scans the subdirectories only one level deep 
+e.g. we are in /2020/ so it scans /2020/2020#01, /2020/2020#02 et seq.
+
+All the files in one folder must be for only one cave, but in principle coule be for several trips.
+However all the files in one folder should relate to a single survex file (troggle assumes this) and
+a survex file should relate to a single trip (we do this, the Austrians and Germans don't)
+'''
 
 loser_dir = "/home/expo/loser"
 #loser_dir = "/mnt/d/CUCC-Expo/Loser/"  # when running on Win10-WSL1
 #loser_dir = "/media/philip/SD-huge/CUCC-Expo/loser/"  # when running on xubuntu laptop 'barbie'
 
-if len(sys.argv) > 1 :
+# GLOBALS
-	if sys.argv[1] != "":
+wallets_needing_scanning = set()
-		loser_dir = sys.argv[1]
+website_needing_updating = set()
+wallets = [] #need to use wallets as a dict/tuple (id,cave,name) 
+people = {}
+cave = ""
+name = ""
 
-dateTimeObj=datetime.now(tz=None)
-timestamp = dateTimeObj.strftime("%d-%b-%Y (%H:%M)")
-
-print "Loser repo (for svx files) is assumed to be in: " + loser_dir + "/"
-
-drawings_dir =  loser_dir[0:len(loser_dir)-5] + "drawings"
-print "Drawings repo (for drawings files) is assumed to be in: " + drawings_dir + "/"
 
 html_base = "<html><body>%(body)s</body></html>"
 html_year_index = html_base % {"body": "<H1>%(year)s surveys: wallets status</H1>\n<p>List of trips: <a href=\"http://expo.survex.com/expedition/%(year)s\">expedition/%(year)s</a> - troggle-processed .svx files and logbook entries on server</p>\nAs of %(timestamp)s\n<H2>Persons</H2>\n<UL>\n%(persons)s</UL>\n<H2>Wallets</H2>\n<table>%(wallets)s</table>\n<H2>Needing Scanning</H2>\n<UL>\n%(needing scanning)s</ul>\n<H2>Website (Guidebook description) needing updating\n</H2>\n<UL style=\"column-count: 3; \">\n%(website needing updating)s</ul>\n"}
@@ -46,8 +66,6 @@ html_person = html_base % {"body": "<H1>%(person)s</H1>\n<p>List of trips: <a hr
 html_complaint_items = "<li>%(count)i %(complaint)s</li>"
 html_items = "<li>%s</li>"
 
-
-
 blank_json = {
  "cave": "", 
  "date": "", 
@@ -67,20 +85,15 @@ blank_json = {
  "survex not required": False, 
  "website updated": False}
 
+def do_item(year, item):
+    global loser_dir
+    global wallets
+    global people
+    global cave, name
+    global wallets_needing_scanning
+    global website_needing_updating
     
-#need to use wallets as a dict/tuple (id,cave,name) - not sure how.
-wallets = []
-wallets_needing_scanning = set()
-website_needing_updating = set()
-people = {}
 
-#use dir this file is in to get current year
-path,year = os.path.split(os.path.dirname(os.path.realpath(__file__)))
-
-print "Year: " + year
-
-for item in sorted(os.listdir(".")):
-    if os.path.isdir(item) and item != year+"indexpages":
     files = []
     for f in os.listdir(os.path.join(".", item)):
         if f not in ["contents.json", "contents.json~","walletindex.html"] and os.path.isfile(os.path.join(".", item, f)):
@@ -88,7 +101,7 @@ for item in sorted(os.listdir(".")):
     contents_path = os.path.join(".", item, "contents.json")
     #print "Trying to read file %s" % (contents_path) 
     if not os.path.isfile(contents_path):
-            print "Creating file %s from template" % (contents_path) 
+        print("Creating file %s from template" % (contents_path)) 
         json_file = open(contents_path, "w")
         json.dump(blank_json, json_file, sort_keys=True, indent = 1) 
         json_file.close()
@@ -97,7 +110,7 @@ for item in sorted(os.listdir(".")):
     try:
         data = json.load(json_file)
     except:
-            print "FAILURE parsing JSON file %s" % (contents_path)
+        print("FAILURE parsing JSON file %s" % (contents_path))
         # Python bug: https://github.com/ShinNoNoir/twitterwebsearch/issues/12
         raise
     if not data["people"]:
@@ -109,8 +122,8 @@ for item in sorted(os.listdir(".")):
     except:
         wallet, cave, name = "", "", ""
     #print data
-        for k, v in blank_json.items():
+    for k, v in list(blank_json.items()):
-            if not data.has_key(k):
+        if k not in data:
             if k == "cave":
                 data[k] = cave
             elif k == "name":    
@@ -120,7 +133,7 @@ for item in sorted(os.listdir(".")):
             write_required = True
     #print write_required
     if write_required:
-            print "Writing file %s" % (contents_path) 
+        print("Writing file %s" % (contents_path)) 
         json_file = open(contents_path, "w")
         json.dump(data, json_file, indent = 1)
         json_file.close()             
@@ -132,8 +145,10 @@ for item in sorted(os.listdir(".")):
     #make wallet descriptions
     
     #Survex
-        survex_required = (data["survex not required"] and data["survex file"] == "") or \
+    not_req = (data["survex not required"] and data["survex file"] == "")
-                          not (not data["survex not required"] and os.path.isfile(os.path.join(loser_dir, data["survex file"])))
+    req = (not data["survex not required"] and os.path.isfile(os.path.join(loser_dir, data["survex file"])))
+    survex_required = not_req or not req
+
     survex_complaint = ""
     if data["survex not required"] and data["survex file"] != "":
         survex_complaint = "Survex is not required and yet there is a survex file!"    
@@ -183,7 +198,6 @@ for item in sorted(os.listdir(".")):
        person_complaints.append(" description(s) needs writing")
     description_needed = "A description is indicated as being needed, so may need adding into this cave page."
 
-
     #QMS
     if not data["qms written"]:
         complaints.append("The QMs needs writing") 
@@ -216,7 +230,7 @@ for item in sorted(os.listdir(".")):
                                                  "survex": survex_description,
                                                  "complaints": reduce(operator.add, ["<p>" + complaint + "</p>" for complaint in complaints], ""),
                                                  "files": reduce(operator.add, 
-                                                                     [html_wallet_file_entry % {"fileurl": urllib.quote(f), 
+                                                                 [html_wallet_file_entry % {"fileurl": urllib.parse.quote(f), 
                                                                                             "filename": f} 
                                                                   for f
                                                                   in files],
@@ -236,10 +250,40 @@ for item in sorted(os.listdir(".")):
             os.remove(person + ".html")
     if person_complaints:
         for person in data["people"]:
-                if not people.has_key(person):
+            if person not in people:
                 people[person] = []
             people[person].append((item, person_complaints))
             
+def main():
+    global loser_dir
+    global wallets
+    global people
+    global cave, name
+    global wallets_needing_scanning
+    global website_needing_updating
+
+    if len(sys.argv) > 1 :
+        if sys.argv[1] != "":
+            loser_dir = sys.argv[1]
+
+    dateTimeObj=datetime.now(tz=None)
+    timestamp = dateTimeObj.strftime("%d-%b-%Y (%H:%M)")
+
+    print("Loser repo (for svx files) is assumed to be in: " + loser_dir + "/")
+
+    drawings_dir =  loser_dir[0:len(loser_dir)-5] + "drawings"
+    print("Drawings repo (for drawings files) is assumed to be in: " + drawings_dir + "/")
+
+
+    #use dir this file is in to get current year
+    path,year = os.path.split(os.path.dirname(os.path.realpath(__file__)))
+
+    print("Year: " + year)
+
+    for item in sorted(os.listdir(".")):
+        if os.path.isdir(item) and item != year+"indexpages":
+            do_item(year, item)
+            
 
     wallets.sort()
     website_needing_updating = list(website_needing_updating)
@@ -248,7 +292,7 @@ wallets_needing_scanning = list(wallets_needing_scanning)
     wallets_needing_scanning.sort()
 
     person_summary = []
-for person, person_wallets in people.items():
+    for person, person_wallets in list(people.items()):
         complaints = reduce(operator.add, [complaints for wallet, complaints in person_wallets], [])
         complaints_summary = []
         for complaint in set(complaints):
@@ -266,34 +310,34 @@ year_index_file.write(html_year_index % {"year": year, "timestamp": timestamp, "
                                                                                                                          in complaints],
                                                                                                                         "")} 
                                                                 for person, complaints
-                                                            in person_summary.items()], ""),
+                                                                in list(person_summary.items())], ""),
                                              "needing scanning": reduce(operator.add,  [html_year_scanning_entry % {"walletname": wallet, 
                                                                                           "cave": cave,
                                                                                           "name": name,
-                                                                                      "walletindex": urllib.quote(wallet) + "/walletindex.html"} 
+                                                                                          "walletindex": urllib.parse.quote(wallet) + "/walletindex.html"} 
                                                                 for (wallet)
                                                                 in wallets_needing_scanning], ""),
                                              "website needing updating": reduce(operator.add,  [html_year_scanning_entry % {"walletname": wallet,
                                                                                           "cave": cave,
                                                                                           "name": name,
-                                                                                      "walletindex": urllib.quote(wallet) + "/walletindex.html"} 
+                                                                                          "walletindex": urllib.parse.quote(wallet) + "/walletindex.html"} 
                                                                 for (wallet)
                                                                 in website_needing_updating], ""),
                                              "wallets": reduce(operator.add, 
                                                                [html_year_wallet_entry % {"walletname": wallet,
                                                                                           "cave": cave,
                                                                                           "name": name,
-                                                                                      "walletindex": urllib.quote(wallet) + "/walletindex.html", 
+                                                                                          "walletindex": urllib.parse.quote(wallet) + "/walletindex.html", 
                                                                                           "complaints": html_status[survex_required or not plan_scanned or not elev_scanned or description_written] + html_survex_required[survex_required] + html_plan_scanned[plan_scanned] + html_elev_scanned[elev_scanned] + html_description_written[description_written] + html_qms_written[qms_written] } 
                                                                 for (wallet, cave, name, survex_required, plan_scanned, elev_scanned, description_written, qms_written) 
                                                                 in wallets])})
     year_index_file.close()
 
-for person, item_complaint_list in people.items():
+    for person, item_complaint_list in list(people.items()):
         person_file = open(person + ".html", "w")
         person_file.write(html_person % {"person": person, "year": year, "timestamp": timestamp,
                                          "wallets": reduce(operator.add, [html_person_wallet_entry % {"walletname": wallet, 
-                                                                                                "walletindex": urllib.quote(wallet) + "/walletindex.html", 
+                                                                                                    "walletindex": urllib.parse.quote(wallet) + "/walletindex.html", 
                                                                                                     "complaints": reduce(operator.add, 
                                                                                                                          [html_items % complaint 
                                                                                                                           for complaint 
@@ -303,3 +347,7 @@ for person, item_complaint_list in people.items():
                                                                           in item_complaint_list], "")
                                          })
         person_file.close()
+
+
+#if __name__ == "__main__":
+main()