expo/troggle
2
0
Fork 1
mirror of https://expo.survex.com/repositories/troggle/.git synced 2024-11-25 16:51:54 +00:00
Code Issues 19 Projects 3 Activity

JSON cache per python instance - working

Browse Source
This commit is contained in:
Philip Sargent 2023-10-23 20:50:28 +03:00
parent 1694d01536
commit c8c21fbe04
1 changed files with 21 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
core/models/wallets.py
View File

@ -79,6 +79,19 @@ class Wallet(models.Model):
but we may change this if we need to do a Django query on e.g. personame but we may change this if we need to do a Django query on e.g. personame
ManyToMany field uses modern Django: a hidden Class, unlike CaveAndEntrances which is explict and visible. ManyToMany field uses modern Django: a hidden Class, unlike CaveAndEntrances which is explict and visible.
We parse all the JSON on initial reset/import but we only keep data on the Wallet objects that we need for
indexing: people, caves and year. So that we can quickly make reports on e.g. all wallets for a particular cave.
All the other fields in the JSON are parsed and loaded from file only dynamiclally, when a report is beging generated,
but since this only happens for a subset of wallets (e.g. for a specific year) the speed penalty is fine. Indeed it
might be faster overall as db operations in django are a bit slow.
This other data is the stuff which generates the tick-lists.
A trick to minimize the number of times we hit the file to load JSON data is to use a field on every wallet object
called 'JSONdata', but this is not part of the schema and there is no corresponding field in the db. The property
'JSONdata' only lives for as long as the ephemeral python wallet object. We use this to cache the JSON data so that
queries of several different things, e.g. 'name', or 'survexnotrequired', do not repeatedly re-read the JSON
which has not changed.
""" """
fpath = models.CharField(max_length=200) fpath = models.CharField(max_length=200)
@ -86,6 +99,8 @@ class Wallet(models.Model):
walletdate = models.DateField(blank=True, null=True) walletdate = models.DateField(blank=True, null=True)
walletyear = models.DateField(blank=True, null=True) walletyear = models.DateField(blank=True, null=True)
caves = models.ManyToManyField("Cave", related_name="wallets") caves = models.ManyToManyField("Cave", related_name="wallets")
# not yet
#persons = models.ManyToManyField("Person", related_name="wallets")
class Meta: class Meta:
ordering = ("walletname",) ordering = ("walletname",)
@ -99,9 +114,11 @@ class Wallet(models.Model):
Do it every time it is queried, to be sure the result is fresh Do it every time it is queried, to be sure the result is fresh
import DataIssue locally to prevent import cycle problem""" import DataIssue locally to prevent import cycle problem"""
# jsonfile = Path(self.fpath, 'contents.json')
# Get from git repo instead if hasattr(self, "JSONdata"):
print(f'seen it already {self}')
return self.JSONdata
# :drawings: walletjson/2022/2022#01/contents.json # :drawings: walletjson/2022/2022#01/contents.json
# fpath = /mnt/d/EXPO/expofiles/surveyscans/1999/1999#02 # fpath = /mnt/d/EXPO/expofiles/surveyscans/1999/1999#02
fp = Path(self.fpath) fp = Path(self.fpath)
@ -145,7 +162,7 @@ class Wallet(models.Model):
from troggle.core.models.troggle import DataIssue from troggle.core.models.troggle import DataIssue
DataIssue.objects.update_or_create(parser="wallets", message=message, url=wurl) DataIssue.objects.update_or_create(parser="wallets", message=message, url=wurl)
self.JSONdata = waldata
return waldata return waldata
def check_survexlist(self): def check_survexlist(self):