Troggle - development pathways

handbook/troggle/trog2030.html

@@ -10,7 +10,7 @@
 <h1>Troggle in 2025-2030</h1>
 
 <h2>5-Year Plan</h2>
-<p>[Philip Sargent, 1 June 2020. Async updates 14 April 2021]
+<p>[Philip Sargent, 1 June 2020. Async updates 21 April 2021]
 <ul>
 <li>I reckon django has at least another 4-5 years left as a very active project (~2025) and at least a decade or so as a well-maintained project.
 <li>I reckon python has another 10-20 years at least.
@@ -27,10 +27,11 @@ Concurrency is not a problem as all data is read-only (this is not <em>entirely<
 We keep the url-rewriting and html-template things in django.[and migrate the unit-tests (a recent innovation) from django to be run stand-alone.]
 <p>
 This means that the "django-ness" part of troggle becomes quite small.
-The more modules we replace, the easier it becomes for new people to work on it - but also the easier it becomes to migrate it to newer django versions. Or the easier it becomes to move entirely from django to Jinja2 + a URL-router + a HTTP-request/response system.
+The more modules we replace, the easier it becomes for new people to work on it - but also the easier it becomes to migrate it to newer django versions. Or the easier it becomes to move entirely from django to Jinja2 [or Mako] + a URL-router 
+[e.g.  <a href="https://werkzeug.palletsprojects.com/en/1.0.x/routing/">werkzeug</a> or routes] + a HTTP-request/response system.
 The data flow through the system becomes obvious to any python programmer with no django knowledge needed.
 <p>
-[This could be harder than it looks if cross-referencing and pointers between collections become unmaintainable - a risk we need to watch.]
+[This could be harder than it looks if cross-referencing and pointers between collections become unmaintainable - a risk we need to watch. But other people are using Redis for this sort of thing. ]
 <p>
 Being memory-resident, we get a 20x speed increase. Which we don't need.
 
@@ -46,14 +47,30 @@ We also use a noSQL db with a direct and easy mapping to python collections. The
 <a href="https://en.wikipedia.org/wiki/Zope_Object_Database">Zope Object Database</a> (ZODB). MongoDb is famous and programmers may want to work on it to get the experience, but ZODB is much closer to python. But ZODB is now rather old, and the Django package django-zodb has not been updated for 10 years. And MongoDb has a bad impedence mismatch: <a href="https://daniel.feldroy.com/when-to-use-mongodb-with-django.html">Short answer is you don't use MongoDB with Django</a> which creates a lot of extra pointless work. If we ever need atomic transactions we should use a database and not try to fudge things ourselves, but not either of those.
 [This needs to be explored, but I suspect we don't gain much compared with the effort of forcing maintainers to learn a new query language. Shelve() is already adequate.]
 
+<h3>Option 2B</h2>
+<p>We migrate to an <a href="https://www.fullstackpython.com/sanic.html">"improved Django"</a> or a "Django-lite". Django is a massive system, and it is moving with agility towards being more asynchronous, but there are already competing projects which do much the same thing, but in a cleaner way and (being 15 years younger) without the historical baggage and cutting out a lot of now-uneeded complexity. This looks like being a very hopeful possibility. <a href="https://sanicframework.org/en/">Sanic</a> and <a href="https://pgjones.gitlab.io/quart/">Quart</a> look like being the first of a new generation.
+<p>The driver for these new Django clones is the 
+<a href="http://masnun.rocks/2016/11/17/exploring-asyncio-uvloop-sanic-motor/">asynchronous capabilities</a> in python 3.4 and the 
+<a href="https://www.encode.io/articles/hello-asgi">ASGI interface</a> for web workers which replaces WSGI. These will have much the same effect on python as Node.js had on JavaScript. 
+<p>But we should not be in too much of a hurry. It will take Sanic (and similar) years to get to a state where things don't 
+break between versions horribly every 6 months. We lived through 
+<a href="https://en.wikipedia.org/wiki/Django_(web_framework)#Version_history">that nastiness</a> 
+with Django 2006-2016 and we don't want to fall into the same mess again. Django has only just got sensibly mature 
+and has stopped breaking so much with every new release. 
+<p>A real architectural option would be to move to <a href="https://redis.io/">Redis</a> as an in-memory database 
+instead of using MariaDB. We don't need a fully-fledged database and these new frameworks are less closely tied to 
+having a SQL database and object-request broker (ORM) than Django is. This ties in neatly which Option 2 above: 
+reduce our use of the database functions within Django to vanishing point.
+<p>ASGI, which Dango supports from v3.2 too, has the interesting effect that we no longer need a webserver like Apache or nginx to buffer requests. We can use very lightweight <a href="https://www.uvicorn.org/">uvicorn</a> instead. 
+
 <h3>Option Zero</h2>
 <div style="color:blue">
 <p>We need to de-cruft troggle *first*: remove 'good ideas' which were never used, trim redundancies and generally reduce it in size significantly.
 <p>
-We should have a good look  at modifying everything so that we do not read in every survey station. This means making a list of functions that we will drop and some we replace by parsing cavern output and some we calculate during importing/reading svx files.
+We should have a good look  at  making a list of functions that we will drop and some we replace by parsing cavern output and some we calculate during importing/reading svx files.
 
 <p>
-Documentation is the key to keeping troggle in a state where someone can pick it up and do a useful week's work, e.g. extracting the parsed logbooks to use shelve() storage throughout instead of SQL. The next time someone like Radost comes along during the next 5 years we want to be able to use them effectively.
+Documentation and a working list of on-going programming projects is the key to keeping troggle in a state where someone can pick it up and do a useful week's work, e.g. extracting the parsed logbooks to use shelve() storage throughout instead of SQL. The next time someone like Radost comes along during the next 5 years we want to be able to use them effectively.
 </div>
 <h3>Things that could be a bit sticky 1 - multi-user safety</h3>
 <p>Multi-user synchronous use could be a bit tricky without a solid multi-user database sitting behind the python code. So removing all the SQL database use may not be what we want to do after all.
@@ -71,7 +88,7 @@ from single-threaded WSGI to asynchronous ASGI</a> began with v3.0 and for 'view
 This makes the server more responsive, 
 but doesn't really change anything from the perspective of our need to stop users overwriting each others' work. If we just store 
 everything in in-memory dictionaries we may need to write our own asyncio python to do that synchronization. That would be a Bad Thing as 
-we are trying to make future maintenance easier, not harder.
+we are trying to make future maintenance easier, not harder. But it looks like Redis could be the solution for us.
 
 <h3>Things that could be a bit sticky 2 - front-end code</h3>
 <p>There is not yet a front-end (javascript) framework on the client, i.e. a phone app or webpage, which is stable enough for us to commit