2022-03-18 - Sys Ops & Management SIG Agenda and Notes
- Ingolf Kuss
- Philip Robinson
Date
Attendees
Nils Paulsson
Meeting Link
- https://zoom.us/j/591934220
- Password: folio-lsp
Goals
Talk about transaction control in FOLIO
Discussion items
| Time | Item | Who | Notes |
|---|---|---|---|
| Welcome | Ingolf | ||
| 60 | Let's talk about transaction control in FOLIO | All | Let's talk about transaction control in FOLIO. It's a core functional issue. Without transaction control, we are likely to see a lot of database corruption. I have found two Wiki pages which have been recently added, so there is some work or some plans going on: https://wiki.folio.org/display/~mage.air/Data+consistency+options Also, there is a Proposal for optimistic locking, which has recently undergone some discussion, and has been last edited in November 2021: Let's have a look at this and talk about this. Optimistic locking will address some database concurrency problems, but not all transaction failures. Ian Walls noted that Koha hasn't had locking for around 20 years, and data conflicts are very rare. Dale had mentioned challenges inherent in the orchestration of asynchronous API calls and their impacts on the data integrity when thinking about locking. Florian Gleixner : many modules are involved with managing data. The data go to different tables in Postgres. If one process fails, then there is the risk of orphaned records and data integrity problems. A transactional approach can help by being able roll back a set of data changes if any one of them fails. Ian Walls : maybe FOLIO's architecture was designed in such a way that it varies from users' expectations based on other systems they've worked with. So it would be a major development effort to implement optimistic locking. jpnelson : New York Times' large archive of articles uses Kafka's transaction log, as a history of changes, to manage updates and rollbacks of a giant data set: https://open.nytimes.com/publishing-with-apache-kafka-at-the-new-york-times-7f0e3b7d2077 Ian Walls noted that the Technical Council plans to review whether Kafka should be "officially blessed" as an integral part of FOLIO. An advantage could be better management of message chains among modules. Ingolf Kuss : optimistic vs pessimistic locking - what is the distinction? Ian Walls : optimistic factors in the versions of changes and reacts to conflicts as asynchronous updates come in. Business logic can help merge changes. Changes need to be made only to the most up-to-date copy. Optimistic is a better fit for the way FOLIO works with asynchronous API calls. Philip Robinson : pessimistic locking prevents any changes while someone is working on a record. Ian Walls : FOLIO would benefit from a common workflow engine / message broker. jpnelson : Stanford uses Airflow for that. Philip Robinson : Cornell uses Prefect as well as Dell Boomi. Ian Walls : ByWater uses N8N workflow tool. Some FOLIO code talks directly to Kafka rather than using mod-pubsub. The SIG generally agrees that optimistic locking is worthy of doing, but resourcing it is difficult. |
Installation/Configuration challenges | Ingolf Kuss noted some issues with ordering deployment of back end modules vs front end modules. In Kiwi, when you enable the front end modules, they tell you they need back-end modules that are not bundled with the package. Philip Robinson mentioned a problem with Kiwi Hot Fix 2; agreement names and descriptions disappeared (not clear whether it's a front end or back end glitch). MinIO server is required for mod-data-export. Florian Gleixner containerized minIO. |