2022-04-29 - Sys Ops & Management Agenda and Meeting notes
- Ingolf Kuss
- Wayne Schneider
Date
Attendees
- Nassib Nassar
- Ingolf Kuss Tod Olson Florian Gleixner Florian Kreft Ian Walls Nils Olof Paulsson Philip Robinson jroot Steffen Köhler
Goals
Discussion items
| Time | Item | Who | Notes |
|---|---|---|---|
| 5 | Welcome | Ingolf | |
| 55 | FOLIO Reporting Platform LDP 1.x Installation | Nassib Nassar head of research at IndexData Wayne Schneider software/devops engineer at IndexData ----------- metadb : a focus on horizontal scaling. Different databases are not supported by LDP 1.x LDP Query Builder App. An interface for querying data that are in LDP 1.x or metadb. LDLite : a python bases client-based implementation of some pieces of LDP. It uses the FOLIO APIs to extract data. folio-analytics : a repository that contains SQL queries for reporting. Basically created by the FOLIO community Reporting SIG. | L |
LDP 1.x runs out of cron to pull data from FOLIO database. LDP: brute force update. Metadb will support incremental updates. LDP Query Builder: Build a query without having to run SQL. Community derived tables : like views; transforms of the data Database in FOLIO is basically for transaction processing. Usually these are short running queries. In an analytic database there is a short number of queries which can run for a long time. This is called analytical processing. The long running processes shred the performance of the database. Analytical queries should be routed over to another database, e.g. LDP database. How much load does the extract put on the FOLIO database ? – not much load The data transformation happens in the LDP software. Ian Walls: there is a certain scale under which having both a FOLIO DB and an LDP DB is more overhead than it's worth. it's a good solution for larger libraries with more transactions, but it's not a one-size-fits-all. ----- Wayne: Operation of LDP LDP 1.x requires that all data are in one database. ldpmarc is not required folio-analytics: contributed by the comminuty; derived tables LDP Administration Guide in https://github.com/library-data-platform/ldp/blob/main/docs postgres 14.2+ recommended history: version of LDP data from public (schema) tables; created daily; incrementally updated folio_reporting: derived tables You need some kind of scheduling service in order to run LDP. The various parts of LDP. LDP server config: need credentials for the source database and the LDP database ldpmarc is written in Go. The docker container is also published. If you use ldpmarc you will basically re-write all your marc storage. So, there is an additional storage requirement. The initial build can take quite a long time. Ca. 12 h for the initial build for 8.5 bio records. folio-analytics is hosted in github.com/folio-org/folio-analytics Derived tables are contained in the subdirectory sql/. Periodic releases related to the LDP versions. psql runs the scripts. Nassib: https://github.com/folio-org/folio-analytics/blob/main/testall.sh mod-ldp and the LDP Query Builder App (Kiwi+) | |||
Tod : Pain Points Technical needs met when standing up FOLIO installs
|
Action items
- Ingolf Kuss Plan Operational Needs session for WOLFCon 2022