2023-09-11 - mod-fqm-manager

Date

Attendees 

Discussion items

TimeItemWhoNotes
1 minScribeAll

Maccabee Levine will take notes today.  Thanks!

< 5 minBackground/ PurposeAll

TCR-29 was recently submitted for mod-fqm-manager.  The self evaluation for this submission has at least one failed criteria.  The thought was to start conversations about this particular criteria now instead of waiting for the TC evaluation.

We also failed to find a volunteer to serve as evaluator/reviewer for this TCR.  We were hoping that after some discussion about the failed criteria, someone might be more inclined to step forward.

45 min

mod-fqm-manager: Failed Criteria

All

From the self evaluation:

  •  [ ] The module doesn't access data in DB schemas other than its own and public
    • FQM's DB schema reaches into other schemas (for the same tenant) via views, however these views isolate the app itself from accessing other schemas so that it doesn't access other schemas directly. The views also allow us to quickly account for external schemas without code changes.
    • Additionally, these views are intended to be used only on DB read replicas, in order to prevent it from negatively impacting read/write DB performance for other modules.
    • Also, there are alternatives to the DB views, which may be used if implemented, such as harvesting data via Kafka events and putting the results into tables with the same schemas as the views, or connecting the DB to a data warehouse DB and pointing the views at that

Discussion Notes:

  • VBar Started off with idea of encapsulated tables within this module; data harvested and populated as needed, then fqm owns the tables.  Concern about moving all that data around.  Why not just put a view in place, fqm is restricted to those views/tables.
    • fqm powers the lists app.  But other FOLIO parts also need to access data in multiple applications, like delete functionality.  i.e. to delete an item record, check loans, fees/fines.  bulk edit, data import could be improved through this.  central control, permissions, of operational cross-app data
  • Jeremy Huff Materialized views?  Matt Weaver no.  
    • Jeremy Huff From fqm perspective it's healthy separation.  From other modules, less difference – data still being accessed as a normal query.  Materialized views would be a more delineated separation.
    • Matt Weaver We want as live data as we can, not stale.  And significant resource cost from materializing.  Views also provide a functional use, as data changes, supports translation.
  • Tod Olson Workflow: Lists app submits something in FQL, module translates that into an SQL call against the views? Matt Weaver Yes
    • Tod Olson When are views created?  Use case analysis?  
    • Matt Weaver Yes driven by product, from PO.
    • VBar Backend module only, consumed by other applications.  Lists app identifies lists it wants to provide, so fqm provides those.  For delete operation, work off a set of rules for each object that defines if it's safe to delete.  So view would support that.
  • Tod Olson When an object representation / schema changes, how do we learn that the views need to be updated?
    • Matt Weaver Right now, when they break.  Very simple right now, 'select * from table', or some that extract data into a more tabular shape.  Simpler ones will stick around.  Translations on column changes would happen in those simple ones.
    • VBar Does liquibase support notification on schema changes?
    • Matt Weaver To a limited degree, could assert that table exists with given column names.
    • Jeremy Huff Does liquibase do runtime analysis of the views?
    • Matt Weaver Only at its own views/tables, not detecting changes beyond that.  Unless contraints added.
  • Tod Olson Intended for general reporting application or foundation?
    • VBar It has come up that this could provide that.
  • Marc Johnson FQM creates a view.  Wherever that view lives, it grabs data from another schema owned by another module.  Not fussed whether the views are materialized or where the views lived.  This breaks the barrier of private databases per module, now integrating via the DB.
    • VBar What happens when FOLIO supports distributed storage for different applications, i.e. user data in a country-localized system.  This still works because FQM would harvest that data by whatever means is acceptable and place it in the FQM table.  So not breaking each module having its own schema.  But for operational reasons, can access the data more efficiently. 
    • "Module does not access data beyond its own DB schema" not changed.
    • Marc Johnson Not really true.  It grabs modules from another module's database.
    • VBar fqm grabs data from its schema.  That data comes from another module's database.