| 45 min | mod-fqm-manager: Failed Criteria | | From the self evaluation: - [ ] The module doesn't access data in DB schemas other than its own and public
- FQM's DB schema reaches into other schemas (for the same tenant) via views, however these views isolate the app itself from accessing other schemas so that it doesn't access other schemas directly. The views also allow us to quickly account for external schemas without code changes.
- Additionally, these views are intended to be used only on DB read replicas, in order to prevent it from negatively impacting read/write DB performance for other modules.
- Also, there are alternatives to the DB views, which may be used if implemented, such as harvesting data via Kafka events and putting the results into tables with the same schemas as the views, or connecting the DB to a data warehouse DB and pointing the views at that
Discussion Notes: - VBar Started off with idea of encapsulated tables within this module; data harvested and populated as needed, then fqm owns the tables. Concern about moving all that data around. Why not just put a view in place, fqm is restricted to those views/tables.
- fqm powers the lists app. But other FOLIO parts also need to access data in multiple applications, like delete functionality. i.e. to delete an item record, check loans, fees/fines. bulk edit, data import could be improved through this. central control, permissions, of operational cross-app data
- Jeremy Huff Materialized views? Matt Weaver no.
- Jeremy Huff From fqm perspective it's healthy separation. From other modules, less difference – data still being accessed as a normal query. Materialized views would be a more delineated separation.
- Matt Weaver We want as live data as we can, not stale. And significant resource cost from materializing. Views also provide a functional use, as data changes, supports translation.
- Tod Olson Workflow: Lists app submits something in FQL, module translates that into an SQL call against the views? Matt Weaver Yes
- Tod Olson When are views created? Use case analysis?
- Matt Weaver Yes driven by product, from PO.
- VBar Backend module only, consumed by other applications. Lists app identifies lists it wants to provide, so fqm provides those. For delete operation, work off a set of rules for each object that defines if it's safe to delete. So view would support that.
- Tod Olson When an object representation / schema changes, how do we learn that the views need to be updated?
- Matt Weaver Right now, when they break. Very simple right now, 'select * from table', or some that extract data into a more tabular shape. Simpler ones will stick around. Translations on column changes would happen in those simple ones.
- VBar Does liquibase support notification on schema changes?
- Matt Weaver To a limited degree, could assert that table exists with given column names.
- Jeremy Huff Does liquibase do runtime analysis of the views?
- Matt Weaver Only at its own views/tables, not detecting changes beyond that. Unless contraints added.
- Tod Olson Intended for general reporting application or foundation?
- VBar It has come up that this could provide that.
- Marc Johnson FQM creates a view. Wherever that view lives, it grabs data from another schema owned by another module. Not fussed whether the views are materialized or where the views lived. This breaks the barrier of private databases per module, now integrating via the DB.
- VBar What happens when FOLIO supports distributed storage for different applications, i.e. user data in a country-localized system. This still works because FQM would harvest that data by whatever means is acceptable and place it in the FQM table. So not breaking each module having its own schema. But for operational reasons, can access the data more efficiently.
- "Module does not access data beyond its own DB schema" not changed.
- Marc Johnson Not really true. It grabs modules from another module's database.
- VBar fqm grabs data from its schema. That data comes from another module's database.
- Marc Johnson In original conversations, the intent was for data to be private to that module.
- VBar Some problems in FOLIO need to consider data from different places. I.e. delete operations. If we had a table with the correct flags, and ran an external app out of FOLIO to populate the data needed? Marc Johnson Yes.
- VBar If it puts data where fqm can access it?
- Marc Johnson That's what we have previously agreed is the way FOLIO can do that. Copying of data is an inevitable thing if we want to preserve those boundaries & create mechanisms to cross them. Originally FOLIO used API-based integration, then message-based integration (Kafka). This module effectively introduces shared DB integration.
- VBar Disagree. It's an operational concern with how the data gets there unbeknownst to FOLIO. Implementation choice on how data flows. Good to look at original architectural intent. Also recognize performance impact, needs that have come up.
- Marc Johnson Disagree. Any time a module accesses another database's private data, that violates the intended constraint. Requires harvester with escalated DB privileges. Understand operational and performance concerns, they are legitimate. But this changes the architecture.
- Jeremy Huff Sympathetic to Marc Johnson 's perspective. Appreciate candor in not checking that box on the module criteria. Views create an explanation that you can lean on, but for practical purposes this module does access data outside its schema to do the tasks at hand. Does FQM's module descripter express a dependency on module for which it has a view?
- Matt Weaver No, to avoid circular dependencies.
- Jeremy Huff But fair to say FQM does "depend" on those other modules.
- Matt Weaver Yes. Optional dependency.
- Jeremy Huff Would it operate if those modules were not available?
- Matt Weaver Yes. Only creates the views if their data is available.
- Ian Walls I have run something similar to fqm in production, how we do reporting at ByWater.
- To Tod Olson 's point, yes used for reporting. Schema changes happen, doesn't come up often, requires human testing.
- Agree with Marc Johnson that it's a fundamental change to the architecture, but think it's a good change. Forcing us to operate at a RESTful level not DB level forces problems with few benefits.
- "Modules don't own data, libraries do." Their stuff. Getting to it is what's important, not what box it's dumped into.
- Maccabee Levine read-only ? Yes
- Maccabee Levine If we did an RFC, would look at risks and alternatives. Do we have a process here? Identifying concrete risks per Marc Johnson 's concerns. And identifying alternatives.
- Jeremy Huff Useful if this had been an RFC to deliberate the specific problem at hand. This is worth talking about in context of architectural discussion. Wish it had been approached that way from the beginning, not in context of TCR.
- VBar What is the architectural change? Originally was about harvesting data externally, now views.
- Jeremy Huff Maybe in FOLIO it is ok for modules to access other modules' data in a given manner, addressing concerns.
- Marc Johnson Architecture establishes constraints to guide implementation decisions. So architectural decision would be changing it to allow this. Likely not a special case for this module, that's worse than deciding for good reasons to allow this DB integration to avoid the performance costs of other approaches. More up-front way of making that decision vs. for this module as exception.
- Tod Olson Trying to find a practical solution without chucking concerns out the window. DB distribution of objects across the system would be a concern. But the current mechanisms do not scale. We can cast this as fqm asking for an exception. Alternatively can cast as fqm being the locus where this cross-module crossing of data can happen; the service provider for that. It doesn't get us away from the controversy. Really not in favor of a DB free-for-all.
- Marc Johnson It's free-for-all or special case. Need to decide whether we are deciding this for Poppy.
- Jeremy Huff I would like to see an RFC that outlines the problem and solution in general terms. Module should go through the process as-is, current criteria.
- Craig McNally Similar to what we did on mod-settings. Your module passes current criteria, but also spin up the RFC for architecture.
- VBar If I had submitted the self-assessment, would have checked that box. Doesn't fail the letter of the criteria. But if we do an RFC, what we are discussing is a module that creates and owns these tables, and a module can be turned on or not, not sanctioned by FOLIO project, but you can choose to do this, swapping out tables as views.
- Craig McNally next steps: continue the conversation on Wednesday, what do with with this TCR and next steps in general.
|