2023-02-01 Meeting notes

Marc Johnson is next, followed by Ingolf Kuss 

All

All

Tod Olson advised that the Technical Council Goals and Objectives group intends to have a draft available to the TC next week

Marc Johnson stated that the Translations group has been on hiatus since before the turn of the year. And suggested that the group might conclude, once it has summarised it's thoughts, in anticipation for product decisions in this area. Zak Burke can likely advise better the groups' current situation

Craig McNally suggested that we need to find a way to wrap up some of these working groups, in particular the longer running ones

Jeremy Huff advised that it is possible that the Breaking Changes group could have an RFC to present (on the definition of breaking changes) to the TC next week

Jeremy Huff advised that the TCR improvement group spent last week putting together improvements to (try to) close the hello world loophole in the current process. He asked how the group should bring the suggested changes back to the TC

Craig McNally suggested they be shared with the TC separately to review and will be discussed next week

Jenn Colt advised that the Charter group produced another draft and asked all TC members to review this prior to discussing it further at next weeks meeting. She described there is already some good conversation happening on the draft and asked that folks raise concerns, especially anything that might block approval, prior to the next week's meeting

Olamide Kolawole advised that the draft review group met yesterday to review the Reducing Kafka Partitions RFC and will be updating the RFC following the discussion. He advised that the group is advocating that FOLIO provide the ability to support either single topic per tenant or mixed tenant topics. The group will meet again next week

Craig McNally suggested that we have one week remaining for this stage of the process. He asked if the group can be finished within that timeframe?

Olamide Kolawole suggested that the extra week will likely be needed, and the group was working on the clock starting when the group was formed

Craig McNally acknowledged that reducing the ambiguity of when the clock starts for these phases and how long each phase needs to be should be addressed during the next review of this process

Martin Tran 

Martin Tran presented the proposed ADR for Splitting Database Read / Write Traffic along with an accompanying .

He advised that the FSE team and Core Platform team implemented this work in RMB during the Morning Glory and Nolana releases, during which most RMB storage modules were upgraded to support the proposed mechanism (meaning that this design is already in use in production). 

These changes were performance tested using circulation and data import flows. He advised that CPU use is more evenly split between read and write nodes, reducing the pressure on the write node when under higher loads (25 concurrent use). This also led to a reduction in CPU usage in the ("back up the chain") modules making the database requests. During circulation only testing, whilst there was a slight deduction in response times for check in, responses times for check out increases in some cases. However, once data import and circulation work flows were running concurrently, the read / write split significantly reduced response times over a single node.

There is remaining work to fully support this across FOLIO:

• Use unambiguous (explicitly either read or write) operations in RMB modules, that are already providing support for this. Only one module has been updated to stop using ambiguous operations (where RMB does not know the whether the module intends to perform a read or write).
• Write in-module automated testing for using separate read and write nodes. This work is waiting for tooling support within RMB
• Functionally validate modules using separate read and write nodes (using integration or smoke tests)
• Change other modules to support this mechanism. This will include updating other shared tooling, e.g. folio-spring-base

Ingolf Kuss asked if this approach only works if the database is clustered? Martin Tran advised that is the case, however there is no advice given for how to achieve that, however EBSCO use AWS RDS for this purpose.

Ingolf Kuss asked if this was specific to a module and required configuration? Martin Tran advised that this involves specifying environment variables for the read and write node. Marc Johnson advised that this is specific to each module because currently only some modules support this mechanism. And that identifying which modules support this is an exercise folks need to do

Craig McNally asked if this was an opt-in model (as the single node configuration is still supported)? Martin Tran confirmed that is the case. Only operators wanting to use the split have to identify which models support this.

Tod Olson asked where do we direct folks for documentation on how to use / configure this mechanism? Martin Tran advised that the details of the current implementation are available in the RMB readme. And that configuration of the cluster is an exercise for system implementors / operators. Marc Johnson  suggested that there are different kinds of documentation that needs to be provided: architectural; per-module configuration; operations documentation

Jeremy Huff asked if this has only been implemented for AWS RDS instances? Martin Tran confirmed that this has only been tested on AWS RDS and that how database nodes synchronise data is outside of the scope of this work

Jeremy Huff requested that the ADR provide more details of the solution (separate to the RMB implementation). Martin Tran agreed to add that into the ADR

Maccabee Levine suggested that there is very little downside if other modules don't adopt this approach, and asked what are the downsides if other modules do adopt this approach? Martin Tran suggested that one downside is that RMB creates separate connection pools for read and write connections, which could use more memory in the database. Marc Johnson suggested some of the other trade offs involved in this could be: doubling the database connections (and port usage) for each module; this architectural pattern introduces the potential for stale reads; doubling the potential configurations (similarly to the approach mentioned in the discussion above about the Kafka RFC) of the system which may need testing.

Tod Olson asked which RMB storage modules have been upgraded to support this mechanism? Martin Tran advised that over 40 of the modules have been upgraded to use the version of RMB that supports this mechanism, however only one has been updated to use unambiguous operations. Marc Johnson clarified that most RMB modules have been updated to (partially) support this mechanism. And shared his understanding that none of these modules having automated tests that exercise this mechanism during builds and that many of them have some use of ambiguous (RMB does not know if they are read or write) operations that need to be changed, meaning there is outstanding work in almost all of the modules that have already been upgraded.

Jeremy Huff asked if a system configured for a single read / write node would also double the amount of connections needed? Martin Tran advised that in this case, there is only a single connection pool (and thus the number of connections remains as before)

Jeremy Huff asked what are the implication for running in a hybrid state (where some modules support read / write traffic splitting and others do no not)? Martin Tran suggested that each module that is moved to using separate read and write nodes reduces the load on the write node. Marc Johnson advised that one of the primary trade offs for having different tooling support different configuration options in this area is increased operations complexity for developers and operations folks needing to understand that different parts of the system are using read and write nodes and others are only using the write node and cognitive load of using different architectural patterns

Craig McNally asked what the next steps are? Maccabee Levine suggested that Martin Tran update the ADR to reflect some of the questions / concerns raised during the conversation. Craig McNally advised that after those changes are made, the TC will reach a conclusion in this ADR either via Slack or in a future meeting.

Craig McNally 

• TC dedicated discussion:  06 Feb 2023 11:00 AM ET
  • Technical Goals and Objectives (Tod Olson)

Craig McNally /All

Craig McNally /All

From Craig McNally in #tech-council:

This was brought to my attention earlier today...
https://blog.nlnetlabs.nl/open-source-software-vs-the-cyber-resilience-act/
While it's still just a proposal, I think FOLIO should keep an eye on, and maybe even try to get ahead of in anticipation of this.  I will add it to the agenda for next week's meeting.  This is a short read that does a decent job of laying it all out.  Please take a look prior to next Wednesday.  Thanks! 

• Have folks had a chance to read through any of this?
• What, if anything do we think the TC should do about this?
  • Raise awareness among other councils?
  • Seek legal advice in anticipation of this being passed?
  • Is there anything else we want to do to be more prepared for this in the event it does get passed?

Today: Deferred

All / Ian Walls 

From last week:

• Ease of installing/deploying FOLIO - Ian Walls , Marc Johnson , Jeremy Huff
  •  Primary task the Tc would take on by making FOLIO easier to get up and running. Would also reduce AWS costs so that the money coming from Membership groups can be flowed to other aspects of FOLIO. Tc is the best equipped group to decide on how to make installing and deploying Folio easier and cheaper.
  • Craig McNally - Brainstorming open ended session with Ian Walls and then discuss further before or after WOLFcon depending on the brainstorming session. Ian Walls and Tod Olson to frame the topics of discussion for the brainstorming. 

Today:

• Probably defer, but keep on the agenda so we don't lose track of this...

All / Ian Walls 

Slack discussion:  Revisiting FOLIO Governance 

• • Ian Walls - should be best discussed in cross council meeting possibly at WOLFcon. Idea to was bring this up at a high community level not necessarily the Pc or TC. Doesn't need to be on TC agenda next week. Aspects to be discussed at WOLFcon.
  • See also:  messages to PC and CC council channels