Versions Compared

Old Version 12

changes.mady.by.user Julian Ladisch

Saved on

compared with

New Version 13

changes.mady.by.user Julian Ladisch

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Attendees 

Discussion items

Next Steps

Further Discussion:The TC agreed to revisit this discussion in the next meeting, with members encouraged to review past work and consider whether they

Status of this?

Hold until after Aug 2.

In the Sys Ops SIG meeting the topic of Reference Data Upgrades came up. The SIG thinks that the solution of this problem for mod-inventory-storage is not enough, but that this problem needs to be solved in a general way, for all modules.

There has been a long discussion 3-4 years ago about how FOLIO should handle reference data upon upgrades. See these links for background:

Previous Notes:

Background and Problem Statement

  • The issue at hand involves the handling of reference data during system upgrades. In previous upgrades, some institutions experienced data loss where customized reference data was overwritten by default settings during the upgrade process.
  • This problem was particularly highlighted in recent upgrade cycles, prompting a need for a more robust solution to preserve institution-specific customizations.

Existing Solutions and Approaches

  • Circulation Module Example:

    • The circulation module had addressed this issue by implementing a solution where reference data is only installed during the initial setup and skipped in subsequent upgrades unless new data is introduced in the upgrade. This approach prevents the overwriting of existing customized data.
    • The TC discussed whether this solution could be standardized and applied across other modules facing similar issues.

  • Challenges with Current Approach:

    • While the circulation module's solution is effective for preserving existing data, it does not account for scenarios where the reference data schema changes in new versions (e.g., adding new fields or making structural changes). In such cases, simply skipping the installation might not be sufficient, and more complex handling might be required.

Discussion Points

  • Standardization Across Modules:
    • The TC debated whether a similar approach should be mandated across all modules to ensure consistency and prevent data loss during upgrades. However, it was noted that this might require module-specific adjustments, depending on how each module manages its reference data.
  • Need for a Broader Solution:
    • There was recognition that a broader, more flexible solution might be necessary to handle various scenarios, such as changes in reference data structures. This could involve more sophisticated checks during the upgrade process to determine whether and how data should be updated or preserved.

Potential Actions

  • Subgroup Formation:
    • The idea of forming a new subgroup to focus on developing a comprehensive strategy for reference data handling during upgrades was floated. This subgroup could explore creating an RFC (Request for Comments) to propose a standardized approach across the project.
  • Review of Previous Work:
    • It was suggested that before moving forward, the TC should review previous efforts and discussions on this topic, including past proposals and proof-of-concept work that may have stalled due to resource constraints. This review would help ensure that any new efforts build on previous work rather than starting from scratch.
Zoom Chat
TimeItemWhoNotes
1 minScribeAll

Julian Ladisch is next, followed by Ingolf Kuss (Julian and Ingolf have swapped)

Reminder:  Please copy/paste the Zoom chat into the notes.  If you miss it, this is saved along with the meeting recording, but having it here has benefits.

5-10 minLiaison Updates

Maccabee Levine

Tod Olson

Jakub Skoczen

Craig McNally

Jenn Colt 

  • CC: Maccabee Levine
    • Continued discussion about AWS cost increases, especially due to Eureka.  ACRG will present at next CC meeting on 9/9.
    • Agenda(s) for CC and cross-council meetings at WOLFcon.
  • PC: Tod Olson
    • Discussion of module review process, moved out of pilot into accepted
  • RMS Group:
    • CSP release cadence: Release on demand (not fixed 2 week or 4 week cadence)
    • Stay on Spring Boot 3.3 for Ramsons
      • Upgrade to 3.4 in Sunflower. This means we'd be unsupported for some period of time. If we're lucky and there aren't any Critical/High security vulnerabilities in these frameworks, all is good. However, if there are, we would be in the position where now we need to get teams to drop what their doing and upgrade unexpectedly, backporting the changes to Ramsons. 
      • Upgrade Ramsons to 3.4 if there's a security vulnerability between 2025-05-23 (End of Support of Spring Boot 3.3) and Trillium release Q4 2025 (= End of Support of Ramsons); this results in a huge CSP and requires extensive testing

    • TC - review possibility to get enterprise support from Spring Framework

    • Reserve time in Trillium to upgrade to version 3.5

      • This requires (milestone or) release candidate versions of Spring Boot for FOLIO bug fest if Trillium bug fest starts November 2025 or earlier.
  • Security Team: 
    • ongoing triage of potential vulnerabilities
    • Jiras gets unembargoed when closed (when fixed or security group approves as not affected).
  • Tri-council Application Formalization:  
    • Discussion of impact of dual okapi/eureka envs on cost and what exit from okapi in community envs would require
    • Discussion of application spreadsheets created by Charlotte and Kirstin
    • Discussion of WolfCon session
1 minUpcoming MeetingsAll
  •  - Regular TC Meeting
  •  - Dedicated Discussion: Refine the OST process to ensure better coordination between SysOps and the TC
  •  - US holiday (regular TC meeting moved to Wed)
  •  - Regular TC Meeting
  •  - Regular TC Meeting
  •  - Dedicated discussion: Continuation of architectural decision-making discussion
5 minTCR Board ReviewAll
  • mod-marc-migrations working on game plan
  • tcr-42 waiting on PC approval
  • tcr-45 moved to Ingolf Kuss via rotation
5 min

Technical Council Sub-Groups Updates

All


1 minRFCs

All

Reminder(s)

  • Review of open items, such as the need for decision log records and the application of the "Go" RFC.
1 minDecision LogAll

Need to log decisions for the following:  (see above)

  • Discussions on the necessity of a Wednesday meeting to address the OST process and involvement of Sysops.
25 min

Officially Supported Technologies (OST)

All

Check Recurring Calendar

  • Discuss message in #tech-council from Julian Ladisch...
    • Spring Boot 3.4 and Spring Framework 6.2 for Ramsons
    • 3.3 ends before Ramsons support period ends, using 3.4 would prevent this. Right now there is a milestone version, RC by bugfest and final in November. Would need to be communicated early to the development groups. But the final would be after code freeze, would have to use RC.
    • Would upgrade to milestone version, around  bugfest go to RC then after bugfest to release, so three times, teams might opt to not do all three
    • CSP might be easier to handle for teams than doing it during bugfest
    • What would happen in the CSP? 3.3 to 3.4. Could be higher risk to do a minor version upgrade that requires small changes
    • Going to GA on an RC of spring boot seems odd. but by the time we get to GA both versions unsupported
    • RC probably has the majority of changes that the spring boot GA would have. If we do non-RC and aCSP then all the spring boot changes go out in a CSP, which is a lot of new spring boot stuff to not be tested
    • Incrementing a point in a CSP seems like too much
    • When we deploy CSP to bugfest we do smoke test
    • Some modules on separate branch have upgraded as PoC
    • Might want to talk with RMS more about CSP approach
    • Frame options for RMS meeting on the 26th
  • Work on Sunflower page on Monday the 26th
    		Time PermittingReference Data Upgrade

    Ingolf Kuss

    Jason Root

    Craig McNally

    Marc Johnson

    Julian Ladisch

      • The Spring Boot support period of 12 months for a single minor release doesn't align with FOLIO's support period of ~ 12 months.
      • Regarding Backend "First Party Libraries / Frameworks": A library/framework need to support only one of the versions allowed in the "Third Party Libraries / Frameworks" list. This should be clarified in the description.
      • Adding folio-s3-client to first party libraries and the AWS and MinIO clients to third party libraries. folio-s3-client is an abstraction for the AWS and MinIO client.
    Time PermittingReference Data Upgrade

    Ingolf Kuss

    Jason Root

    Craig McNally

    Marc Johnson

    Julian Ladisch

    Status of this?

    Hold until after Aug 2.

    In the Sys Ops SIG meeting the topic of Reference Data Upgrades came up. The SIG thinks that the solution of this problem for mod-inventory-storage is not enough, but that this problem needs to be solved in a general way, for all modules.

    There has been a long discussion 3-4 years ago about how FOLIO should handle reference data upon upgrades. See these links for background:

    Previous Notes:

    Background and Problem Statement

    • The issue at hand involves the handling of reference data during system upgrades. In previous upgrades, some institutions experienced data loss where customized reference data was overwritten by default settings during the upgrade process.
    • This problem was particularly highlighted in recent upgrade cycles, prompting a need for a more robust solution to preserve institution-specific customizations.

    Existing Solutions and Approaches

    • Circulation Module Example:

      • The circulation module had addressed this issue by implementing a solution where reference data is only installed during the initial setup and skipped in subsequent upgrades unless new data is introduced in the upgrade. This approach prevents the overwriting of existing customized data.
      • The TC discussed whether this solution could be standardized and applied across other modules facing similar issues.

    • Challenges with Current Approach:

      • While the circulation module's solution is effective for preserving existing data, it does not account for scenarios where the reference data schema changes in new versions (e.g., adding new fields or making structural changes). In such cases, simply skipping the installation might not be sufficient, and more complex handling might be required.

    Discussion Points

    • Standardization Across Modules:
      • The TC debated whether a similar approach should be mandated across all modules to ensure consistency and prevent data loss during upgrades. However, it was noted that this might require module-specific adjustments, depending on how each module manages its reference data.
    • Need for a Broader Solution:
      • There was recognition that a broader, more flexible solution might be necessary to handle various scenarios, such as changes in reference data structures. This could involve more sophisticated checks during the upgrade process to determine whether and how data should be updated or preserved.

    Potential Actions

    • Subgroup Formation:
      • The idea of forming a new subgroup to focus on developing a comprehensive strategy for reference data handling during upgrades was floated. This subgroup could explore creating an RFC (Request for Comments) to propose a standardized approach across the project.
    • Review of Previous Work:
      • It was suggested that before moving forward, the TC should review previous efforts and discussions on this topic, including past proposals and proof-of-concept work that may have stalled due to resource constraints. This review would help ensure that any new efforts build on previous work rather than starting from scratch.

    Next Steps

    • Further Discussion:
      • The TC agreed to revisit this discussion in the next meeting, with members encouraged to review past work and consider whether they are interested in participating in a potential subgroup. The goal would be to establish a clear path forward for managing reference data during upgrades across all modules.

    Today:

    • ...
    		NA
      • upgrades across all modules.

    Today:

    • ...
    NAZoom Chat


    		17:01:56 Von Jenn Colt an Alle:
    https://folio-org.atlassian.net/wiki/spaces/TC/pages/418742317/2024-08-26+Meeting+notes
    17:23:01 Von Jenn Colt an Alle:
    Maybe that’s the sales mechanism for enterprise support!
    17:24:12 Von Julian Ladisch an Alle:
    There's a 6 month overlap of two consecutive minor releases of Spring Boot: https://spring.io/projects/spring-boot#support
    17:38:33 Von Craig McNally an Alle:
    Thanks for clarifying @Taras
    17:41:11 Von Taras an Alle:
    This is
    https://github.com/folio-org/folio-s3-client
    17:50:48 Von Root, Jason M an Alle:
    Inside Tamu Lib’s MinIO instance - no reference directly to S3 API versions. This is all I see:
    Version2021-06-17T03:17:14Z
    PlatformHost: minio-0 | OS: linux | Arch: amd64
    RuntimeVersion: go1.16.5 | CPUs: 8
    17:54:54 Von Julian Ladisch an Alle:
    Jenkins slave docker: https://github.com/folio-org/folio-tools/tree/master/jenkins-slave-docker
    17:55:40 Von Maccabee Levine an Alle:
    I do see Jenkins referenced in what Index Data proposed to support. Details IDK. https://folio-org.atlassian.net/wiki/spaces/CC/pages/68321283/2024-03-11+Community+Council+Meeting+Notes


    Topic Backlog

    Decision Log ReviewAll

    Review decisions that are in progress.  Can any of them be accepted?  rejected?

    Translation SubgroupAllSince we're having trouble finding volunteers for a subgroup, maybe we can make progress during a dedicated discussion session?
    Communicating Breaking ChangesAll

    Currently there is a PoC, developed by Maccabee Levine, of a utility to catalog Github PRs that have been labeled with the "breaking change" label. We would like to get developer feedback on the feasibility of this label being used more often, and the usefulness of this utility. 

    Officially Supported Technologies - UpkeepAll

    Previous Notes:

    • A workflow for these pages. When do they transition from one state to another. Do we even need statuses at all ?

    Stripes architecture group has some questions about the Poppy release.

    Zak: A handshake between developers, dev ops and the TC. Who makes that decision and how do we pass along that knowledge ? E.g. changes in Nodes and in the UI boxes. How to communicate this ? We have a large number of teams, all have to be aware of it.  TC should be alerted that changes are happening. We have a couple of dedicated channels for that. Most dev ops have subscribed to these channels. How can dev ops folk raise issues to the next level of community awareness ? There hasn't been a specific piece of TC to move that along.

    Craig: There is a fourth group, "Capacity Planning" or "Release Planning". Slack is the de facto communication channel.  There are no objections to using Slack. An example is the Java 17 RFC. 

    Craig: The TC gets it on the agenda and we will discuss it. The TC gets the final say.

    Marc Johnson: We shouldn’t use the DevOps Channel. The dev ops folks have made it clear that it should only be used for support requests made to them.

    Jakub: Our responsibility is to avoid piling up technical debt.

    Marc: Some set of people have to actually make the call. Who lowers the chequered flag ?

    Craig: It needs to ultimately come to the TC at least for awareness. There is a missing piece. Capacity Planning needs to provide input here. 

    Marc: Stakeholders / Capacity Planning could make that decision. Who makes the decision ? Is it the government or is it some parts of the body ?

    Marc: the developers community, the dev ops community and sys ops are involved. For example the Spring Framework discussion or the Java 17 discussion. But it was completely separate to the TC decision. It is a coordination and communication effort.

    Marc: Maybe the TC needs to let go that they are the decision makers so that they be a moderating group.

    Jakub: I agree with Marc. But we are not a system operating group. Dependency management should be in the responsibility of Release management. There are structures in the project for that.

    Jason Root: I agree with Jakub and with Marc also. Policies should drive operational/release/support aspects of Folio.

    Jason Root: If the idea of “support” is that frameworks are supported, then of course the project should meet that.

    Marc Johnson
    Some group needs to inform OleksAii when a relevant policy event occurs.
    These documents effectively ARE the manifestation of the policy.

    Craig: This is a topic for the next Monday session.

    Craig to see if Oleksii Petrenko could join us to discuss the process for updating the officially supported technologies lists.


    Dev Documentation VisibilityAll

    Possible topic/activity for a Wednesday session:

    Discuss/brainstorm:

    • Ideas for the type of developer-facing documentation we think would be most helpful for new developers
    • How we might bring existing documentation up to date and ensure it's consistent 
    • etc.
    API linting within our backend modulesAll

    https://folio-project.slack.com/archives/CAQ7L02PP/p1713343461518409


    Hello team, I would like to discuss API linting within our backend modules. Some time ago, we transitioned our linting process from Jenkins to GitHub Actions as outlined in https://folio-org.atlassian.net/browse/FOLIO-3678. I am assuming that this move was done via some technical council decision. Please correct me if I'm wrong.
    In my observations, I've found two problems:
    1. Schema linting does not occur if the schemas are in YAML format.
    2. There are issues with resolving some deeper references during API linting.
    Although I'm unsure about how to improve the existing linting implementations within Folio, I propose to consider an open-source solution that handles OpenAPI linting effectively and allows us to define custom rules. For your reference: https://stoplight.io/open-source/spectral A test of this solution can be found in this PR: https://github.com/folio-org/mod-search/pull/567. The same PR also provides an example of custom rule definition: https://github.com/folio-org/mod-search/pull/567/files#diff-d5da7cb43c444434994b76f3b04aa6e702c09e938de09dbc09d72569d611d9ab.Also, by employing 'Spectral', I discovered AsyncAPI (https://www.asyncapi.com/en), an API design tool similar to OpenAPI but for asynchronous interactions. I suggest that we consider using AsyncAPI in FOLIO to generate documentation for Kafka interactions.


    PR TemplatesAll

    https://folio-project.slack.com/archives/CAQ7L02PP/p1713445649504769

    Hello team, Small request to consider.
    Regarding pr templates.
    1. From my perspective, pr template is not good idea. Even the biggest open source projects that are contributed by many people don't have any pr template. Currently what we have for acq modules https://github.com/folio-org/mod-orders-storage/blob/master/PULL_REQUEST_TEMPLATE.md
    2. These pr template is inconsistent in different teams.
    What I suggest is that, pr template shouldn't be any instructions, because most developer who are creating pr have already understand the rules. If we put just two section into template, it will encourage developers to write more about their work and that lead to knowledge  sharing among developers.
    Java 21All

    https://folio-project.slack.com/archives/CAQ7L02PP/p1713445764285349


    Is Tech Council considering to update to java 21, I head good things from Netflix engineering teams about Garbage collector
    https://www.infoq.com/presentations/netflix-java/ (edited)

    Proposed Mod KafkaAll

    https://folio-project.slack.com/archives/CAQ7L02PP/p1714471592534689

    Mike Taylor

    Proposal. If and only if a FOLIO instance is running Kafka, it should insert and enable a module called mod-kafka, which consists entirely of a module descriptor that says it provides the interface kafka. The purpose is so that other modules can use the standard <IfInterface> and similar tools to determine whether they should attempt Kafka operations. Rationale: the FOLIO ILS depends absolutely on Kafka, but other uses of the platform will not. One such example: a dev platform that includes only mod-users, used as a source of change events for Metadb.

    ...