2023-04-21 - Sys Ops & Management SIG Agenda and Meeting notes

https://github.com/folio-org/rfcs/pull/4

This is an RFC (request for comment) submitted to the Technical Council (TC).

Background: If you host Kafka as a Cloud Service, there can be quite high charges which increase per so-called Kafka Topic. A Kafka Topic can be thought of as a queue of messages which a module subscribes to. If each tenant hosts ca. 70 modules, which have one Topic per module, an you host many tenants – which is typically a scenario for hosting providers or self-hosting library unions (German Verbundzentralen) – the number of Topics and thus the hosting costs for Kafka do quickly increase.

To mediate this, this RFC proposes Topics per so-called tenant collection. Summary of the RFC "With this RFC, either Kafka topics are created for a collection of tenants or Kafka topics for each tenant can be configured for each deployed module instance. Using a collection of tenants will reduce the number of Kafka topics and subsequently the number of partitions."

Ingolf brings this to the attention of SysOps SIG. For SysOps, some parts of the proposal are of interest:

"For a module instance the KAFKA_PRODUCER_TENANT_COLLECTION environment variable

• is set to configure Kafka topics for a tenant collection, or
• is unset to configure Kafka topics for each tenant."

See also the sections "Migration To Tenant Collection Topics" and "Migration to New Module Version" in that document:

Feedback on the Pull Request by the TC: rfcs/0000-kafka-topics-per-tenant.md at kafka_topics_per_tenant · folio-org/rfcs (github.com)

The RFC is near the end of the review stage.

• Orchid R1-2023 is out
• Nolana HF#1 has been released
• ----------------------------------------------------------------------------
• Nolana GA: Jason reports calendar bugs. Some patches fixed most of the problems. The Nolana #1 hotfix fixed the fina bits (hopefully!).
• A&M: Uprade to Lotus in August before WolfCon. After the Upgrade, some of the stuff just broke. We spent some time to catch up with the developers. A&M plans two upgrades a year.

Jeremy is authoring a repository folio-notebooks, jermnelson/folio-notebooks: Examples of Using Jupyter Notebooks with the FOLIO LSP (github.com)

"A Repository that contains Jupyter notebooks for interacting with the FOLIO Library Systems Platform along with a corresponding Library Data Platform."

It is run in a Python virtual environment.

----------

Jeremy is using Apache Airflow for data migration. System librarians can actively participate in the migration process: What is Airflow? — Airflow Documentation (apache.org)

------------

Ingolf is using Catmandu for data manipulation (pre-migration): Catmandu (librecat.org) . MARC data can be "fixed" (converted, transformed, added, deleted...) on a field-by-field basis.

Jason is using Camunda instead.

Ingolf mentions Metafacture – Welcome , a tools which is similar to Catmandu and has been developed by his colleagues at hbz.

---------

FSE FOLIO Migration Tools: About the tools — FSE FOLIO Migration Tools documentation (folio-migration-tools.readthedocs.io)

A comprehensive toolset for migration tasks to FOLIO: 

"

FOLIO Migration Tools currently covers the following data sets:

• Catalog (Inventory and SRS in FOLIO terminology)

• Circulation transactions (Open loans and requests)

• Users/Patrons (In FOLIO, these share the same app/database)

• Courses and Reserves (Course reserves)

• Organizations (Used in ERM and Aquisitions)"

There is a group which has a FOLIO Slack channel and holds regular meetings: #fse_folio_migration_tools .

This group has been led by Theodor Tolstoy until last week. Group is now being maintained by Brooks Travis.

Of this group, Jeremy and Ingolf use FSE migration tools.

• Overview: List of Integrations - System Operations and Management SIG - FOLIO Wiki
• Discovery: Discovery Integration Subgroup - System Operations and Management SIG - FOLIO Wiki