Ramsons (R2 2024) - OKAPI Bugfest env preparation - Issues/preparation we needed to resolve on environment

1. General recomendations and platform-complete Issues

Recomendation 1: edge-oai-pmh

Need to create clean environment with new unik edge keys for all tenants or align it. Faced a lot of essues related to edge keys.
BF-903: Full harvesting via Central tenant fails on ECS bugfest environmentClosed

BF-831: 403 Forbidden error for cs00000int_0002 tenant and fail to do central tenant harvesting on ECS bugfest environmentClosed

Recomendation 2: Cloudfront and ALB configuration
While creating new ALB for Quesnelia environment faced with AWS limit of ALB creation on FSE AWS account.
Had to extend limit by contacting AWS support

Recomendation 3: Tenant creation parameters:
For each tenat creation use:

tenantParameters=loadReference%3dtrue%2cloadSample%3dfalse

2. Review of Release Notes

We reviewed the release notes for the Ramsons release and adjusted our configs. Here are the changes made to specific modules:

Module	Instructions	Comments

Module	Instructions	Comments
mod-fqm-manager	A `centralTenantId` parameter should be included in the API request to enable mod-fqm-manager when initially configuring new tenants in ECS environments. Note: This is not necessary at all in non-ECS environments	If the tenant is already part of a consortium (e.g., when upgrading an existing environment), this isn’t necessary, as FQM will retrieve the central tenant ID automatically. If this is missed when setting up the tenant, it will cause FQM to produce incorrect data in same cases. This can be fixed by simply doing the API request again (with`centralTenantId` set, if the tenant has not been added to the consortium)
mod-lists	The Lists module now uses a system user, so the `OKAPI_URL` and `SYSTEM_USER_PASSWORD` environment variables will need to be configured.
mod-login-saml	Configure `DB_` environment variables for PostgreSQL database, see DB_ variables documentation.	Typically, you need to set these: `DB_HOST` `DB_PORT` `DB_USERNAME` `DB_PASSWORD` `DB_DATABASE`
mod-data-export	Configure `PLATFORM` environment variable. Configure `EXPORT_TMP_STORAGE` to store files in volumes if needed	`PLATFORM` specifies the way how permissions for user are checked. The values are `okapi` or `eureka`. Default value is `okapi`.
mod-data-export-worker	Configure `PLATFORM` environment variable. For s3 communication it necessary to configure `S3_URL` , `S3_REGION`, `S3_BUCKET`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_SUB_PATH`, `S3_LOCAL_SUB_PATH` , `S3_IS_AWS`	`PLATFORM` specifies the way how permissions for user are checked. The values are `okapi` or `eureka`. Default value is `okapi` . LOCAL_* variables were removed. Instead of separate bucket folder local was introduced(`S3_LOCAL_SUB_PATH`). `S3_SUB_PATH` - is a S3 sub path for files storage. Environment variables with AWS-_* were renamed to S3_*
mod-bulk-operations	Configure `PLATFORM` environment variable.	`PLATFORM`specifies the way how permissions for user are checked. The values are `okapi` or `eureka`. Default value is `okapi` .
mod-orders-storage	New kafka consumers to fetch events from mod-inventory-storage are added. Make sure that all kafka related properties/env variables such as in other modules are set	New worker verticles that process events from kafka are added, by default each of them uses 5 threads in the pool. It can be changed if needed by setting system properties: `item.create.consumer.pool.size` `item.update.consumer.pool.size` `holding.create.consumer.pool.size` `holding.update.consumer.pool.size`
mod-oai-pmh	The module should be started using `-Dfile.encoding=UTF-8` java option

mod-search:
1. Locations reindex needed after the upgrade.
Use re-index API: https://s3.amazonaws.com/foliodocs/api/mod-search/u/mod-search.html#api-IndexManagement-reindexInventoryRecords with resourceName = location

2. After the upgrade use new API to index instance records.
The env variables that are obsolete now:
- INSTANCE_SUBJECTS_INDEXING_RETRY_ATTEMPTS
- INSTANCE_CONTRIBUTORS_INDEXING_RETRY_ATTEMPTS
- KAFKA_CONTRIBUTORS_TOPIC_PARTITIONS
- KAFKA_CONSORTIUM_INSTANCE_TOPIC_PARTITIONS
- KAFKA_SUBJECTS_TOPIC_PARTITIONS
mod-inventory-storage:
Async migration execution is required after the upgrade. Could work in the background.
Information about API that manages async migration: https://s3.amazonaws.com/foliodocs/api/mod-inventory-storage/r/async-migration.html
To start migration execute the endpoint:
POST /inventory-storage/migrations/jobs Body: { "migrations": [ "publicationPeriodMigration" ] }
To track status of migration execute the endpoint:
GET /inventory-storage/migrations/jobs/<jobId>
To validate that migration is finished SQL script could be executed:
SELECT 1 FROM <tenant>_mod_inventory_storage.instance WHERE jsonb -> 'publicationPeriod' IS NOT NULL LIMIT 1;
If nothing returned then migration was successful.
mod-source-record-storage
BF-956: Run script on Okapi Ramsons BF env to fix versions data for records containing invalid 005 valueClosed
BF-826: Run script to update records with invalid 005 field on bugfestClosed
Change configuration for the BugFest environment
BF-906: Change configuration for the BugFest environmentClosed

3. Deployment of Modules

We encountered various issues during module deployment. Here are the details and their resolutions:

mod-data-import:
Tables overfulled with data
BF-735: Upload file is not proceeding in Data import appClosed
Resolution
SELECT id, jsonb, creation_date, created_by
    FROM fs09000000_mod_data_import.upload_definitions
    WHERE jsonb->> 'status' :: text in ('NEW', 'IN_PROGRESS', 'LOADED')
    ORDER BY creation_date DESC;
mod-service-interaction, mod-serials-management, mod-agreements:
This resolution aplicable for mod-service-interaction, mod-serials-management, mod-agreements issues
Delete next data from federation_lock table

95d8d734-b027-421f-96cc-baa4b21ea8d1	tenant:cs00000int_0009	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
b73805df-aef6-49ce-ac99-4ec4c3a36f18	tenant:cs00000int_0010	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
6446e4be-020f-43b7-a5ba-c6cea25d536c	tenant:cs00000int_0011	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
b2d01573-cad8-4245-ba76-bf3d567bacfa	tenant:cs00000int	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
e7c2f601-ffe9-4653-92aa-02f71cc8f9bb	tenant:fs09000000	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
c6cde6dc-3d6b-49aa-9974-2c351395158b	tenant:seeding	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
2f5cb100-c7c3-434d-9516-8cc89d5e7b0d	tenant:cs00000int_0001	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
e298b1f8-076f-4b41-8ec6-4cf79226ece2	tenant:cs00000int_0002	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
e49f09b8-f19f-44a8-ab71-0ac3a71fced6	tenant:cs00000int_0003	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
967d56cc-3dde-485c-8309-7841798ae079	tenant:cs00000int_0004	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
0c6255db-6f25-4b82-bef4-1dcc0ce37f4b	tenant:cs00000int_0005	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
0837cbcf-49d2-405c-a0b1-70243580abd0	tenant:cs00000int_0006	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
4cd78b89-0bdb-4c7e-9829-a006cddb53c1	tenant:cs00000int_0007	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7
28e35002-6279-4a2d-9a2c-49219437d9c8	tenant:cs00000int_0008	7.0.4	f59c6d1a-5133-4572-8f87-c96c2e3b00a7

Wait 5-10 minutes for mod-agreements to setup after deployment

mod-consortia: Excluded deployment for non-consortia tenants.

4. Other Issues

"Select MARC authority" plugin is disabled
BF-946: "Select MARC authority" plugin is disabled in okapi bugfest environmentClosed
Resolution:
Fix in comments
MARC bibliographic/authority specifications are empty
Resolution:
BF-818: MARC bibliographic/authority specifications are empty in MARC validation rulesClosed
Kafka consolidation
Was done to reduce the number of topics load on the Kafka cluster.
1) List of kafka modules

[ { "id": "mod-audit", "action": "enable" }, { "id": "mod-entities-link", "action": "enable" }, { "id": "mod-orders-storage", "action": "enable" }, { "id": "mod-invoice", "action": "enable" }, { "id": "mod-data-export-spring", "action": "enable" }, { "id": "mod-quick-marc", "action": "enable" }, { "id": "mod-inventory", "action": "enable" }, { "id": "mod-search", "action": "enable" }, { "id": "mod-data-import", "action": "enable" }, { "id": "mod-remote-storage", "action": "enable" }, { "id": "mod-pubsub", "action": "enable" }, { "id": "mod-data-export-worker", "action": "enable" }, { "id": "mod-source-record-manager", "action": "enable" }, { "id": "mod-inventory-storage", "action": "enable" }, { "id": "mod-source-record-storage", "action": "enable" }, { "id": "mod-circulation", "action": "enable" }, { "id": "mod-invoice", "action": "enable" }, { "id": "mod-dcb", "action": "enable" }, { "id": "mod-consortia", "action": "enable" }, { "id": "mod-users", "action": "enable" } { "id": "mod-record-specifications", "action": "enable" } ]

2) Added variables to each kafaka related module.

{ "name": "KAFKA_PRODUCER_TENANT_COLLECTION", "value": "ALL" }, { "name": "ENV", "value": "$CLUSTER" },

Quesnelia environment general info

Tenant	Name	URL

Tenant	Name	URL
fs09000000	Integration	https://bugfest-ramsons.int.aws.folio.org/logout
fs09000002	Plus	https://bugfest-ramsons-plus.int.aws.folio.org/
fs09000003	AQA	https://bugfest-ramsons-aqa.int.aws.folio.org/
cs00000int	Consortium	https://bugfest-ramsons-consortium.int.aws.folio.org/

Modules list for NON ECS tenants

[
    {
        "id": "edge-caiasoft-2.3.2"
    },
    {
        "id": "edge-connexion-1.3.1"
    },
    {
        "id": "edge-courses-1.5.0"
    },
    {
        "id": "edge-dcb-1.2.1"
    },
    {
        "id": "edge-dematic-2.3.1"
    },
    {
        "id": "edge-erm-1.3.0"
    },
    {
        "id": "edge-fqm-3.0.1"
    },
    {
        "id": "edge-ncip-1.10.1"
    },
    {
        "id": "edge-oai-pmh-2.10.0"
    },
    {
        "id": "edge-orders-3.1.0"
    },
    {
        "id": "edge-patron-5.2.0"
    },
    {
        "id": "edge-rtac-2.8.0"
    },
    {
        "id": "edge-sip2-3.3.1"
    },
    {
        "id": "folio_acquisition-units-5.2.0"
    },
    {
        "id": "folio_agreements-11.1.1"
    },
    {
        "id": "folio_bulk-edit-4.2.3"
    },
    {
        "id": "folio_calendar-11.0.3"
    },
    {
        "id": "folio_checkin-9.2.1"
    },
    {
        "id": "folio_checkout-11.0.2"
    },
    {
        "id": "folio_circulation-10.0.1"
    },
    {
        "id": "folio_circulation-log-5.0.1"
    },
    {
        "id": "folio_courses-6.1.1"
    },
    {
        "id": "folio_dashboard-6.1.1"
    },
    {
        "id": "folio_data-export-6.2.0"
    },
    {
        "id": "folio_data-import-8.0.4"
    },
    {
        "id": "folio_developer-9.0.0"
    },
    {
        "id": "folio_eholdings-10.0.1"
    },
    {
        "id": "folio_erm-comparisons-7.1.0"
    },
    {
        "id": "folio_erm-usage-10.0.1"
    },
    {
        "id": "folio_export-manager-3.2.0"
    },
    {
        "id": "folio_finance-7.0.1"
    },
    {
        "id": "folio_gobi-settings-3.2.0"
    },
    {
        "id": "folio_handler-stripes-registry-2.3.0"
    },
    {
        "id": "folio_inventory-12.0.6"
    },
    {
        "id": "folio_invoice-6.1.1"
    },
    {
        "id": "folio_licenses-11.1.1"
    },
    {
        "id": "folio_lists-3.1.3"
    },
    {
        "id": "folio_local-kb-admin-8.1.0"
    },
    {
        "id": "folio_marc-authorities-6.0.0"
    },
    {
        "id": "folio_myprofile-9.2.0"
    },
    {
        "id": "folio_notes-10.0.0"
    },
    {
        "id": "folio_oai-pmh-5.2.0"
    },
    {
        "id": "folio_orders-7.0.3"
    },
    {
        "id": "folio_organizations-5.2.0"
    },
    {
        "id": "folio_plugin-bursar-export-4.0.3"
    },
    {
        "id": "folio_plugin-create-inventory-records-5.0.1"
    },
    {
        "id": "folio_plugin-eusage-reports-3.2.0"
    },
    {
        "id": "folio_plugin-find-agreement-11.1.0"
    },
    {
        "id": "folio_plugin-find-authority-4.1.0"
    },
    {
        "id": "folio_plugin-find-contact-5.2.0"
    },
    {
        "id": "folio_plugin-find-eresource-7.1.0"
    },
    {
        "id": "folio_plugin-find-erm-usage-data-provider-6.2.0"
    },
    {
        "id": "folio_plugin-find-fund-3.2.0"
    },
    {
        "id": "folio_plugin-find-import-profile-8.0.0"
    },
    {
        "id": "folio_plugin-find-instance-8.0.0"
    },
    {
        "id": "folio_plugin-find-interface-5.2.0"
    },
    {
        "id": "folio_plugin-find-license-11.1.0"
    },
    {
        "id": "folio_plugin-find-organization-5.2.0"
    },
    {
        "id": "folio_plugin-find-package-title-6.2.0"
    },
    {
        "id": "folio_plugin-find-po-line-5.2.0"
    },
    {
        "id": "folio_plugin-find-user-7.2.0"
    },
    {
        "id": "folio_plugin-query-builder-1.2.5"
    },
    {
        "id": "folio_quick-marc-9.0.1"
    },
    {
        "id": "folio_reading-room-1.0.0"
    },
    {
        "id": "folio_receiving-6.0.4"
    },
    {
        "id": "folio_remote-storage-6.0.0"
    },
    {
        "id": "folio_requests-11.0.1"
    },
    {
        "id": "folio_serials-management-1.1.1"
    },
    {
        "id": "folio_service-interaction-3.1.1"
    },
    {
        "id": "folio_servicepoints-7.2.0"
    },
    {
        "id": "folio_stripes-authority-components-5.0.1"
    },
    {
        "id": "folio_stripes-core-10.2.4"
    },
    {
        "id": "folio_stripes-marc-components-1.1.0"
    },
    {
        "id": "folio_stripes-smart-components-9.2.5"
    },
    {
        "id": "folio_tags-8.2.0"
    },
    {
        "id": "folio_tenant-settings-9.0.0"
    },
    {
        "id": "folio_users-11.0.7"
    },
    {
        "id": "mod-agreements-7.1.3"
    },
    {
        "id": "mod-audit-2.10.1"
    },
    {
        "id": "mod-authtoken-2.16.1"
    },
    {
        "id": "mod-batch-print-1.2.0"
    },
    {
        "id": "mod-bulk-operations-2.1.5"
    },
    {
        "id": "mod-calendar-3.2.0"
    },
    {
        "id": "mod-circulation-24.3.2"
    },
    {
        "id": "mod-circulation-item-1.1.0"
    },
    {
        "id": "mod-circulation-storage-17.3.1"
    },
    {
        "id": "mod-configuration-5.11.0"
    },
    {
        "id": "mod-copycat-1.7.0"
    },
    {
        "id": "mod-courses-1.4.11"
    },
    {
        "id": "mod-data-export-5.1.3"
    },
    {
        "id": "mod-data-export-spring-3.4.2"
    },
    {
        "id": "mod-data-export-worker-3.3.4"
    },
    {
        "id": "mod-data-import-3.2.1"
    },
    {
        "id": "mod-dcb-1.2.3"
    },
    {
        "id": "mod-di-converter-storage-2.3.1"
    },
    {
        "id": "mod-ebsconet-2.3.0"
    },
    {
        "id": "mod-email-1.18.0"
    },
    {
        "id": "mod-entities-links-3.1.1"
    },
    {
        "id": "mod-erm-usage-5.0.0"
    },
    {
        "id": "mod-erm-usage-harvester-5.0.1"
    },
    {
        "id": "mod-eusage-reports-3.0.0"
    },
    {
        "id": "mod-event-config-2.8.0"
    },
    {
        "id": "mod-feesfines-19.2.1"
    },
    {
        "id": "mod-finance-5.0.0"
    },
    {
        "id": "mod-finance-storage-8.7.3"
    },
    {
        "id": "mod-fqm-manager-3.0.4"
    },
    {
        "id": "mod-gobi-2.9.0"
    },
    {
        "id": "mod-graphql-1.13.1"
    },
    {
        "id": "mod-inventory-21.0.3"
    },
    {
        "id": "mod-inventory-storage-28.0.2"
    },
    {
        "id": "mod-inventory-update-4.0.0"
    },
    {
        "id": "mod-invoice-5.9.1"
    },
    {
        "id": "mod-invoice-storage-5.9.1"
    },
    {
        "id": "mod-kb-ebsco-java-5.0.0"
    },
    {
        "id": "mod-licenses-6.1.2"
    },
    {
        "id": "mod-lists-3.0.3"
    },
    {
        "id": "mod-login-7.12.1"
    },
    {
        "id": "mod-login-saml-2.9.3"
    },
    {
        "id": "mod-ncip-1.15.6"
    },
    {
        "id": "mod-notes-6.0.0"
    },
    {
        "id": "mod-notify-3.3.0"
    },
    {
        "id": "mod-oai-pmh-3.14.3"
    },
    {
        "id": "mod-orders-12.9.3"
    },
    {
        "id": "mod-orders-storage-13.8.3"
    },
    {
        "id": "mod-organizations-2.0.0"
    },
    {
        "id": "mod-organizations-storage-4.8.1"
    },
    {
        "id": "mod-password-validator-3.3.0"
    },
    {
        "id": "mod-patron-6.2.1"
    },
    {
        "id": "mod-patron-blocks-1.11.1"
    },
    {
        "id": "mod-permissions-6.6.0"
    },
    {
        "id": "mod-pubsub-2.15.2"
    },
    {
        "id": "mod-quick-marc-6.0.0"
    },
    {
        "id": "mod-reading-room-1.0.0"
    },
    {
        "id": "mod-record-specifications-1.0.2"
    },
    {
        "id": "mod-remote-storage-3.3.2"
    },
    {
        "id": "mod-rtac-3.7.0"
    },
    {
        "id": "mod-search-4.0.2"
    },
    {
        "id": "mod-sender-1.13.0"
    },
    {
        "id": "mod-serials-management-1.1.1"
    },
    {
        "id": "mod-service-interaction-4.1.1"
    },
    {
        "id": "mod-settings-1.1.0"
    },
    {
        "id": "mod-source-record-manager-3.9.3"
    },
    {
        "id": "mod-source-record-storage-5.9.3"
    },
    {
        "id": "mod-tags-2.3.0"
    },
    {
        "id": "mod-template-engine-1.21.0"
    },
    {
        "id": "mod-user-import-3.9.0"
    },
    {
        "id": "mod-users-19.4.4"
    },
    {
        "id": "mod-users-bl-7.9.3"
    },
    {
        "id": "okapi-6.1.0"
    }
]

Releases