In production when mapping rules are changed - there’s a need to run migration process using mod-marc-migration module API. The whole process consist of two stages: data mapping and data saving.

The goal of this round of testing is to observe and measure the marc-migration process duration in ECS environment from Central tenant and member tenants, and find any possible issues and or performance problems.

Tests were done with two parameter sets (CHUNK_FETCH_IDS_COUNT, RECORDS_CHUNK_SIZE). The first tests were with optimal parameters defined in report for authority migration (12000 and 4000 accordingly). The second tests were with default parameters (500, 500)

Original ticket: https://folio-org.atlassian.net/browse/PERF-1045

• Tests performed on two tenants (central and member).

• All data mapping processes finished successfully.

  • No issues or errors found during data mapping process.

  • It consumes up to 25% CPU on mod-marc-migration service and up to 20-22% of DB CPU.

• All tests got different success rate on data saving, however tests for central and member tenant with parameters: “CHUNK_FETCH_IDS_COUNT = 500, RECORDS_CHUNK_SIZE = 500” got highest number of saved instances (89% for central tenant, 99% for member tenant).

  • Records failed to save because of issue found in DB logs - optimistic locking.

  • During data saving process runtime reindexing happening on the background (mod-search) causing high DB load for a long time (more than 1,5 hours) after data saving is completed.

  • Deadlocks is observed in DB, however they not causing any issues with marc-migration process as this deadlocks happens on mod-search schemas.

  • During data saving CPU usage on mod-marc-migration was 5% and 3% on mod-inventory-storage.

  • DB CPU reached 100% and stayed high even after process finished doe to mod-search runtime reindexing.

  • No signs of memory leaks found during all tests

• It’s clearly visible in test results that there’s small success rate of data saving with high values for CHUNK_FETCH_IDS_COUNT and RECORDS_CHUNK_SIZE parameters. mod-marc-migration is probably failing to save data because of high load caused by mod-search on CPU during data saving, probably batch update transaction is taking to long to complete and issue Apparent connection leak detected happening. mod-marc-migration doing retry of the same batch and getting back optimistic lock for records, as some records already completed with current batch in previous try.

• In order to get high performance it’s recommended to have 8 mod-inventory-storage tasks

• mod-marc-migration data saving fail https://folio-org.atlassian.net/browse/MODMARCMIG-43

• mod-search deadlocks tickethttps://folio-org.atlassian.net/browse/MSEARCH-932

• mod-search slow query affecting DI performance https://folio-org.atlassian.net/browse/MSEARCH-972

• Possibly related ticket for mod-search https://folio-org.atlassian.net/browse/MSEARCH-924

Note: in test 3 and 4 different umber of instances mapped as in between tests few data import were performed.

Memory utilization across all modules was stable, with no memory leaks observed. The most heavily used module, mod-marc-migrations, experienced an increase in memory usage from 25% to approximately 80% during testing with 'Optimal configurations.' However, after the process was completed, memory usage stabilized and returned to normal levels

Open

CPU utilization during the data-saving process was stable and returned to normal levels after the test was completed. The most utilized service was mod-consortia, with a peak usage of approximately 4.5%. The entire process, including reindexing by mod-search, took approximately 4 hours. With 'Optimal configurations,' the data-saving process was faster by about 1 hour and 30 minutes compared to the default configuration, where it took 2 hours and 40 minutes.

Open

Here is the CPU usage of the mod-marc-migrations module during the data-mapping process. It is evident that mod-marc-migrations can handle high loads and return to normal resource usage after the process is completed.

Note: Once the data-mapping process is finished, the CPU usage of mod-marc-migrations becomes less noticeable, as most of the logic for the next step (data saving) occurs on the database side.

DB CPU usage reached 100% during data saving

NOTE: ater data saving finished (see the marks on screenshot) runtime reindexing did proceed and occupied more that 90% of DB CPU for additional time.

Open

During data saving process a lot of deadlock found. This deadlocks happening on mod-search side and does not affect data saving process.

Open

Open

Open

Open

Open

Open

mod-marc-migration error:

DB Errors:

PTF -environment rcon

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• db.r6.xlarge database instances, writer

• MSK fse-test

  • 4 kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x (KRaft mode)

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

• OpenSearch 2.13 ptf-test cluster

  • r6g.2xlarge.search 4 data nodes

  • r6g.large.search 3 dedicated master nodes

R - Ramsons release

1. Initiate the migration operation for instances by sending a POST request to the /marc-migrations endpoint with the following request body:

   { "entityType": "instance", "operationType": "remapping" }

2. Track the status of the data mapping phase by sending GET requests to the /marc-migrations/{operationId} endpoint, where {operationId} is the id of the migration job received in response from the 1-st step. During the ongoing data mapping, the "status" field in the response will be data_mapping. Upon completing the data mapping step successfully the job status changes to data_saving_completed. If errors occur, the status will be data_saving_failed. Also, when data mapping is completed successfully the mappedNumOfRecords field should be equal to the totalNumOfRecords field value. The duration of the data mapping process can be measured using startTimeMapping and endTimeMapping fields from the endpoint response.

3. Once the data mapping process has been successfully completed, initiate the data saving step by sending a PUT request to the /marc-migrations/{operationId} endpoint with the following request body:

   { "status": "data_saving" }

   Receiving 204 status code in a response indicates that the data saving step has initiated. Track the status of this step using the GET endpoint described in 2-nd step. The status in the response will display data_saving when the job is in progress and will change to data_saving_completed or data_saving_failed upon completion. Similar to data mapping, savedNumOfRecords field should be equal to totalNumOfRecords field when the data saving step is completed successfully. The duration of the data saving step can be measured using startTimeSaving and endTimeSaving fields from response of the endpoint described in the 2-nd step

4. Adjust needed parameters in task definition of mod-marc-migration CHUNK_FETCH_IDS_COUNT = X, RECORDS_CHUNK_SIZE = Y and repeat tests.

Client-side logs:

Test 1

{"id":"3cece4e4-2c6a-48f9-aa0a-4d2e92b4b25c","userId":"9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42","entityType":"instance","operationType":"remapping","status":"data_saving_failed","totalNumOfRecords":1160063,"mappedNumOfRecords":1160063,"savedNumOfRecords":4063,"startTimeMapping":"2025-02-10T13:11:55.919082Z","endTimeMapping":"2025-02-10T13:22:11.999588Z","startTimeSaving":"2025-02-10T13:28:32.416862Z","endTimeSaving":"2025-02-10T14:41:11.265436Z"}

Test 2

{"id":"e6480683-9130-4a6c-bc11-983aba886375","userId":"9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42","entityType":"instance","operationType":"remapping","status":"data_saving_failed","totalNumOfRecords":1160063,"mappedNumOfRecords":1160063,"savedNumOfRecords":1043563,"startTimeMapping":"2025-02-10T19:08:11.806591Z","endTimeMapping":"2025-02-10T19:25:02.508308Z","startTimeSaving":"2025-02-10T19:38:07.026149Z","endTimeSaving":"2025-02-10T23:24:19.756021Z"}

As defined in report above each test failed with a different % of completion on data saving part of Marc-migration process. It’s reported from different teams that they faced with same issues in production cluster as well. The primary issue is DB that leaks connections during data saving process - leading to fail of whole batch. The purpose of additional testing is to define recommendations for Marc-migrations process when it has a higher chances to complete successfully data saving.

As defined in report above and in https://folio-org.atlassian.net/browse/MODINVSTOR-1369 the main reason of data saving failure is DB often leaking connections due to high load. So in order to achieve better percentage of success recommendations is following

• Scale DB to at least 2xLarge shape (tested with 1.2 M records);

• Disable mod-search. (set task count equal to '0'). After migration it may be enabled again to complete reindexing of edited records otherwise Kafka topic inventory.instances should be cleaned up and additional reindex done separately.

• Set task count for mod-inventory storage to 8. In order to use all 8 tasks properly set property in mod-marc-migrations task definition.

  {"name": "CHUNK_PROCESSING_MAX_PARALLELISM","value": "8"},

• Use as low as possible parameters "RECORDS_CHUNK_SIZE" and "CHUNK_FETCH_IDS_COUNT" or leave them as default (by not specifying them at all). Default values if 500 for both parameters. This parameters regulate chunk sizes. Taking into account that DB is a bottleneck here smaller chunk size means more stability.