Overview

This document contains the results of testing marc-migration for over 16 mln records in the Ramsons release. https://folio-org.atlassian.net/browse/PERF-929

Summary

• The duration for marc-migration is up to 9,5 hours (7 hours mapping, 2.5 hours saving ) for default configurations with background jobs (e.g. Deleting old records from marc_indexers).
• The best results are for test #9 with CHUNK_FETCH_IDS_COUNT=12000 and RECORDS_CHUNK_SIZE=4000 - total duration 4 hours 2 min (3 hours 35 minutes for data mapping and 27 minutes for data saving).
• Memory utilization increased for mod-entities-links up to 75%, mod-search -62%, and mod-marc-migrations - 60% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all modules.
• CPU utilization was for mod-entities-links up to 453%, mod-search -20%, mod-marc-migrations - 15%. Average CPU usage did not exceed 10 % for all other modules. But the CPU for mod-marc-migrations is set to 0 and actual CPU usage was about 30% for the AWS instance where mod-marc-migrations was located.
• For all tests, approximate DB CPU usage is up to 70%.

Additional tests

Test with 2 containers (tasks) for mod-marc-migrations and 2 tenants in parallel:

• It does not work for 2 tenants in parallel, even with 2 containers for mod-marc-migrations. The 2nd job will queued with the status - "new" while the 1st is running. The 2nd job will be started and finished successfully after the 1st one.

Test with 1 container (task) for mod-marc-migrations:

• There is no reason to use 2 containers for mod-marc-migrations because the duration for 1 container is the same as for 2 (the process uses only one mod-marc-migrations container of 2 allocated).

Recommendations and Jiras

• Increase default CPU allocation for mod-entities-links service or set it to 0.
• Use CHUNK_FETCH_IDS_COUNT=12000 and RECORDS_CHUNK_SIZE=4000 to decrease migration time, but mod-entities-links will use 25% more CPU.
• Use only 1 container (task) for mod-marc-migrations.
• While the data mapping is running, files with data will be stored directly in the working mod-marc-migrations container. Further, all files will be deleted from the container and relocated to the S3 bucket (if the S3 bucket is not provided - data mapping fails).
  If the container falls during the data mapping process - all files will be lost and data mapping will hang forever.

Results

Migrated and saved 16 462 866 records.

 * - Tests were performed with enabled "Deleting old records from marc_indexers" jobs
4a - repeat of test 4

Additional testing 

Tested with 1 container for mod-marc-migrations.

Test with 2 containers for mod-marc-migrations and 2 tenants in parallel.

Resource utilization for Test #9

Memory Utilization

• Memory utilization increased for mod-entities-links up to 75%, mod-search -62%, and mod-marc-migrations - 60% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all modules.

mod-entities-links, mod-search, mod-marc-migrations

Instance CPU Utilization

Instance CPU Utilization was up to 32%. The red line on the graph is the instance i-063e3abc2c04eb056, where one of the containers of mod-marc-migration is located.

Service CPU Utilization 

• CPU utilization was for mod-entities-links up to453%, mod-search -20%, mod-marc-migrations - 15% (It is set to 0 in the configurations). Average CPU usage did not exceed 10 % for all other modules.

mod-entities-links, mod-search, mod-marc-migrations

RDS CPU Utilization 

• Approximate DB CPU usage is up to 70%.

RDS Database Connections

Cluster metrics

Appendix

Infrastructure

Records count :

• mod_entities_links.authority = 16462866

PTF -environment rcp1 

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database instance - writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK fse-tenant
  
  • 4 kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

Modules memory and CPU parameters

Methodology/Approach

To test Baseline for marc-migration JMeter scripts were used.

Test preparation: 

• Background 'Deleting old records from marc_indexers' jobs were disabled:
  added configuration to mod-source-record-storage

                  {
                      "name": "srs.marcIndexers.delete.interval.seconds",
                      "value": "86400"
                  },

• 16 mln Marc Authority records added to the database

Check Migration Operation from the database:
- select end_time_mapping - start_time_mapping as mapping_duration,end_time_saving - start_time_saving as saving_duration,* from {tenant}_mod_marc_migrations.operation order by start_time_mapping desc

More info workflows-scripts/marc-migration/readme.md