...

• The duration for marc-migration is up to 9,5 hours (7 hours mapping, 2.5 hours saving ) for default configurations with background jobs (e.g. Deleting old records from marc_indexers).
• The best results are for test #9 with CHUNK_FETCH_IDS_COUNT=12000 and RECORDS_CHUNK_SIZE=4000 - total duration 4 hours 2 min (3 hours 35 minutes for data mapping and 27 minutes for data saving).
• Memory utilization increased for mod-entities-links up to 75%, mod-search -62%, and mod-marc-migrations - 60% due to previous module restarting and everyday cluster shutdown processes - no memory leak is suspected for all modules.
• CPU utilization was for mod-entities-links up to 453%, mod-search -20%, mod-marc-migrations - 15%. Average CPU usage did not exceed 10 % for all other modules. But the CPU for mod-marc-migrations is set to 0 and actual CPU usage was about 30% for the AWS instance where mod-marc-migrations was located.
• For all tests, approximate DB CPU usage is up to 70%.

...

Instance CPU Utilization was up to 32%. The red line on the graph is the instance i-063e3abc2c04eb056, where one of the containers of mod-marc-migration is located.

Service CPU Utilization 

• CPU utilization was for mod-entities-links up to453%, mod-search -20%, mod-marc-migrations - 15% (It is set to 0 in the configurations). Average CPU usage did not exceed 10 % for all other modules.

...

• Approximate DB CPU usage is up to 70%.

RDS Database Connections

Cluster metrics

Appendix

Infrastructure

Records count :

• mod_entities_links.authority = 16462866

PTF -environment rcp1 

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database instance - writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK fse-tenant
  
  • 4 kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...