Overview

The Data Import Task Force (DITF) implements a feature that splits large input MARC files into smaller ones, resulting in smaller jobs, so that the big files could be imported and be imported consistently. Previous tests for this feature were conducted (see report) but this particular set of tests were performed on the Cornell Perf environment which has proven to be challenging for Data Import jobs in the past.  We performed the baseline tests and tests with this feature enabled using the most optimal RECORDS_PER_SPLIT_FILE(RPSF) value found in previous experiments. 

Summary

• Differences in processing time Before the deployment of DITF's file-splitting feature and after DITF's file-splitting feature was deployed are:
  • File_7518   records 2 minutes 40 seconds (more than baseline);
  • File_67522 records 11 minutes (more);
  • File_7518  records during CICO test (25VU)  3 minutes 7 seconds (more)

These are not huge degradations from the baseline tests.

• Instance CPU utilization for all 3 tests, with two DITF's file-splitting feature configurations, were the same, Maximal CPU utilization was about 43% after deployment of DITF's file-splitting feature for Test 3.
  
• Service CPU utilization for all 3 tests, with two DITF's file-splitting feature configurations, were the same. Some of the services after deployment of DITF's file-splitting feature have less CPU utilization than before deployment  for about 5%.  nod-invoice-b service reached about 300 percent of CPU utilization (due to health checking requests).
• Memory utilization. No memory leaks are suspected for all instances and modules.
• Approximately DB CPU usage is up to 96%.
• No negative impact on CICO processes after  DITF's file-splitting feature was deployed.

Recommendations and Jiras

• In addition to the above information, it is advisable to follow the recommendations provided in the test report Data Import Splitting Feature test report (Orchid) ocp3 + retesting Poppy FSF and RTR;
• Investigate the problem related to the "mod-invoice-b" service. During the DI tests, it was observed that the CPU utilization of the service reached approximately 300 percent and did not return to normal even after the tests were completed. Upon analyzing the logs, it was discovered that the service was still responding to health-checking requests. This behavior indicates an issue with the service's performance or resource management. It is recommended to conduct a thorough investigation to identify the root cause of this abnormal CPU utilization and ensure that appropriate measures are taken to rectify the problem. The issue with the "mod-invoice-b" service's CPU utilization does not impact the results of the performance testing, since this service is not engaged to the import date process. 
  

Results

*During the test, 2 errors occurred

Instance CPU Utilization 

Testing before deployment of DITF's file-splitting feature

Test 1.1. Maximal CPU utilization on instance was about 28%

Test 1.2. Maximal CPU utilization on instance was about 26%

Test 1.3. Maximal CPU utilization on instance was about 40%

Testing after deployment of DITF's file-splitting feature 

Test 1.1. Maximal CPU utilization on instance was about 27%

Test 1.2. Maximal CPU utilization on instance was about 25%

Test 1.3. Maximal CPU utilization on instance was about 43%

Service CPU Utilization 

After DI tests started the mod-invoice-b service reached about 300 percent of CPU utilization and didn`t stop after the tests were finished. Logs analysis showed that the service was responding on health-checking requests. 

Testing before deployment of DITF's file-splitting feature 

Testing after deployment of DITF's file-splitting feature 

Service Memory Utilization

Memory usage was stable over three tests no memory leak is suspected for DI modules, except mod-inventory-b service.

Testing before deployment of DITF's file-splitting feature 

Testing after deployment of DITF's file-splitting feature 

RDS CPU Utilization

Approximately DB CPU usage is up to 95%

RDS Database Connections

Test 3 With CI/CO 25 users and DI 7.5k records on 1 tenant

Additional information "time of file splitting" 

 Test 1) During CICO a 7500K record file was uploaded at "2023-10-05 12:15:40.744+00" The first split file started processing at "2023-10-05 12:15:43.03+00"
Response time graph

Service CPU utilization

Test 2) During CICO a 67522 record file was uploaded at "2023-10-06 12:41:18.392+00" The first split file started processing at "2023-10-06 12:41:57.204+00"

Response time graph


Service CPU utilization

Appendix

Records count :

• fs00001034_mod_source_record_storage.marc_records_lb = 17901545
• fs00001034_mod_source_record_storage.raw_records_lb = 17954316
• fs00001034_mod_source_record_storage.records_lb = 17959758
• fs00001034_mod_source_record_storage.marc_indexers =  1001685192
• fs00001034_mod_inventory_storage.authority = 0
• fs00001034_mod_inventory_storage.holdings_record = 11401620
• fs00001034_mod_inventory_storage.instance = 10566424
• fs00001034_mod_inventory_storage.item = 10013571

PTF -environment CPTF2 

• 11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 2 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
    log.retention.minutes=480
    default.replication.factor=3

Methodology/Approach

During the performance testing of DITF's file-splitting feature modules, the tests with the following configurations were run: 

1 primary tenant, 3 baseline tests performed one after the other in sequence
    Test 1) CICO 25 VU, 250 sec of ramp-up period, 3600-sec test duration + DI file with 7518 records (20230307monosLoad.mrc) about 20 minutes after CICO started on one tenant only with profile "EBSCO ebooks new and updated".

    Test 2) Small DI file with 7518 records (20230307monosLoad.mrc) on one tenant only with profile "EBSCO ebooks new and updated".

    Test 3) Large DI file with 67522 records (202301monos_me_processed.mrc).

After DITF's file-splitting feature was deployed and enabled, the number for RECORDS_PER_SPLIT_FILE was set to 1000, and the baseline tests were repeated.