Overview

The Data Import Task Force (DITF) implements a feature that splits large input MARC files into smaller ones, resulting in smaller jobs, so that the big files could be imported and be imported consistently.  This document contaest contains 1. Test with 1, 2, and 3 tenants' concurrent jobs with configurins the configurations the results of performance tests on the feature and also an analysis the feature's performance with respect to the baseline tests.  The following Jiras were implemented. 

Summary

• The file-splitting feature is stable and offers more robustness to Data Import jobs even with the current infrastructure configuration. If there were failures, it's easier now to find the exact failed records to take actions on them. 
  • No stuck jobs in all tests performed.
  • There were errors (see below) in some partial jobs, but they still completed so the entire job status is "Completed with error".
  • Both of kinds of imports, create and update MARC BIBs worked well with this file-splitting feature enabled and also disabled. 
• There is no performance degradations, jobs not getting slower, on single-tenant imports. On multi-tenants imports, performance is be a little better
• Duration for DI correlates with number of the records imported (100k records- 38 min, 250k - 1 hour 32 min, 500k - 3 hours 29 min).
• Multitenant DI could be performed successfully for up to 9 jobs in parallel. If jobs are big they will start one by one in order for each tenant but processed in parallel on 3 tenants. Small DI (1 record) could be finished faster not in order. 
• No memory leak is suspected for all of the modules.
• Average CPU usage for mod-inventory -was 144%, mod-di-converter-storage was about 107%, and for all other modules did not exceed 100 %. We can observe spikes in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 260%.  Big improvement over the previous version (without file-splitting) for 500K imports where mod-di-converter-storage's CPU utilization was 462% and other modules were above 100% and up to 150%. 
• Approximately DB CPU usage is up to 95%.

...

1. One record on one tenant could be discarded with error: io.netty.channel.StacklessClosedChannelException.

   Jira Legacy
   server FOLIO Issue TrackerSystem Jira serverId 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc key MODDATAIMP-748

   Reproduces in both cases with and without splitting feature enabled in at least 30% of test runs with 500k record files and multitenant testing.
2. During the new Data Import splitting feature testing, items for update were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get. Less than 1% of records could be discarded due to missing permission for  'data-import-system-user'. Permission was not added automatically during the service deployment. I added permission manually to the database and the error does not occur anymore.

   Jira Legacy
   server FOLIO Issue TrackerSystem Jira serverId 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc key MODDATAIMP-930

3. UI issue, when canceled or completed with error Job progress bar cannot be deleted from the screen.

   Jira Legacy
   server FOLIO Issue TrackerSystem Jira serverId 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc key MODDATAIMP-929

4. Usage:
   • Should not use less than 1000 for RECORDS_PER_SPLIT_FILE. The system is stable enough to ingest 1000 records consistently and smaller amounts will incur more overheads, resulting in longer jobs' durations.  CPU utilization for mod-di-converter-storage for 500 RECORDS_PER_SPLIT_FILE(RPSF) = 160%, for 1000RPSF =180%, for 5K RPSF =380% and for 10K RPSF =433%, so in the case of selecting configurations 5K or 10K we recommend to add more CPU to mod-di-converter-storage service.
   • When toggling the file-splitting feature, mod-source-record-storage, mod-source-record-manager's tasks need to be restarted.
   • Keep in mind about the Kafka broker's disk size (as bigger jobs - up to 500K - can be run now), consecutive jobs may use up the disk quickly because the messages' retention time currently is set at 8 hours. For example with 300GB disk size, consecutive jobs of 250K, 500K, 500K sizes will exhaust the disk. 
5. More CPU could be allocated to mod-inventory and mod-di-converter-storage

...

 * - One record on one tenant could be discarded with error: io.netty.channel.StacklessClosedChannelException.

...

 ** -  up to 10 items were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get. Less than 1% of records could be discarded due to missing permission for  'data-import-system-user'. Permission was not added automatically during the service deployment. I added permission manually to the database and the error does not occur anymore.

Test 1,2. 100k, 250K, 500k and Multitenant MARC BIB Create

...

With CI/CO 20 users and DI 25k records on each of the 3 tenants Splitting Feature Disabled

ocp3-mod-data-import:12

Image Modified

Data Import Robustness Enhancement

...

Memory utilization rich maximal value for mod-source-record-storage-b 88%  and for mod-source-record-manager-b 85%.

Test 2. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 10K, 2 runs for each test.

...

Test 2. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 10K, 2 runs for each test.

CPU utilization of  mod-di-converter-storage-b

RDS CPU Utilization 

Test 1. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 500, 2 runs for each test. Maximal  CPU Utilization = 95%

...

Retest the DI feature to be sure that the new changes have not affected performance negatively.  Retest the DI file-splitting feature for the following scenarios:

Brief comparison summary

...

The goal of the tests was to investigate how the file-splitting feature caused Data-import on Poppy release and the impact of Refresh Token Rotation (RTR). The tests were performed on ocp3(Poppy), pcp1(Poppy) and ncp5(Orchid)  environments.

...

• tenant0_mod_source_record_storage.marc_records_lb = 9674629
• tenant2_mod_source_record_storage.marc_records_lb = 0
• tenant3_mod_source_record_storage.marc_records_lb = 0
• tenant0_mod_source_record_storage.raw_records_lb = 9604805
• tenant2_mod_source_record_storage.raw_records_lb = 0
• tenant3_mod_source_record_storage.raw_records_lb = 0
• tenant0_mod_source_record_storage.records_lb = 9674677
• tenant2_mod_source_record_storage.records_lb = 0
• tenant3_mod_source_record_storage.records_lb = 0
• tenant0_mod_source_record_storage.marc_indexers =  620042011
• tenant2_mod_source_record_storage.marc_indexers =  0
• tenant3_mod_source_record_storage.marc_indexers =  0
• tenant0_mod_source_record_storage.marc_indexers with field_no 010 = 3285833
• tenant2_mod_source_record_storage.marc_indexers with field_no 010 = 0
• tenant3_mod_source_record_storage.marc_indexers with field_no 010 = 0
• tenant0_mod_source_record_storage.marc_indexers with field_no 035 = 19241844
• tenant2_mod_source_record_storage.marc_indexers with field_no 035 = 0
• tenant3_mod_source_record_storage.marc_indexers with field_no 035 = 0
• tenant0_mod_inventory_storage.authority = 4
• tenant2_mod_inventory_storage.authority = 0
• tenant3_mod_inventory_storage.authority = 0
• tenant0_mod_inventory_storage.holdings_record = 9592559
• tenant2_mod_inventory_storage.holdings_record = 16
• tenant3_mod_inventory_storage.holdings_record = 16
• tenant0_mod_inventory_storage.instance = 9976519
• tenant2_mod_inventory_storage.instance = 32
• tenant3_mod_inventory_storage.instance = 32 
• tenant0_mod_inventory_storage.item = 10787893
• tenant2_mod_inventory_storage.item = 19
• tenant3_mod_inventory_storage.item = 19

PTF -environment ocp3 

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

...