Overview

The Data Import Task Force (DITF) implements a feature that splits large input MARC files into smaller ones, resulting in smaller jobs, so that the big files could be imported and be imported consistently.  This document contaest contains 1. Test with 1, 2, and 3 tenants' concurrent jobs with configurins the configurations the results of performance tests on the feature and also an analysis the feature's performance with respect to the baseline tests.  The following Jiras were implemented. 

...

• The file-splitting feature is stable and offers more robustness to Data Import jobs even with the current infrastructure configuration. If there were failures, it's easier now to find the exact failed records to take actions on them. 
  • No stuck jobs in all tests performed.
  • There were errors (see below) in some partial jobs, but they still completed so the entire job status is "Completed with error".
  • Both of kinds of imports, create and update MARC BIBs worked well with this file-splitting feature enabled and also disabled. 
• There is no performance degradations, jobs not getting slower, on single-tenant imports. On multi-tenants imports, performance is be a little better
• Duration for DI correlates with number of the records imported (100k records- 38 min, 250k - 1 hour 32 min, 500k - 3 hours 29 min).
• Multitenant DI could be performed successfully for up to 9 jobs in parallel. If jobs are big they will start one by one in order for each tenant but processed in parallel on 3 tenants. Small DI (1 record) could be finished faster not in order. 
• No memory leak is suspected for all of the modules.
• Average CPU usage for mod-inventory -was 144%, mod-di-converter-storage was about 107%, and for all other modules did not exceed 100 %. We can observe spikes in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 260%.  Big improvement over the previous version (without file-splitting) for 500K imports where mod-di-converter-storage's CPU utilization was 462% and other modules were above 100% and up to 150%. 
• Approximately DB CPU usage is up to 95%.

...

1. One record on one tenant could be discarded with error: io.netty.channel.StacklessClosedChannelException.

   Jira Legacy
   server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODDATAIMP-748

   Reproduces in both cases with and without splitting feature enabled in at least 30% of test runs with 500k record files and multitenant testing.
2. During the new Data Import splitting feature testing, items for update were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get. Less than 1% of records could be discarded due to missing permission for  'data-import-system-user'. Permission was not added automatically during the service deployment. I added permission manually to the database and the error does not occur anymore.

   Jira Legacy
   server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODDATAIMP-930

3. UI issue, when canceled or completed with error Job progress bar cannot be deleted from the screen.

   Jira Legacy
   server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODDATAIMP-929

4. Usage:
   • Should not use less than 1000 for RECORDS_PER_SPLIT_FILE. The system is stable enough to ingest 1000 records consistently and smaller amounts will incur more overheads, resulting in longer jobs' durations.  CPU utilization for mod-di-converter-storage for 500 RECORDS_PER_SPLIT_FILE(RPSF) = 160%, for 1000RPSF =180%, for 5K RPSF =380% and for 10K RPSF =433%, so in the case of selecting configurations 5K or 10K we recommend to add more CPU to mod-di-converter-storage service.
   • When toggling the file-splitting feature, mod-source-record-storage, mod-source-record-manager's tasks need to be restarted.
   • Keep in mind about the Kafka broker's disk size (as bigger jobs - up to 500K - can be run now), consecutive jobs may use up the disk quickly because the messages' retention time currently is set at 8 hours. For example with 300GB disk size, consecutive jobs of 250K, 500K, 500K sizes will exhaust the disk. 
5. More CPU could be allocated to mod-inventory and mod-di-converter-storage

...

 * - One record on one tenant could be discarded with error: io.netty.channel.StacklessClosedChannelException.

...

 ** -  up to 10 items were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get. Less than 1% of records could be discarded due to missing permission for  'data-import-system-user'. Permission was not added automatically during the service deployment. I added permission manually to the database and the error does not occur anymore.

...

With CI/CO 20 users and DI 25k records on each of the 3 tenants Splitting Feature Disabled

ocp3-mod-data-import:12

Image Modified

Data Import Robustness Enhancement

...

Memory utilization rich maximal value for mod-source-record-storage-b 88%  and for mod-source-record-manager-b 85%.

Test 2. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 10K, 2 runs for each test.

...

Test 2. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 10K, 2 runs for each test.

CPU utilization of  mod-di-converter-storage-b

RDS CPU Utilization 

Test 1. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 500, 2 runs for each test. Maximal  CPU Utilization = 95%

...

Retest the DI feature to be sure that the new changes have not affected performance negatively.  Retest the DI file-splitting feature for the following scenarios:

...

• 250K MARC BIB Create PTF - Create 2 ---> 44 minutes
• 250K MARC BIB UpdatePTF - Updates Success - 1 -→ 45 minutes
• Multitenant MARC Create (100k, 50k, and 1 record)PTF - Create 2 -→1 hour 35 minutes
  • Check-Out without DI ~ 200ms
  • Check-In without DI ~ 650ms65ms
  • Check-Out with DI ~ 770ms
  • Check-in with DI ~ 330ms

...

• Service CPU utilization on Poppy is about the same as on the Orchid;
• Memory utilization on Poppy is about the same as on the Orchid;
• RDS CPU Utilization  during all tests and on both releases was about 96%;
• The number of connections to DB on both releases were was about the same from 550(Test 1.1) to 1200(Test 1.4).

Test 1.  Single tenant(primary fs09000000): create and update 250K file 

On Poppy with the split feature disabled, large files stopped processing. Created ticket to this problem

Test 1.4 With CI/CO 20 users and DI 25k records on each of the 3 tenants 

...

Service CPU Utilization 

The shark spike sharp spike of CPU at the beginning of test 1, We see similar behavior in all of the DI tests. СPU consumption was uniform during the test.

...

The goal of the tests was to investigate how the file-splitting feature caused Data-import on Poppy release and the impact of Refresh Token Rotation (RTR). The tests were performed on ocp3(Poppy), pcp1(Poppy) and ncp5(Orchid)  environments.

...

• Refresh Token Rotation configuration does not affect the data import process in any way, whether creating or updating a profile.
• In the Poppy release 250,000 records of data import with PTF - Create-2 job profile failed, and 50,000 records of data import with PTF - Updates Success - 1 job profile also failed in all of the tests, except configuration when FSF=ture;
• Data import works slowly on Poppy compared to the Orchid
• As the number of records in the file for data import increases, the processing time also increases. Up to 25,000 records, the duration of the data import is approximately the same.
• In the Poppy release data-import with an enabled file-splitting feature works slower compared to data-import with a disabled file-splitting feature.
• Data import is performed approximately 5% faster when the file-splitting feature parameters are absent in the task definition configuration.

Test results

Column with "pcp1 FSF false without token" has testing results on the configuration similar to "ocp3 FSF deleted false without RTR token".

Resource utilization during testing

...

• tenant0_mod_source_record_storage.marc_records_lb = 9674629
• tenant2_mod_source_record_storage.marc_records_lb = 0
• tenant3_mod_source_record_storage.marc_records_lb = 0
• tenant0_mod_source_record_storage.raw_records_lb = 9604805
• tenant2_mod_source_record_storage.raw_records_lb = 0
• tenant3_mod_source_record_storage.raw_records_lb = 0
• tenant0_mod_source_record_storage.records_lb = 9674677
• tenant2_mod_source_record_storage.records_lb = 0
• tenant3_mod_source_record_storage.records_lb = 0
• tenant0_mod_source_record_storage.marc_indexers =  620042011
• tenant2_mod_source_record_storage.marc_indexers =  0
• tenant3_mod_source_record_storage.marc_indexers =  0
• tenant0_mod_source_record_storage.marc_indexers with field_no 010 = 3285833
• tenant2_mod_source_record_storage.marc_indexers with field_no 010 = 0
• tenant3_mod_source_record_storage.marc_indexers with field_no 010 = 0
• tenant0_mod_source_record_storage.marc_indexers with field_no 035 = 19241844
• tenant2_mod_source_record_storage.marc_indexers with field_no 035 = 0
• tenant3_mod_source_record_storage.marc_indexers with field_no 035 = 0
• tenant0_mod_inventory_storage.authority = 4
• tenant2_mod_inventory_storage.authority = 0
• tenant3_mod_inventory_storage.authority = 0
• tenant0_mod_inventory_storage.holdings_record = 9592559
• tenant2_mod_inventory_storage.holdings_record = 16
• tenant3_mod_inventory_storage.holdings_record = 16
• tenant0_mod_inventory_storage.instance = 9976519
• tenant2_mod_inventory_storage.instance = 32
• tenant3_mod_inventory_storage.instance = 32 
• tenant0_mod_inventory_storage.item = 10787893
• tenant2_mod_inventory_storage.item = 19
• tenant3_mod_inventory_storage.item = 19

PTF -environment ocp3 

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

...