Page Comparison

...

 2) During the new Data Import splitting feature testing, items for update were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get. Less than 1% of records could be discarded due to missing permission for  'data-import-system-user'. Permission was not added automatically during the service deployment. I added permission manually to the database and the error does not occur anymore.

...

 ** -  up to 10 items were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get. Less than 1% of records could be discarded due to missing permission for  'data-import-system-user'. Permission was not added automatically during the service deployment. I added permission manually to the database and the error does not occur anymore.

...

Data Import Robustness Enhancement 

...

Instance CPU Utilization 

Test 1. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 500, 2 runs for each test. The maximal CPU Utilization value is 38%. 

Image Added

Test 2. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 10K, 2 runs for each test.

Memory Utilization

Test 1. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 500, 2 runs for each test.

Most of the modules were stable during the test, and no memory leak is suspected for DI modules, only 2 modules increased memory consumption usage after the beginning of the tests
Image Added

Memory utilization rich maximal value for mod-source-record-storage-b 88%  and for mod-source-record-manager-b 85%.

Image Added

Test 2. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 10K, 2 runs for each test.

Service CPU Utilization 

Test 1. Test with 1, 2, and 3 tenants' concurrent jobs with configuration RECORDS_PER_SPLIT_FILE = 500, 2 runs for each test.

Image Added

Test #3 With CI/CO 20 users and DI 25k records on each of the 3 tenants

...

Test#2 Multitenant  DI (9 concurrent jobs)

Test#3 With CI/CO

Service CPU Utilization 

MARC BIB CREATE

Average CPU usage for mod-inventory -was 144%, mod-di-converter-storage was about 107%, and for all other modules did not exceed 100 %. We can observe spikes in CPU usage of mod-data-import at the beginning of the Data Import jobs up to 260%.

...

Test#2 Multitenant

Test#3 With CI/CO

Instance CPU Utilization

Test#1 500k records DI

Test#2 Multitenant DI (9 concurrent jobs)

...

• tenant0_mod_source_record_storage.marc_records_lb = 9674629
• tenant2_mod_source_record_storage.marc_records_lb = 0
• tenant3_mod_source_record_storage.marc_records_lb = 0
• tenant0_mod_source_record_storage.raw_records_lb = 9604805
• tenant2_mod_source_record_storage.raw_records_lb = 0
• tenant3_mod_source_record_storage.raw_records_lb = 0
• tenant0_mod_source_record_storage.records_lb = 9674677
• tenant2_mod_source_record_storage.records_lb = 0
• tenant3_mod_source_record_storage.records_lb = 0
• tenant0_mod_source_record_storage.marc_indexers =  620042011
• tenant2_mod_source_record_storage.marc_indexers =  0
• tenant3_mod_source_record_storage.marc_indexers =  0
• tenant0_mod_source_record_storage.marc_indexers with field_no 010 = 3285833
• tenant2_mod_source_record_storage.marc_indexers with field_no 010 = 0
• tenant3_mod_source_record_storage.marc_indexers with field_no 010 = 0
• tenant0_mod_source_record_storage.marc_indexers with field_no 035 = 19241844
• tenant2_mod_source_record_storage.marc_indexers with field_no 035 = 0
• tenant3_mod_source_record_storage.marc_indexers with field_no 035 = 0
• tenant0_mod_inventory_storage.authority = 4
• tenant2_mod_inventory_storage.authority = 0
• tenant3_mod_inventory_storage.authority = 0
• tenant0_mod_inventory_storage.holdings_record = 9592559
• tenant2_mod_inventory_storage.holdings_record = 16
• tenant3_mod_inventory_storage.holdings_record = 16
• tenant0_mod_inventory_storage.instance = 9976519
• tenant2_mod_inventory_storage.instance = 32
• tenant3_mod_inventory_storage.instance = 32 
• tenant0_mod_inventory_storage.item = 10787893
• tenant2_mod_inventory_storage.item = 19
• tenant3_mod_inventory_storage.item = 19

PTF -environment ocp3 

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

...