Data Import Creates + Updates multi tenant with file split enabled
Overview
This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.
The purpose for this test is to define how concurrent DI affect duration of DI jobs on the central tenant and to check possible issues during smoke test with 50k DI Create job running concurrently on all 3 tenants.
Ticket: 
Summary
Data Import duration of 10k and 25k jobs approximately doubles when increasing the number of concurrent jobs on different tenants. This trend is consistent across the main/first tenant and other tenants.
Smoke test with 50k didn't reveal some issues. Duration for three concurrent DI Create jobs was 3x higher than one DI on the main tenant; this only confirm previous statement about the concurrency effect.
Maximum average CPU utilization was different during create and update jobs. Top two modules during DI Create jobs in mod-inventory-b - 123%, mod-quick-marc-b - 76%, Update jobs mod-inventory-b - 182%, mod-quick-marc-b - 122%.
Memory consumption was almost the same for DI create and update jobs: Nevertheless it was slightly higher for update jobs in mod-inventory-b - 98%, mod-permission-b - 79%, mod-source-record-storage-b - 73%.
RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs
DB connections were higher during DI Create jobs. With 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870
Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call
Test Runs
Test # | Scenario | Load level |
|---|---|---|
1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 tenants |
Test Results
As the number of concurrent Data Import jobs increases and file size grows, the duration of DI jobs grows proportionally.
Smoke Test finished successfully for 3 concurrent DI Create jobs of 50K each.
DI Create | # of test | Number of concurrent jobs | Main tenant (fs09000000) | Second tenant (fs07000001) | Third tenant (fs07000002) |
|---|---|---|---|---|---|
10K | Baseline | 1 | 00:04:56 |
|
|
1 | 2 | 00:10:43 | 00:10:37 |
| |
2 | 3 | 00:21:12 | 00:21:06 | 00:20:57 * | |
25K | Baseline | 1 | 00:11:24 |
|
|
3 | 2 | 00:23:44 | 00:23:30 |
| |
4 | 3 | 00:37:11 | 00:37:05 | 00:36:58 | |
DI Update |
|
|
|
|
|
10K | Baseline | 1 | 00:06:32 |
|
|
5 | 2 | 00:09:47 | 00:11:26 |
| |
6 | 3 | 00:19:08 | 00:19:06 | 00:18:31 | |
25K | Baseline | 1 | 00:15:13 |
|
|
7 | 2 | 00:30:49 | 00:30:52 |
| |
8 | 3 | 00:47:47 | 00:48:17 | 00:47:54 | |
DI Create (Smoke test) |
|
|
|
|
|
50K | 9 | 1 | 00:22:31 |
|
|
10 | 3 | 01:12:54 | 01:12:44 | 01:12:35 |
* - Errors occurred only in 10K DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.
io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null
Service CPU Utilization
DI Create jobs
DI Update jobs
Service Memory Utilization
DI Create jobs
DI Update jobs
DB CPU Utilization
RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs
Create jobs
Update jobs
DB Connections
Create jobs DB connections for 2 tenants - 710, for 3 tenants - 870
Update jobs DB connections for 2 tenants - 630, for 3 tenants - 785
DB connections needed for every additional job processing concurrently on different tenant - 150.
DB Connections for Create jobs
DB Connections for Update jobs
DB load
Create jobs
Update jobs
Appendix
Errors & Exceptions
During successfully finished tests exceptions were observed:
Infrastructure
PTF -environment pcp1
10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Number of records in DB:
fs09000000
instances - 25.129.941
items - 26.299.440
holdings - 25.392.570
fs07000001
nstances - 10.039.613
items - 1.423.844
holdings - 10.461.259
fs07000002
nstances - 1.114.273
items - 1.106.537
holdings - 1.106.539
MSK tenant
4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB