Overview
This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.
The purpose for this test is to define how concurrent DI affect duration of DI jobs on the central tenant and to check possible issues during smoke test with 50k DI Create job running concurrently on all 3 tenants.
Ticket: PERF-715 - Getting issue details... STATUS
Summary
Data Import duration of 10k and 25k jobs approximately doubles when increasing the number of concurrent jobs on different tenants. This trend is consistent across the main/first tenant and other tenants.
Smoke test with 50k didn't reveal some issues. Duration for three concurrent DI Create jobs was 3x higher than one DI on the main tenant; this only confirm previous statement about the concurrency effect.
Maximum average CPU utilization was different during create and update jobs. Top two modules during DI Create jobs in mod-inventory-b - 123%, mod-quick-marc-b - 76%, Update jobs mod-inventory-b - 182%, mod-quick-marc-b - 122%.
Memory consumption was almost the same for DI create and update jobs: Nevertheless it was slightly higher for update jobs in mod-inventory-b - 98%, mod-permission-b - 79%, mod-source-record-storage-b - 73%.
RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs
DB connections were higher during DI Create jobs. With 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870
Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call
Test Runs
Test # | Scenario | Load level |
|---|---|---|
| 1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 tenants |
Test Results
As the number of concurrent Data Import jobs increases and file size grows, the duration of DI jobs grows proportionally.
Smoke Test finished successfully for 3 concurrent DI Create jobs of 50K each.
| DI Create | # of test | Number of concurrent jobs | Main tenant (fs09000000) | Second tenant (fs07000001) | Third tenant (fs07000002) |
|---|---|---|---|---|---|
| 10K | Baseline | 1 | 00:04:56 | ||
| 1 | 2 | 00:10:43 | 00:10:37 | ||
| 2 | 3 | 00:21:12 | 00:21:06 | 00:20:57 * | |
| 25K | Baseline | 1 | 00:11:24 | ||
| 3 | 2 | 00:23:44 | 00:23:30 | ||
| 4 | 3 | 00:37:11 | 00:37:05 | 00:36:58 | |
| DI Update | |||||
| 10K | Baseline | 1 | 00:06:32 | ||
| 5 | 2 | 00:09:47 | 00:11:26 | ||
| 6 | 3 | 00:19:08 | 00:19:06 | 00:18:31 | |
| 25K | Baseline | 1 | 00:15:13 | ||
| 7 | 2 | 00:30:49 | 00:30:52 | ||
| 8 | 3 | 00:47:47 | 00:48:17 | 00:47:54 | |
| DI Create (Smoke test) | |||||
| 50K | 9 | 1 | 00:22:31 | ||
| 10 | 3 | 01:12:54 | 01:12:44 | 01:12:35 |
* - Errors occurred only in 10K DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.
- io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
- java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null
Service CPU Utilization
DI Create jobs
DI Update jobs
Service Memory Utilization
DI Create jobs
DI Update jobs
DB CPU Utilization
RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs
Create jobs
Update jobs
DB Connections
DB connections for 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870
DB connections for 2 tenants Create jobs - 630, for 3 tenants Create jobs - 785
DB connections needed for every additional job processing concurrently on different tenant - 150.
Create jobs
Update jobs
DB load
Create jobs
Update jobs
Appendix
Errors & Exceptions
During successfully finished tests exceptions were observed:
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - Number of records in DB:
- fs09000000
- instances - 25.129.941
- items - 26.299.440
- holdings - 25.392.570
- fs07000001
- nstances - 10.039.613
- items - 1.423.844
- holdings - 10.461.259
- fs07000002
- nstances - 1.114.273
- items - 1.106.537
- holdings - 1.106.539
- fs09000000
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
| Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
| pcp1-pvt | |||||||||
| mod-remote-storage | 10(11)* | 3.0.0 | 2 | 4920 | 4472 | 1024 | 3960 | 512 | 512 |
| mod-data-import | 18(20)* | 3.0.7 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 |
| mod-authtoken | 13(16)* | 2.14.1 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
| mod-configuration | 9(10)* | 5.9.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| mod-users-bl | 9(10)* | 7.6.0 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
| mod-inventory-storage | 12(15)* | 27.0.3(27.0.4)* | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
| mod-circulation-storage | 12(14)* | 17.1.3(17.1.7)* | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
| mod-source-record-storage | 15(18)* | 5.7.3(5.7.5)* | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
| mod-inventory | 11(14)* | 20.1.3(20.1.7)* | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 |
| mod-di-converter-storage | 15(18)* | 2.1.2(2.1.5)* | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| mod-circulation | 12(14)* | 24.0.8(24.0.11)* | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 |
| mod-pubsub | 11(13)* | 2.11.2(2.11.3)* | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 |
| mod-patron-blocks | 9(10)* | 1.9.0 | 2 | 1024 | 896 | 1024 | 768 | 88 | 128 |
| mod-source-record-manager | 14(17)* | 3.7.4(3.7.8)* | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 |
| mod-quick-marc | 9(11)* | 5.0.0(5.0.1)* | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 |
| nginx-okapi | 9 | 2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 |
| okapi-b | 11 | 5.1.2 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
| mod-feesfines | 10(11)* | 19.0.0 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 |
| pub-okapi | 9 | 2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 |
Methodology/Approach
DI tests were started from UI concurrently with 1 job on each tenant, fs09000000 first and then on fs07000001 so in total two jobs on two tenants. Then 1 job on three tenants concurrently with several seconds delay - started with tenant fs09000000, second tenant -fs07000001 and third tenant - fs07000002.
DI Create jobs were conducted with 10k and 25k first. Then DI Update jobs.