Data Import Creates + Updates multi tenant with file split enabled
- Leonid Kolesnykov
- Martin Tran
Overview
This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.
The purpose for this test is to define how concurrent DI affect duration of DI jobs on the central tenant and to check possible issues during smoke test with 50k DI Create job running concurrently on all 3 tenants.
Ticket: PERF-715 - Getting issue details... STATUS
Summary
Data Import duration of 10k and 25k jobs approximately doubles when increasing the number of concurrent jobs on different tenants. This trend is consistent across the main/first tenant and other tenants.
Smoke test with 50k didn't reveal some issues. Duration for three concurrent DI Create jobs was 3x higher than one DI on the main tenant; this only confirm previous statement about the concurrency effect.
Maximum average CPU utilization was different during create and update jobs. Top two modules during DI Create jobs in mod-inventory-b - 123%, mod-quick-marc-b - 76%, Update jobs mod-inventory-b - 182%, mod-quick-marc-b - 122%.
Memory consumption was almost the same for DI create and update jobs: Nevertheless it was slightly higher for update jobs in mod-inventory-b - 98%, mod-permission-b - 79%, mod-source-record-storage-b - 73%.
RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs
DB connections were higher during DI Create jobs. With 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870
Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call
Test Runs
Test # | Scenario | Load level |
|---|---|---|
| 1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 tenants |
Test Results
As the number of concurrent Data Import jobs increases and file size grows, the duration of DI jobs grows proportionally.
Smoke Test finished successfully for 3 concurrent DI Create jobs of 50K each.
| DI Create | # of test | Number of concurrent jobs | Main tenant (fs09000000) | Second tenant (fs07000001) | Third tenant (fs07000002) |
|---|---|---|---|---|---|
| 10K | Baseline | 1 | 00:04:56 | ||
| 1 | 2 | 00:10:43 | 00:10:37 | ||
| 2 | 3 | 00:21:12 | 00:21:06 | 00:20:57 * | |
| 25K | Baseline | 1 | 00:11:24 | ||
| 3 | 2 | 00:23:44 | 00:23:30 | ||
| 4 | 3 | 00:37:11 | 00:37:05 | 00:36:58 | |
| DI Update | |||||
| 10K | Baseline | 1 | 00:06:32 | ||
| 5 | 2 | 00:09:47 | 00:11:26 | ||
| 6 | 3 | 00:19:08 | 00:19:06 | 00:18:31 | |
| 25K | Baseline | 1 | 00:15:13 | ||
| 7 | 2 | 00:30:49 | 00:30:52 | ||
| 8 | 3 | 00:47:47 | 00:48:17 | 00:47:54 | |
| DI Create (Smoke test) | |||||
| 50K | 9 | 1 | 00:22:31 | ||
| 10 | 3 | 01:12:54 | 01:12:44 | 01:12:35 |
* - Errors occurred only in 10K DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.
- io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
- java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null
Service CPU Utilization
DI Create jobs
DI Update jobs
