PTF - Data Import Creates + Updates multi tenant [non-ECS]
- 1 Overview
- 2 Summary
- 3 Test Runs
- 4 Test Results
- 4.1 Comparison
- 4.2 Service CPU Utilization
- 4.3 Service Memory Utilization
- 4.4 MSK tenant cluster
- 4.4.1 Disk usage by broker
- 4.4.2 CPU (User) usage by broker
- 4.5 DB CPU Utilization
- 4.6 DB Connections
- 4.7 DB load
- 4.7.1 TOP SQL Queries
- 5 Appendix
Overview
This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Quesnelia release.
The purpose for this test is to define how concurrent DI affect duration of DI jobs on the central tenant and to check possible issues during smoke test with 50k DI Create job running concurrently on all 3 tenants.
Compared with Data Import Creates + Updates Poppy pcp1
Ticket: 
Summary
Data Import duration of 10k and 25k jobs multiplies in correspondence with number of concurrent DI jobs. Duration doubles with 2 concurrent jobs (1 job for each tenant) and 3 times longer for 3 concurrent jobs. This trend is consistent across tenants. The same as in Poppy.
Running smoke test 50k file with DI create job revealed issue with module mod-permission which spiked and led to 4 split jobs got SNAPSHOT_UPDATE_ERROR at the beginning of the test on third fs07000002 tenant. No error messages in UI. Reason - No action.
Average CPU utilization was for top two modules during DI Create jobs in mod-inventory-b - 121%, mod-quick-marc-b - 92%, Update jobs mod-inventory-b - 150%, mod-quick-marc-b - 92%. Modules used less resources in quesnelia. More information can be found in Service CPU Utilization tab.
Memory consumption: mod-inventory-b - 87%, mod-data-import-b - 80%, mod-source-record-storage-b - 73%, mod-permissions-b - 71% which is slightly less than in Poppy
RDS CPU utilization was 88% for all jobs that is 10% less than in Poppy
Create jobs DB connections for 2 tenants - 990, for 3 tenants - 1100, Update jobs DB connections for 2 tenants - 950, for 3 tenants - 1000
Top long query was SELECT jsonb FROM fs07000002_mod_permissions.permissions - rows 4275 rows/sec, average latency 1579 ms/call
Test Runs
Test # | Scenario | Load level |
|---|---|---|
1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 tenants |
Test Results
Duration of DI jobs grow proportionally to file size.
Smoke Test finished successfully for 3 concurrent DI Create jobs of 50K each.
DI Create | # of test | Number of concurrent jobs | Main tenant (fs09000000) | Second tenant (fs07000001) | Third tenant (fs07000002) |
|---|---|---|---|---|---|
10k | Baseline | 1 | 00:05:01 | 00:05:01 | 00:05:00 |
1 | 2 | 00:09:04 | 00:09:08 | ||
2 | 3 | 00:14:25 | 00:15:12 | 00:15:22 | |
25k | Baseline | 1 | 00:10:43 | 00:11:15 | 00:11:22 |
3 | 2 | 00:22:44 | 00:22:47 | ||
4 | 3 | 00:40:32 | 00:40:17 | 00:40:25 | |
DI Update | |||||
10k | Baseline | 1 | 00:06:25 | ||
5 | 2 | 00:14:08 | 00:13:44 | ||
6 | 3 | 00:20:13 | 00:21:03 | 00:20:41 | |
25k | Baseline | 1 | 00:15:17 | ||
7 | 2 | 00:35:27 | 00:35:34 | ||
8 | 3 | 00:58:49 | 00:58:59 | 00:59:05 | |
DI Create (Smoke test) | |||||
50K | 9 | 1 | 22:44.2 | ||
10 | 3 | 01:21:37 | 01:22:49 | 01:22:49 |
Comparison
QUESNELIA | POPPY | COMPARISON Quesnelia/Poppy | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
DI Create | # of test | Number of concurrent jobs | Central tenant | Second tenant | Third tenant | DI Create | Number of concurrent jobs | Central tenant | Second tenant | Third tenant | Delta fs09000000 | Delta fs07000001 | Delta fs07000002 |
10k | Baseline | 1 | 00:05:01 | 00:05:01 | 00:05:00 | 10k | 1 | 00:04:56 | 00:00:05 | ||||
1 | 2 | 00:09:04 | 00:09:08 | 2 | 00:10:43 | 00:10:37 | 00:01:39 | 00:01:29 | |||||
2 | 3 | 00:14:25 | 00:15:12 | 00:15:22 | 3 | 00:21:12 | 00:21:06 | 00:20:57 | 00:06:47 | 00:05:54 | 00:05:35 | ||
25k | Baseline | 1 | 00:10:43 | 00:11:15 | 00:11:22 | 25k | 1 | 00:11:24 | 00:00:41 | ||||
3 | 2 | 00:22:44 | 00:22:47 | 2 | 00:23:44 | 00:23:30 | 00:01:00 | 00:00:43 | |||||
4 | 3 | 00:40:32 | 00:40:17 | 00:40:25 | 3 | 00:37:11 | 00:37:05 | 00:36:58 | 00:03:21 | 00:03:12 | 00:03:27 | ||
DI Update | # of test | Number of concurrent jobs | Central tenant | Second tenant | Third tenant | DI Update | Number of concurrent jobs | Central tenant | Second tenant | Third tenant | Delta fs09000000 | Delta fs07000001 | Delta fs07000002 |
10k | Baseline | 1 | 00:06:25 | 10k | 1 | 00:06:32 | 00:00:07 | ||||||
5 | 2 | 00:14:08 | 00:13:44 | 2 | 00:09:47 | 00:11:26 | 00:04:21 | 00:02:18 | |||||
6 | 3 | 00:20:13 | 00:21:03 | 00:20:41 | 3 | 00:19:08 | 00:19:06 | 00:18:31 | 00:01:05 | 00:01:57 | 00:02:10 | ||
25k | Baseline | 1 | 00:15:17 | 25k | 1 | 00:15:13 | 00:00:04 | ||||||
7 | 2 | 00:35:27 | 00:35:34 | 2 | 00:30:49 | 00:30:52 | 00:04:38 | 00:04:42 | |||||