IN PROGRESS
...
Test # | Scenario | Load level | Comment |
|---|---|---|---|
| 1 | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants | |
| 2 | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants | |
| 3 | DI MARC Bib Create | 50k concurrently on 3 tenants - smoke test |
Test Results
Data import
As the number of concurrent jobs and file size grow duration of DI jobs grow proportionally.
Smoke tests finished successfully for 3 concurrent DI Create jobs with 50k.
| DI Create | # of test | Number of concurrent jobs | Central tenant | Second tenant | Third tenant |
|---|---|---|---|---|---|
| 10k | Baseline | 1 | 00:04:56 | ||
| 1 | 2 | 00:10:43 | 00:10:37 | ||
| 2 | 3 | 00:21:12 | 00:21:06 | 00:20:57 * | |
| 25k | Baseline | 1 | 00:11:24 | ||
| 3 | 2 | 00:23:44 | 00:23:30 | ||
| 4 | 3 | 00:37:11 | 00:37:05 | 00:36:58 | |
| DI Update | |||||
| 10k | Baseline | 1 | 00:06:32 | ||
| 5 | 2 | 00:09:47 | 00:11:26 | ||
| 6 | 3 | 00:19:08 | 00:19:06 | 00:18:31 | |
| 25k | Baseline | 1 | 00:15:13 | ||
| 7 | 2 | 00:30:49 | 00:30:52 | ||
| 8 | 3 | 00:47:47 | 00:48:17 | 00:47:54 | |
| DI Create (smoke test) | |||||
| 50k | 9 | 1 | 00:22:31 | ||
| 10 | 3 | 01:12:54 | 01:12:44 | 01:12:35 |
* - Errors occurred only in 10k DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.
- io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
- java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null
Comparison
Service CPU Utilization
...
DB Connections
DB load
Appendix
Errors & Exceptions
During successfully finished tests exceptions were observed:
| Expand | |||||
|---|---|---|---|---|---|
| |||||
index [pcp1_instance_fs09000000], id [f7aea9b8-614e-4050-9dbd-e2f8a884c06b], message [OpenSearchException[OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16502737514/15.3gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16499671264/15.3gb], new bytes reserved: [3066250/2.9mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3103382/2.9mb]]]]
org.folio.search.exception.SearchOperationException: Failed to perform elasticsearch request [index=pcp1_contributor_fs09000000, type=bulkApi, message: 30,000 milliseconds timeout on connection http-outgoing-265 [ACTIVE]]
WARN essageBatchProcessor Failed to process batch, attempting to process resources one by one
|
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...