IN PROGRESS
...
Ticket:
| Jira Legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Summary
...
Summary
Data import duration approximately doubling with 10k and 25k jobs when increasing the number of concurrent jobs on different tenants. This trend is consistent across the central tenant and other tenants.
Smoke test with 50k didn't reveal some issues. Duration for 3 concurrent DI Create jobs was three times higher than one DI on central tenant that only confirm previous statement about concurrency affect.
Maximum average CPU utilization was different during create and update jobs. Top two modules:
- Create jobs
- mod-inventory-b - 123%
- mod-quick-marc-b - 76%
- Update jobs
- mod-inventory-b - 182%
- mod-quick-marc-b - 122%
Memory consumption for top 3 modules were almost the same for DI create and update jobs: mod-inventory-b - 98%, mod-permission-b - 79%, mod-source-record-storage-b - 73%.
RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs
DB connections were higher during DI Create jobs. With 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870
Test Runs
Test # | Scenario | Load level | Comment |
|---|---|---|---|
| 1 | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants | |
| 2 | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants | |
| 3 | DI MARC Bib Create | 50k concurrently on 3 tenants - smoke test |
...
- io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
- java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null
Comparison
Service CPU Utilization
| Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
...
| Expand | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
...
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...