| Table of Contents |
|---|
...
Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call
Test Runs
Test # | Scenario | Load level |
|---|---|---|
| 1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 tenants |
...
Update jobs
DB Connections
Create jobs DB connections for 2 tenants Create jobs tenants - 710, for 3 tenants Create jobs - 870
Update jobs DB connections for 2 tenants Create jobs tenants - 630, for 3 tenants Create jobs - 785
DB connections needed for every additional job processing concurrently on different tenant - 150.
DB Connections for Create jobs
DB Connections for Update jobs
DB load
Create jobs
...
| Expand | |||||
|---|---|---|---|---|---|
| |||||
failure in bulk execution - 186 errors during all update jobs, >4000 errors during create jobs
index [pcp1_instance_fs09000000], id [f7aea9b8-614e-4050-9dbd-e2f8a884c06b], message [OpenSearchException[OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16502737514/15.3gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16499671264/15.3gb], new bytes reserved: [3066250/2.9mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3103382/2.9mb]]]]
org.folio.search.exception.SearchOperationException: Failed to perform elasticsearch request [index=pcp1_contributor_fs09000000, type=bulkApi, message: 30,000 milliseconds timeout on connection http-outgoing-265 [ACTIVE]]
WARN essageBatchProcessor Failed to process batch, attempting to process resources one by one
number of errors - 23400. The errors happen only during DI in fs07000002 tenant | filter @logStream like "pcp1/mod-authtoken" | filter @message like "ERROR FilterApi" 13:48:00 [595516/users] [fs07000002] [] [mod-authtoken] ERROR FilterApi Unable to retrieve permissions for system-user: User does not exist: 8cc96687-ea63-44cb-ab5f-a73bc6985324 request took 7 ms |
...
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - Number of records in DB:
- fs09000000
- instances - 25.129.941
- items - 26.299.440
- holdings - 25.392.570
- fs07000001
- nstances - 10.039.613
- items - 1.423.844
- holdings - 10.461.259
- fs07000002
- nstances - 1.114.273
- items - 1.106.537
- holdings - 1.106.539
- fs09000000
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...