| Table of Contents |
|---|
...
| Table of Contents |
|---|
This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.
...
Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call
Test Runs
Test # | Scenario | Load level |
|---|---|---|
| 1 - Concurrent Create imports | DI MARC Bib Create | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 2 - Concurrent Update imports | DI MARC Bib Update | 10K, 25K concurrently (with 5 min pause) on 2 and 3 tenants |
| 3 - Concurrent Create imports ("smoke test") of 50K | DI MARC Bib Create | 50k concurrently on 3 tenants |
...
During successfully finished tests exceptions were observed:
| Expand | |||||
|---|---|---|---|---|---|
| |||||
failure in bulk execution - 186 errors during all update jobs, >4000 errors during create jobs
index [pcp1_instance_fs09000000], id [f7aea9b8-614e-4050-9dbd-e2f8a884c06b], message [OpenSearchException[OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16502737514/15.3gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16499671264/15.3gb], new bytes reserved: [3066250/2.9mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3103382/2.9mb]]]]
org.folio.search.exception.SearchOperationException: Failed to perform elasticsearch request [index=pcp1_contributor_fs09000000, type=bulkApi, message: 30,000 milliseconds timeout on connection http-outgoing-265 [ACTIVE]] org.folio.auth.authtokenmodule.tokens.TokenValidationException: Access token has expired
WARN essageBatchProcessor Failed to process batch, attempting to process resources one by one
number of errors - 23400. The errors happen only during 25k DI in fs07000002 tenant| filter @logStream like "pcp1/mod-authtoken" | filter @message like "ERROR FilterApi" 13:48:00 [595516/users] [fs07000002] [] [mod-authtoken] ERROR FilterApi Unable to retrieve permissions for system-user: User does not exist: 8cc96687-ea63-44cb-ab5f-a73bc6985324 request took 7 ms |
Infrastructure
PTF -environment pcp1
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, writer/reader
instancesDB fs09000000Name Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 Memory GIB vCPUs max_connections db.r6g.xlarge
32 GiB 4 vCPUs 2731 - Number of records in DB:
- fs09000000
- instances - 25.129.941
- items - 26.299.440
- holdings - 25.392.570
- fs07000001
- nstances - 10.039.613
- items - 1.423.844
- holdings - 10.461.259
- fs07000002
- nstances - 1.114.273
- items - 1.106.537
- holdings - 1.106.539
- fs09000000
- MSK tenant
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
...