Overview

This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.

The purpose for this test is to define how concurrent DI affect duration of DI jobs on the central tenant and to check possible issues during smoke test with 50k DI Create job running concurrently on all 3 tenants.

Ticket:  PERF-715 - Getting issue details... STATUS

Summary

Data import duration approximately doubling with 10k and 25k jobs when increasing the number of concurrent jobs on different tenants. This trend is consistent across the central tenant and other tenants.

Smoke test with 50k didn't reveal some issues. Duration for 3 concurrent DI Create jobs was three times higher than one DI on central tenant that only confirm previous statement about concurrency affect.

Maximum average CPU utilization was different during create and update jobs. Top two modules during DI Create jobs in mod-inventory-b - 123%, mod-quick-marc-b - 76%, Update jobs mod-inventory-b - 182%, mod-quick-marc-b - 122%.

Memory consumption was almost the same for DI create and update jobs: Nevertheless it was slightly higher for update jobs in mod-inventory-b - 98%, mod-permission-b - 79%, mod-source-record-storage-b - 73%.

RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs

DB connections were higher during DI Create jobs. With 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870

Top long query for failed job on third tenant during DI Create job with 10k- SELECT jsonb,id FROM fs07000002_mod_inventory_storage.instance_holdings_item_view. Average latency- 386455.99 ms/call

Test Runs 

Test Results

Data import

As the number of concurrent jobs and file size grow duration of DI jobs grow proportionally. 

Smoke tests finished successfully for 3 concurrent DI Create jobs with 50k.

* - Errors occurred only in 10k DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.

• io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
• java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null

Service CPU Utilization

DI Create jobs

DI Update jobs

Service Memory Utilization

DI Create jobs

DI Update jobs

DB CPU Utilization

RDS CPU utilization was 97% for all Create jobs and 94% for Update jobs

Create jobs

Update jobs

DB Connections

DB connections for 2 tenants Create jobs - 710, for 3 tenants Create jobs - 870

DB connections for 2 tenants Create jobs - 630, for 3 tenants Create jobs - 785

DB connections needed for every additional job processing concurrently on different tenant - 150.

Create jobs

Update jobs

DB load

Create jobs

Update jobs

Appendix

Errors & Exceptions

During successfully finished tests exceptions were observed:

index [pcp1_instance_fs09000000], id [f7aea9b8-614e-4050-9dbd-e2f8a884c06b], message [OpenSearchException[OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16502737514/15.3gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16499671264/15.3gb], new bytes reserved: [3066250/2.9mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3103382/2.9mb]]]]

feign.FeignException$Unauthorized: [401 Unauthorized] during [GET] to [http://inventory-view/instances?query=id%3D%3D%28%221e9b752b-6cc3-433b-ae90-cbafdc307cb6%22%29&limit=1] [InventoryViewClient#getInstances(CqlQuery,int)]: [Invalid token]

Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-294 [ACTIVE]

pcp1/mod-authtoken

org.folio.auth.authtokenmodule.tokens.TokenValidationException: Access token has expired

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

 * - The newest version was used in this test to compare with the previous test

Methodology/Approach

DI tests were started from UI with 5-minute pauses between the tests.