IN PROGRESS

Overview

This document contains the results of testing concurrent Data Import with file splitting feature for MARC Bibliographic records in the Poppy release.

The purpose for this test define how concurrent DI affect duration

Ticket:  PERF-715 - Getting issue details... STATUS

Summary

Test Runs 

Test Results

Data import

As the number of concurrent jobs and file size grow duration of DI jobs grow proportionally. 

Smoke tests finished successfully for 3 concurrent DI Create jobs with 50k.

* - Errors occurred only in 10k DI Create jobs running on third tenant during 3 concurrent jobs test. The errors did not reproduce during subsequent tests.

• io.vertx.core.impl.NoStackTraceThrowable: [{"id":"cf64277b-9945-49a1-93c0-007643c46efe","error":"Timeout for DB_HOST:DB_PORT=db.pcp1.folio-eis.us-east-1:5432","holdingId":"bd17bc47-72eb-480b-8a83-e0a1bc16e0f4"}]
• java.lang.NullPointerException: Cannot invoke "org.folio.processing.mapping.defaultmapper.processor.parameters.MappingParameters.getLinkingRules()" because "mappingParameters" is null

Comparison

Service CPU Utilization

Service Memory Utilization

DB CPU Utilization

DB Connections

DB load

Appendix

Errors & Exceptions

During successfully finished tests exceptions were observed:

index [pcp1_instance_fs09000000], id [f7aea9b8-614e-4050-9dbd-e2f8a884c06b], message [OpenSearchException[OpenSearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16502737514/15.3gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16499671264/15.3gb], new bytes reserved: [3066250/2.9mb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=3103382/2.9mb]]]]

feign.FeignException$Unauthorized: [401 Unauthorized] during [GET] to [http://inventory-view/instances?query=id%3D%3D%28%221e9b752b-6cc3-433b-ae90-cbafdc307cb6%22%29&limit=1] [InventoryViewClient#getInstances(CqlQuery,int)]: [Invalid token]

Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-294 [ACTIVE]

pcp1/mod-authtoken

org.folio.auth.authtokenmodule.tokens.TokenValidationException: Access token has expired

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

 * - The newest version was used in this test to compare with the previous test

Methodology/Approach

DI tests were started from UI with 5-minute pauses between the tests.