Overview

This document contains the results of testing Data Import Create MARC holdings records in Orchid to detect performance trends.

...

Recommendations & Jiras

Test Results

*errors found during the data import job:

• Connection is closed
• io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: db.ncp5.folio-eis.us-east-1/10.23.34.172:5432
• io.netty.channel.StacklessClosedChannelException
• io.vertx.core.impl.NoStackTraceThrowable: Couldn't update JobExecution status, JobExecution already marked as ERROR
• io.vertx.core.impl.NoStackTraceThrowable: Fail to read any response from the server, the underlying connection might get lost unexpectedly.
• io.vertx.core.impl.NoStackTraceThrowable: Timeout
• io.vertx.core.impl.NoStackTraceThrowable: Timeout for DB_HOST:DB_PORT=db.ncp5.folio-eis.us-east-1:5432
• io.vertx.pgclient.PgException: FATAL: the database system is in recovery mode (57P03)
• java.io.IOException: Connection reset by peer
• org.folio.processing.exceptions.EventProcessingException: 504 Gateway Time-out
• org.folio.processing.exceptions.EventProcessingException: Invalid token: Unexpected user response code 500 for user id 9eb67301-6f6e-468f-9b1a-6134dc39a684
• org.folio.processing.exceptions.EventProcessingException: Timeout for DB_HOST:DB_PORT=db.ncp5.folio-eis.us-east-1:5432
• org.folio.processing.exceptions.EventProcessingException: Unable to retrieve permissions for user with id '9eb67301-6f6e-468f-9b1a-6134dc39a684': Unable to retrieve permissions (code 400): Timeout for DB_HOST:DB_PORT=db.ncp5.folio-eis.us-east-1:5432
• This connection has been closed.
• Timeout

Instance CPU Utilization

Image AddedImage Added

Service CPU Utilization

Image Added

Image Added

Image Added

Image Added

Memory Utilization

Image Added

Image Added

Image AddedImage Added

DB CPU Utilization

Image Added

Image Added

DB Connections

Image Added

Image Added

DB Load

Image AddedImage Added

SQL queries

Image Added

Image Added

nfrastructure

PTF -environment ncp5 

• 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, one reader, and one writer

  Name	API Name	Memory GIB	vCPUs	max_connections
  R6G Extra Large	db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• Kafka topics partitioning: - 2 partitions for DI topics

Modules memory and CPU parameters

512