Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Ramsons Eureka [non-ECS] environment.

Testing results of 1k, 5k, 10k, 25k, 50k, and 100k data import, create, and update profiles are used to compare Eureka's performance with Okapi’s.
Current Ticket: PERF-1085: [Ramsons] [Eureka] [Data import] Update and Create MARC BIB RecordsClosedPreview

Summary

1. Data imports for 1k, 5k, 10k, 25k, and 50k create are slower on the Eureka environment compared to the Okapi

   1. 50K completed with errors, on UI status was “COMPLETED with errors“ but 100% of the records were created, details in the ticket;

   2. 100K Failed to complete, problem could be in previously opened ticket.

2. Data imports for the 1k, 5k, 10k, 25k, and 50k update were completed successfully, but are slower, about 2 times slower on the Eureka environment compared to the Okapi. The problem is in mod-search queries.
   Additional tests(tests results page last column) with mod-search disabled showed that the duration of DI of create and update is about the same on Eureka as on OKAPI.

3. The DI-related modules, including mod-data-import, mod-srs, mod-srm, mod-inventory, and mod-inventory-storage, demonstrated stable and consistent CPU utilization throughout both test scenarios. Additionally, memory usage remained smooth and uniform, with no notable issues or fluctuations observed

4. Eureka related modules, including kong and keycloak modules, during both tests had small CPU utilization(<5%). Memory utilization for folio-keycloak was high on both tests(~90%), another modules were consuming about 50%.

5. Sidecars CPU rich maximum 50% and demonstrated stable and consistent utilization, memory showed a growing trend and needs to be investigated.

Recommendations & Jiras

• SideCars configuration folio-module-sidecar | Minimal CPU RequirementsPreview

• During >50K records display incorrect status - MODDATAIMP-1180: During DI >50K records displays incorrect statusClosedPreview

• Data-import. Slow DB query [SELECT insert_journal_records($1::jsonb[])] MODSOURMAN-1294: Data-import. Slow DB query [SELECT insert_journal_records($1::jsonb[])]ClosedPreview

• Increase memory for folio-keycloak

• Investigate memory growing on the key DI modules sidecars PERF-1096: Investigate memory growing on sidecars during DIClosedPreview

• Mod-serach queries with high latency MSEARCH-972: Investigation - slow running query during data import on big datasetsClosedPreview

Results

*1 DI tests were run in the scope of the ticket PERF-1096: Investigate memory growing on sidecars during DIClosedPreview and resource utilization is not documented in this report.

Cluster resource utilization graphs

Test 1. Data-import Create

Modules CPU Utilization graphs

CPU Usage is stable for all modules involved for MARC BIB creates process

Open

Eureka components were stable during the DI create and consume, just up to 5% of CPU

Open

Modules memory utilization graphs

Memory usage for Create test sets showing stable trend. No memory leak suspects observed.

Open

Eureka component during DI Create showed a stable trend. But folio-keycloak used about 90%, so the PTF recommendation is to add more memory for this module.

Open

Sidecars resource utilization for DI Create

CPU Usage is stable for all sidecar modules.

Open

The memory usage graph indicates that the DI related modules exhibit a steadily increasing trend in memory consumption. This pattern warrants further investigation to rule out potential memory leak issues.

Open

RDS Metrics

The database remains stable and displays a CPU utilization pattern consistent with previous reports.

Open

RDS Database Connections

Open

Database load (Performance insights metrics)

Open

Open

Slow queries WITH cte AS…… were investigated in the previous Data-import report ECS and non-ECS reports Data Import test report Ramsons [ECS]Preview Data Import test report Ramsons [non-ECS]Preview

Test 2. Data-import Update

Modules CPU Utilization graphs

CPU Usage is stable for all modules involved for MARC BIB updates

Open

Eureka components were stable during the DI update and consume, just up to 5% of CPU, so about the same behaviour like during create DI

Open

Sidecars

Modules memory utilization graphs

Memory usage for Updatetest sets showing stable trend. No memory leak suspects observed.

Open

Eureka component during DI Update showed a stable trend. But folio-keycloak used about 88%(like during DI Create ), so the PTF recommendation is to add more memory for this module.

Open

Sidecars resource utilization for DI Update

CPU Usage is stable for all sidecar modules.

Open

The memory usage graph indicates that the DI related modules exhibit a steadily increasing trend in memory consumption. This pattern same during DI creating proccess.

Open

On this graph we can see DI related sidecar modules that have groving memory utilization trend

Open

RDS Metrics

The database remains stable and displays a CPU utilization pattern consistent with previous reports.

Open

RDS Database Connections

Open

Database load (Performance insights metrics)

Open

Open

MSK Resource utilization

CPU (User) usage by broker during DI Create

Open

CPU (User) usage by broker during DI Update

Open

OpenSearch resource utilization

Maximum CPU utilization percentage for all data nodes during Test 1. DI Create

Open

CPU utilization percentage for the master node.

Open

Maximum CPU utilization percentage for all data nodes during Test 2. DI Update

Open

Open

Appendix

Infrastructure

PTF -environment RECP1

• 10 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• db.r6.xlarge database instances, writer

• MSK fse-test

  • 4 kafka.m7g.xlarge brokers in 2 zones

  • Apache Kafka version 3.7.x (KRaft mode)

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true

  • log.retention.minutes=480

  • default.replication.factor=3

• OpenSearch 2.13 ptf-test cluster

  • r6g.2xlarge.search 4 data nodes

  • r6g.large.search 3 dedicated master nodes

RECP1 Dataset

Open