Overview

This document contains the results of testing Data Import for MARC Bibliographic records at Ramsons Eureka [non-ECS] environment.

Testing results of 1k, 5k, 10k, 25k, 50k, and 100k data import, create, and update profiles are used to compare Eureka's performance with Okapi’s.
Current Ticket: https://folio-org.atlassian.net/browse/PERF-1085

Summary

Data imports for 1k, 5k, 10k, 25k, and 50k create are slower on the Eureka environment compared to the Okapi
1. 50K completed with errors, on UI status was “COMPLETED with errors“ but 100% of the records were created, details in the ticket;
2. 100K Failed to complete, problem could be in previously opened ticket.
Data imports for the 1k, 5k, 10k, 25k, and 50k update were completed successfully, but are slower, about 2 times slower on the Eureka environment compared to the Okapi. The problem is in mod-search queries.
Additional tests(tests results page last column) with mod-search disabled showed that the duration of DI of create and update is about the same on Eureka as on OKAPI.
The DI-related modules, including mod-data-import, mod-srs, mod-srm, mod-inventory, and mod-inventory-storage, demonstrated stable and consistent CPU utilization throughout both test scenarios. Additionally, memory usage remained smooth and uniform, with no notable issues or fluctuations observed
Eureka related modules, including kong and keycloak modules, during both tests had small CPU utilization(<5%). Memory utilization for folio-keycloak was high on both tests(~90%), another modules were consuming about 50%.
Sidecars CPU rich maximum 50% and demonstrated stable and consistent utilization, memory showed a growing trend and needs to be investigated.

Recommendations & Jiras

SideCars configuration folio-module-sidecar | Minimal CPU Requirements
During >50K records display incorrect status - https://folio-org.atlassian.net/browse/MODDATAIMP-1180
Data-import. Slow DB query [SELECT insert_journal_records($1::jsonb[])] https://folio-org.atlassian.net/browse/MODSOURMAN-1294
Increase memory for folio-keycloak
Investigate memory growing on the key DI modules sidecars https://folio-org.atlassian.net/browse/PERF-1096
Mod-serach queries with high latency https://folio-org.atlassian.net/browse/MSEARCH-972

Results

Test #	Data-import test	Profile	Duration Ramsons (rcp1) mod-search disabled	Duration Ramsons (rcp1) mod-search enabled	Duration Ramsons (rcon)	Duration Ramsons Eureka RECP1 mod-search enabled	Duration Ramsons Eureka RECP1 *mod-search disabled (1)**

Test #	Data-import test	Profile	Duration Ramsons (rcp1) mod-search disabled	Duration Ramsons (rcp1) mod-search enabled	Duration Ramsons (rcon)	Duration Ramsons Eureka RECP1 mod-search enabled	Duration Ramsons Eureka RECP1 *mod-search disabled (1)**
1	5k MARC BIB Create	PTF - Create 2	1 min	3 min 7 s	-	3 min 10 s	2 min 3 sec
	10k MARC BIB Create	PTF - Create 2	4 min 30 s	6 min 15 s	5 min 10 s	7 min 35 s	3 min 51 s
2	25k MARC BIB Create	PTF - Create 2	11 min	17 min	10 min 30 s	19 min 17 s	9 min 38 s
3	50k MARC BIB Create	PTF - Create 2	22 min	41 min 25 s	15 min 43 s	36 min 50 s Completed with errors	19 min 59 s
4	100k MARC BIB Create	PTF - Create 2	46 min	1 hr 19 min	31 min 51 s	Failed to complete	40 min 10 sec
	5k MARC BIB Update	PTF - Updates Success - 6	3 min 33 s	6 min 33 s	-	9 min 10 s	3 min 42 s
6	10k MARC BIB Update	PTF - Updates Success - 6	6 min 46 s	11 min 14 s	7 min 10 s	19 min 06 sec	6 min 43 s
7	25k MARC BIB Update	PTF - Updates Success - 6	16 min 40 s	28 min 43 s	19 min 3 s	48 min 41 s	17 min 12
8	50k MARC BIB Update	PTF - Updates Success - 6	33 min 45 s	58 min 30 s	38 min 53 sec	1 hr 40 min	34 min 14 s
9	100k MARC BIB Update	PTF - Updates Success - 6	1 hr 8 min	2 hr 14 min	1 hr 23 min	not tested	not tested

*1 DI tests were run in the scope of the ticket https://folio-org.atlassian.net/browse/PERF-1096 and resource utilization is not documented in this report.

Cluster resource utilization graphs

Test 1. Data-import Create

Modules CPU Utilization graphs

CPU Usage is stable for all modules involved for MARC BIB creates process

Eureka components were stable during the DI create and consume, just up to 5% of CPU

Modules memory utilization graphs

Memory usage for Create test sets showing stable trend. No memory leak suspects observed.

Eureka component during DI Create showed a stable trend. But folio-keycloak used about 90%, so the PTF recommendation is to add more memory for this module.

Sidecars resource utilization for DI Create

CPU Usage is stable for all sidecar modules.

The memory usage graph indicates that the DI related modules exhibit a steadily increasing trend in memory consumption. This pattern warrants further investigation to rule out potential memory leak issues.

RDS Metrics

The database remains stable and displays a CPU utilization pattern consistent with previous reports.

RDS Database Connections

Database load (Performance insights metrics)

Slow queries WITH cte AS…… were investigated in the previous Data-import report ECS and non-ECS reports Data Import test report Ramsons [ECS] Data Import test report Ramsons [non-ECS]