Overview

PERF-964 - Getting issue details... STATUS

[ A brief introduction about the content of the page:

What we are testing? Provide context of the test. Is it for a new service? Is it an experiment? Is it regression test?
- Include major things like environment settings (ECS, non-ECS, Eureka, non-Eureka, w/RW split, etc…)
What are the goals of the testing? Ex: Want to see the effect of using a different ec2 instance type. If regression: to see how vB compares to vA
- Include defined SLAs, if available
Reference the Jira(s)

]

Summary

[ A bulleted-list of the most important and relevant observations from the test results. What are the most important things the readers need to know about this testing effort? Some suggestions

Comparison to previous test or release of response times or API durations
Any notable changes
Particular response time or durations
Service memory and/or CPU utilization
RDS memory and/or CPU utilization
Other interesting observations

The summary points should answer the goals stated in Overview: did the test achieve the goals laid out? What goals were not met and why? SLAs were met or not?

]

Recommendations & Jiras (Optional)

[ If there are recommendations for the developers or operations team, or anything worth calling out, list them here.

Configuration options
Memory/CPU settings
Environment variables settings.

Also include any Jiras created for follow-up work]

Results

Test #	Data-import test	Profile	Duration Ramsons (rcon)	Duration Quesnelia (qcon)	Duration Quesnelia (qcp1)	Difference, %	Results
	10k MARC BIB Create	PTF - Create 2	5 min 10 s	4 min 14 sec	6 minutes	-29%	Completed
	25k MARC BIB Create	PTF - Create 2	10 min 30 s	9 min 41 sec	13 min 41 sec	-29%	Completed
	50k MARC BIB Create	PTF - Create 2	15 min 43 s	18 min 18 sec	21 min 59 sec	-16%	Completed
	100k MARC BIB Create	PTF - Create 2	31 min 51 s	38 min 36 sec	40 min 16 sec	-4%	Completed
	500k MARC BIB Create	PTF - Create 2	2 hr 37 min	3 hours 30 min	3 hours 27 min	+1.84%	Completed
11	10k MARC BIB Update	PTF - Updates Success - 6	7 min 10 s	5 min 59 sec	10 min 27 sec	-42%	Completed
12	25k MARC BIB Update	PTF - Updates Success - 6	19 min 3 s	19 min 52 sec	23 min 16 sec	-14%	Completed
13	50k MARC BIB Update	PTF - Updates Success - 6	38 min 53 sec	37 min 53 sec	40 min 52 sec	-7%	Completed
14	100k MARC BIB Update	PTF - Updates Success - 6	1 hr 23 min	1 hrs 14 min	1 hrs 2 min	+19%	Completed
15	500k MARC BIB Update	PTF - Updates Success - 6		6 hrs 39 min	5 hrs 31 min	+21%	Completed

Memory Utilization

[Description of notable observations of memory utilization with screenshots(of all modules and involved modules) and tables]

	Nolana Avg	Nolana Min	Nolana Max
mod-circulation-storage	24%	23%	25%
mod-patron-blocks	34%	33%	34%

CPU Utilization

[Description of notable observations of modules and eCPU utilization with screenshots (of all modules and involved modules) and tables]. Annotate graphs to show when a specific test started or ended, and select only the modules that are relevant to the test to show on the graphs]

RDS CPU Utilization

[Description of notable observations of reader and writer instances CPU utilization with screenshots and tables, RDS Database connections, and other Database metrics]

Database CPU was stable for 10 K 205K 50 K 100 K and 500,000 records

Open Search service

Open search CPU utilisation didn’t exceeds 30% on both data and master nodes, showing stable trend.

Additional information from module and database logs

Discussion (Optional)

[ This section gives more space to elaborate on any observations and results. See Perform Lookups By Concatenating UUIDs (Goldenrod)#Discussions for example. Anything that was discussed at length at the DSUs are worthy to be included here]

Errors

This section should detail any errors encountered during the testing process, their impact on testing outcomes, and the steps taken to address these issues.

Appendix

Infrastructure

PTF -environment rcon

11 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
db.r6.xlarge database instances, writer
MSK fse-test
- 4 kafka.m7g.xlarge brokers in 2 zones
- Apache Kafka version 3.7.x (KRaft mode)
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Cluster Resources - rcon-pvt (Fri Jan 17 14:41:06 UTC 2025)

Cluster Resources - rcon-pvt

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size	R/W Split Enabled
mod-remote-storage	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.3.3	2	4920	4472	0	3960	512	512	false
mod-finance-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.7.3	2	1024	896	0	700	88	128	false
mod-ncip	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.15.6	2	1024	896	0	768	88	128	false
mod-agreements	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.1.4	2	1592	1488	0	0	0	0	false
mod-ebsconet	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.3.1	2	1248	1024	0	700	128	256	false
mod-organizations	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:2.0.0	2	1024	896	0	700	88	128	false
mod-consortia	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.2.2	2	5136	4776	0	2048	512	1024	false
edge-sip2	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.3.1	2	1024	896	0	768	88	128	false
mod-serials-management	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.1.2	2	2480	2312	0	1792	384	512	false
mod-settings	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.1.0	2	1024	896	0	768	88	128	false
mod-data-import	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.2.4	1	2048	1844	0	1292	384	512	false
mod-search	18	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:4.0.7	2	2592	2480	0	1440	512	1024	false
edge-dematic	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.3.1	1	1024	896	0	768	88	128	false
mod-inn-reach	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	0	2880	512	1024	false
mod-record-specifications	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-record-specifications:1.0.2	2	1024	896	0	768	88	128	false
mod-tags	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.3.0	2	1024	896	0	768	88	128	false
mod-authtoken	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.16.1	2	1440	1152	0	922	88	128	false
edge-courses	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.5.1	2	1024	896	0	768	88	128	false
mod-notify	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.3.0	2	1024	896	0	768	88	128	false
mod-inventory-update	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:4.0.0	2	1024	896	0	768	88	128	false
mod-configuration	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.11.0	2	1024	896	0	768	88	128	false
mod-orders-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.8.3	2	1024	896	0	700	88	128	false
edge-caiasoft	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.3.2	2	1024	896	0	768	88	128	false
mod-login-saml	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.9.3	2	1024	896	0	768	88	128	false
mod-erm-usage-harvester	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:5.0.1	2	1024	896	0	768	88	128	false
mod-gobi	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.9.0	2	1024	896	0	700	88	128	false
mod-licenses	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.1.2	2	2480	2312	0	1792	384	512	false
mod-password-validator	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.3.0	2	1440	1298	0	768	384	512	false
edge-dcb	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.2.1	2	1024	896	0	768	88	128	false
mod-bulk-operations	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.1.8	2	3072	2600	0	1536	384	512	false
mod-fqm-manager	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:3.0.7	2	3000	2600	0	768	88	128	false
mod-graphql	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.13.1	2	1024	896	0	768	88	128	false
mod-finance	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:5.0.1	2	1024	896	0	700	88	128	false
mod-erm-usage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:5.0.0	2	2800	2550	0	1800	384	512	false
mod-batch-print	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.2.0	2	1024	896	0	768	88	128	false
mod-tlr	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128	false
mod-lists	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:3.0.5	2	6000	2600	0	768	88	128	false
mod-copycat	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.7.0	2	1024	512	0	768	88	128	false
mod-entities-links	11	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.1.3	2	2592	2480	0	1440	0	1024	false
mod-permissions	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.6.1	2	1684	1544	512	1024	384	512	false
pub-edge	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.14	2	1024	896	0	768	0	0	false
mod-orders	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.9.9	2	2048	1740	0	1024	384	512	false
edge-patron	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.2.1	2	1024	896	0	768	88	128	false
mod-marc-migrations	26	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-marc-migrations:1.0.0	2	1024	896	0	768	88	128	false
edge-ncip	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.10.1	2	1024	896	0	768	88	128	false
edge-inn-reach	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128	false
mod-users-bl	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.9.3	2	1440	1152	0	922	88	128	false
mod-oa	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128	false
mod-inventory-storage	12	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:28.0.4	2	4096	3690	0	3076	512	1024	false
mod-invoice	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.9.2	2	1440	1152	0	922	88	128	false
mod-user-import	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.9.0	2	1024	896	0	768	88	128	false
mod-sender	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.13.0	2	1024	896	0	768	88	128	false
edge-oai-pmh	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.10.0	2	1512	1360	0	1440	384	512	false
mod-data-export-worker	10	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.3.6	2	3072	2048	0	2048	384	512	false
mod-rtac	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.7.0	2	1024	896	0	768	88	128	false
mod-circulation-storage	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.3.3	2	2880	2592	0	1814	384	512	false
mod-source-record-storage	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.9.5	2	5600	5000	0	3500	384	512	false
mod-calendar	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.2.0	2	1024	896	0	768	88	128	false
mod-event-config	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.8.0	2	1024	896	0	768	88	128	false
mod-courses	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.11	2	1024	896	0	768	88	128	false
mod-circulation-item	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.1.0	2	1024	896	0	0	0	0	false
mod-inventory	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:21.0.5	2	2880	2592	0	1814	384	512	false
mod-email	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.18.1	2	2800	2550	0	1800	384	512	false
mod-requests-mediated	4	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128	false
mod-circulation	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.3.8	2	2880	2592	0	1814	384	512	false
mod-pubsub	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.15.3	2	1536	1440	0	922	384	512	false
mod-di-converter-storage	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.3.1	2	1024	896	0	768	88	128	false
edge-rtac	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.8.0	2	1024	896	0	768	88	128	false
edge-orders	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.1.0	2	1024	896	0	768	88	128	false
mod-users	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.4.5	2	1024	896	0	768	88	128	false
mod-template-engine	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.21.0	2	1024	896	0	768	88	128	false
mod-patron-blocks	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.11.1	2	1024	896	0	768	88	128	false
mod-audit	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.10.2	2	1024	896	0	768	88	128	false
edge-fqm	9	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:3.0.2	2	1024	896	0	768	88	128	false
mod-source-record-manager	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.9.5	2	5600	5000	0	3500	384	512	false
nginx-edge	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.14	2	1024	896	0	0	0	0	false
mod-quick-marc	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:6.0.0	1	2288	2176	0	1664	384	512	false
nginx-okapi	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14	2	1024	896	0	0	0	0	false
okapi-b	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:6.1.1	3	1684	1440	1024	922	384	512	false
mod-feesfines	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.2.1	2	1024	896	0	768	88	128	false
mod-invoice-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.9.1	2	1872	1536	0	1024	384	512	false
mod-reading-room	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-reading-room:1.0.0	2	1024	896	0	768	88	128	false
mod-dcb	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.2.4	2	1024	896	0	768	88	128	false
mod-service-interaction	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.1.1	2	2048	1844	0	1290	384	512	false
mod-patron	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.2.5	2	1024	896	0	768	88	128	false
mod-data-export	13	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.1.5	1	2048	1844	0	0	0	0	false
mod-oai-pmh	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.14.3	2	4096	3690	0	3076	384	512	false
edge-connexion	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.3.1	2	1024	896	0	768	88	128	false
mod-notes	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:6.0.0	2	1024	896	0	952	384	512	false
mod-kb-ebsco-java	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:5.0.0	2	1024	896	0	768	88	128	false
mod-organizations-storage	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.8.1	2	1024	896	0	700	88	128	false
mod-data-export-spring	8	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.4.3	1	2048	1844	0	1536	384	512	false
mod-login	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.12.1	2	1440	1298	0	768	384	512	false
pub-okapi	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14	2	1024	896	0	768	0	0	false
edge-erm	5	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-erm:1.3.0	2	1024	896	0	768	88	128	false
mod-eusage-reports	7	579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:3.0.0	2	1024	896	0	768	88	128	false

R - Ramsons release

Dataset Size:

Record type	Number of records
Instances	1 163 924
Holdings	1 348 036
Items	2 091 901

Methodology/Approach

Pre-generated files were used for DI Create job profile
- 10K, 25K, 50K, 100K and 500K files.
Run DI Create on a single tenant(cs00000int_0001) one by one with the delay with files using PTF - Create 2 profile.
Prepare files for DI Update with the Data export app, using previously imported items
Run DI Update on a single tenant(cs00000int_0001) one by one with the delay with prepared files using PTF - Update Success 2 profile

1K, 10K, 25K, 50K, 100K and 500K files.

Data-import durations were obtained from DB using SQL query

select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0001_mod_source_record_manager.job_execution order by started_date desc limit 1000;

Additional Screenshots of graphs or charts

[ Include additional screenshots of graphs on the Cloudwatch and Grafana dashboards for completeness sake. Include any raw data that includes the timestamps of tests or any screenshots/charts/graphs. These data may be separate files or may be one Miror board or one Sheet/Doc that has everything in it. Raw data is important to consult for additional insights if the report omits them initially. ]

Test Artifacts

Attach the test artifacts - excluding any sensitive data. These artifacts are deviations from the main files that were checked into Github, but are relevant for this test.

Data Import test report Ramsons [non-ECS]