Test status: PASSED

Overview

Regression testing of Check-In/Check-Out (CI/CO) fixed load tests on okapi based environment in Ramsons non-ECS release.
The purposes of CI/CO testing:
- To define response times of transaction controllers for Check-In and Check-Out
- To define the response times for requests that take longer than 100 milliseconds
- To find any trends for resource utilization and recommend improvements
- To check how system behaves over extended period during longevity test
- Compare results (current and previous

Summary

Common results:
- CI/CO tests showed stable response times across low loads (8, 20 vUsers), a 50% increase under high load (75 vUsers), moderate degradation during longevity test, no memory leaks and up to 14% (80 ms) degradation in CO and up to 20% (100 ms) in CI 20 vUsers flow compared to Quesnelia release.

Tests #1, #2, #3, #4

- Average response times in tests with 8, 20, 30 virtual users (vUsers) were the same in average. Average in 20 vUsers test CI - 482 ms, CO - 835 ms.
- Average response times in test with 75 vUsers response times grew +50% compared to 20 vUsers. CI - 606 ms, CO - 1100 ms.
Test #5

- Average response times in test #5 with 30 vUsers. CI - 519 ms, CO - 1130 ms.. There's expected degradation during 24 hours test if to compare with 30 vUsers test #3. CI - 37%, CO - 14%.
- No memory leaks during longevity test. Two tests perform to get the results and both tests began erroring after 19 hours of running. The root course is under investigation.
Comparison with Quesnelia results:
- CI/CO response times degradation (Tests #1, #2, #3, #4):
  - Test,# vUsers Check-Out Controller (CO) Check-In Controller (CI)
    1 8 10% 15%
    2 20 14% 20%
    3 30 7% 7%
    4 75 6% 4%
- CI/CO response times degraded (test #5 - longevity test):
  - 30 vUsers - 6% in CO and 14% in CI flow.

Test,#	vUsers	Check-Out Controller (CO)	Check-In Controller (CI)
1	8	10%	15%
2	20	14%	20%
3	30	7%	7%
4	75	6%	4%

Resources

CPU utilization
- Tests #1, #2, #3, #4 and longevity tests used CPU in correspondence with number of vUsers but there are some modules that spiked during 75 vUsers test - mod-users-b - 132%, mod-authtoken-b - 98%, nginx-okapi - 84%.

Memory consumption
- Tests #1, #2, #3, #4 and longevity tests didn't reveal some problems with memory usage by modules. So no memory leaks detected.

RDS CPU utilization average
- 8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 63% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test. The same CPU utilization as it was in quesnelia.

CPU (User) usage by broker
- As MSK cluster is linked to all PTF clusters so the time range which can reflect only CI/CO during longevity test (test #5) - from midnight till 7 a.m. Max consumption rate here - 10%. Also we may observe impact of other CI/CO tests - the max consumption rate - 40% for all clusters (tests #1, #2, #3, #4).

Recommendations & Jiras

The previous results report:
- Check-in-check-out Test Report (Quesnelia)
The current ticket: PERF-970 - [Ramsons] [non-ECS] CI/CO In Review

mod-serials-management-b affect DB connection growth 200 connection in average. Disabling this module do not affect response times or error rate but significantly decrease DB connection number.
Revisions for modules should have CPU=0 as a default value so this should be changed on module deployment level.

Test Runs

The following table contains tests configuration information

Test #	vUsers	Ramp-up, sec	Duration, sec
1	8	80	2700
2	20	200	2700
3	30	300	2700
4	75	750	2700
5	30	300	86400

Results

Errors:

Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. 422/Unprocessable Entity. Happen expectedly if instance was checked out already. Error rate - 0.002% which is acceptable.

Response time

The table contains results of Check-in, Check-out tests in Ramsons release.

Test #1, #2, #3, #4

Graph (Response times) Test #1

Graph (Response times) Test #2

Graph (Response times) Test #3

Graph (Response times) Test #4

	8 vUsers (test #1)			20 vUsers (test #2)			30 vUsers (test #3)			75 vUsers (test #4)
Requests	Response Times (ms)			Response Times (ms)			Response Times (ms)			Response Times (ms)
Label	#Samples	95th pct	Average	#Samples	95th pct	Average	#Samples	95th pct	Average	#Samples	95th pct	Average
Check-Out Controller	1888	1008.75	811	4496	1006.3	835.63	6588	954.55	822.01	14592	1422	1100.28
Check-In Controller	1322	568.85	470	3386	559	482.65	4973	506	455.46	10971	775	606.04
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	1888	386	289.98	4498	368	302.89	6589	335	288.99	14595	519	384.38
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	1322	279.7	227.94	3394	273	234.69	4982	240	209.28	10991	369	277.67
GET_circulation/loans (Submit_barcode_checkout)	1888	235.55	164.45	4496	216	162.68	6588	195	162.48	14592	319	218.14

Test #5

Graph (Response times) Test #5

30 vUsers Longevity test
Requests		Samples, Response Times (ms)
Label	#Samples	95th pct	Average
Check-Out Controller	1888	1008.75	1130
Check-In Controller	1322	568.85	519
POST_circulation/check-out-by-barcode (Submit_barcode_checkout)	1888	386	453.5
POST_circulation/check-in-by-barcode (Submit_barcode_checkin)	1322	279.7	331.15

Comparisons

This table has comparison between average values of response times of Ramsons and Quesnelia releases

	8 vUsers (test #1)				20 vUsers (test #2)				30 vUsers (test #3)				75 vUsers (test #4)
Requests	Response Times, milliseconds
	Quesnelia	Ramsons			Quesnelia	Ramsons			Quesnelia	Ramsons			Quesnelia	Ramsons
Label	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%	Average		Delta,ms	Difference,%
Check-Out Controller	741	811	70	9.45%	729	835.63	106.63	14.63%	767	822.01	55.01	7.17%	1039	1100.28	61.28	5.90%
Check-In Controller	408	470	62	15.20%	404	482.65	78.65	19.47%	427	455.46	28.46	6.67%	580	606.04	26.04	4.49%

Comparison of longevity test

30 vUsers Longevity (test #5)
Response Times, milliseconds
Quesnelia	Ramsons
Average	Average	Delta,ms	Difference,%
1065	1130	65	6.10%
454	519	65	14.32%

API requests where response times >= 100 milliseconds

API	30 vUsers Ramsons Average, ms
POST checkout-by-barcode	288
POST checkin-by-barcode	209
GET circulation/loans	162

API

30 vUsers Ramsons

Average, ms

POST checkout-by-barcode

288

POST checkin-by-barcode

209

GET circulation/loans

162

Resources Utilization

CPU Utilization

During 45 minute tests CPU utilized mostly during high load (75 vUsers) by okapi - 84%, mod-authtoken spiked every 3 minutes from 5 to 30%, mod-inventory-storage - 23%, mod-inventory - 17%, mod-pubsub - 17%, nginx-okapi - 10%, mod-circulation - 10%, mod-circulation-storage - 3%

During longevity CPU utilized mostly by okapi - 37%, mod-authtoken spiked every 3 minutes from 5 to 20%, mod-inventory - 12%, mod-pubsub - 11%, mod-circulation - 5%, mod-circulation-storage - 3%

CPU utilization by modules

CPU	#1 8 vUsers	CPU	#2 20 vUsers	CPU	#3 30 vUsers	CPU	#4 75 vUsers	CPU	#5 30 vUsers Longevity
mod-authtoken-b	16.93	okapi-b	24.66	okapi-b	36.73	okapi-b	85.34	okapi-b	39.51
okapi-b	10.48	mod-authtoken-b	22.45	mod-authtoken-b	22.57	mod-authtoken-b	31.54	mod-inventory-b	12.55
mod-inventory-b	9.52	mod-inventory-b	11.55	mod-inventory-b	12.17	mod-inventory-storage-b	24.14	mod-pubsub-b	11.67
mod-pubsub-b	8.6	mod-pubsub-b	10.78	mod-pubsub-b	11.74	mod-inventory-b	17.6	mod-authtoken-b	9.99
mod-inventory-storage-b	2.48	mod-inventory-storage-b	5.77	mod-inventory-storage-b	8.45	mod-pubsub-b	17.1	mod-inventory-storage-b	8.51
mod-entities-links-b	2.2	mod-circulation-b	4.11	mod-circulation-b	4.68	mod-circulation-b	10.52	mod-circulation-b	5.05
mod-circulation-b	1.49	mod-circulation-storage-b	2.58	mod-circulation-storage-b	3.29	nginx-okapi	10.42	mod-circulation-storage-b	3.43
mod-circulation-storage-b	1.37	mod-entities-links-b	2.22	mod-entities-links-b	2.77	mod-circulation-storage-b	6.63	mod-entities-links-b	2.17
mod-source-record-storage-b	1.32	mod-search-b	1.72	mod-search-b	2.06	mod-patron-blocks-b	2.53	mod-patron-blocks-b	1.7
mod-search-b	1.28	mod-source-record-storage-b	1.37	mod-patron-blocks-b	1.32	mod-entities-links-b	2.22	mod-search-b	1.4
mod-data-import-b	1.22	mod-data-import-b	1.35	mod-data-import-b	1.31	mod-search-b	1.71	mod-source-record-storage-b	1.31
edge-patron-b	1.1	mod-patron-blocks-b	1.23	mod-source-record-storage-b	1.3	mod-users-b	1.54	mod-data-import-b	1.29
mod-patron-blocks-b	0.57	edge-patron-b	1.14	edge-patron-b	1.13	mod-data-import-b	1.47	edge-patron-b	1.11
mod-source-record-manager-b	0.56	mod-source-record-manager-b	0.59	nginx-okapi	0.83	mod-source-record-storage-b	1.37	nginx-okapi	0.88
mod-users-bl-b	0.44	mod-users-b	0.55	mod-users-b	0.73	edge-patron-b	1.2	mod-users-b	0.78
mod-users-b	0.26	mod-users-bl-b	0.53	mod-source-record-manager-b	0.58	mod-configuration-b	0.92	mod-source-record-manager-b	0.58
nginx-okapi	0.23	nginx-okapi	0.51	mod-users-bl-b	0.47	mod-feesfines-b	0.63	mod-users-bl-b	0.48
mod-dcb-b	0.13	mod-configuration-b	0.28	mod-configuration-b	0.39	mod-source-record-manager-b	0.61	mod-configuration-b	0.4
mod-quick-marc-b	0.13	mod-feesfines-b	0.25	mod-feesfines-b	0.3	pub-okapi	0.53	mod-feesfines-b	0.35
mod-configuration-b	0.13	pub-okapi	0.14	pub-okapi	0.22	mod-users-bl-b	0.52	pub-okapi	0.23
mod-feesfines-b	0.11	mod-quick-marc-b	0.13	mod-quick-marc-b	0.13	mod-quick-marc-b	0.15	mod-quick-marc-b	0.14
pub-okapi	0.06	mod-dcb-b	0.12	mod-dcb-b	0.12	mod-dcb-b	0.13	mod-dcb-b	0.13
mod-oa-b	0.05	mod-oa-b	0.05	mod-oa-b	0.04	mod-oa-b	0.05	mod-oa-b	0.04
mod-di-converter-storage-b	0.03	mod-di-converter-storage-b	0.03	mod-di-converter-storage-b	0.04	mod-di-converter-storage-b	0.04	mod-di-converter-storage-b	0.04
mod-patron-b	0.03	mod-patron-b	0.03	mod-patron-b	0.03	mod-patron-b	0.03	mod-patron-b	0.03
pub-edge	0	pub-edge	0	pub-edge	0	pub-edge	0	pub-edge	0

Tests #1, #2, #3, #4

Test #5

During longevity CPU utilized mostly by okapi - 37%, mod-authtoken spiked every 3 minutes from 5 to 20%, mod-inventory - 12%, mod-pubsub - 11%, mod-circulation - 5%, mod-circulation-storage - 3%

Memory Consumption

Tests #1, #2, #3, #4 and test #5 didn't reveal some problems with memory usage by modules. So no memory leaks detected. Modules that consumed max memory - mod-search - 94%, mod-oa - 78%, mod-inventory - 72%, mod-dcb - 71%

Memory usage by modules

Memory	75 vUsers	Memory	30 vUsers Longevity
mod-oa-b	78.29	mod-oa-b	79.24
mod-inventory-b	78.2	mod-pubsub-b	68.44
mod-dcb-b	60.51	mod-dcb-b	62.05
mod-data-import-b	48.18	mod-inventory-b	59.98
okapi-b	46.92	mod-data-import-b	59.06
mod-pubsub-b	45.17	okapi-b	43.52
mod-users-b	40.16	mod-search-b	42.38
mod-search-b	38.06	mod-users-b	41.48
mod-feesfines-b	33.39	mod-feesfines-b	34.33
mod-configuration-b	32.08	mod-circulation-storage-b	31.93
mod-patron-blocks-b	30.69	mod-di-converter-storage-b	31.53
mod-circulation-storage-b	27.62	mod-patron-blocks-b	31.53
mod-quick-marc-b	27.34	mod-configuration-b	31.47
mod-di-converter-storage-b	26.27	mod-quick-marc-b	28.95
mod-entities-links-b	26.21	mod-users-bl-b	27.94
mod-inventory-storage-b	25.74	mod-entities-links-b	27.89
mod-circulation-b	24.34	mod-inventory-storage-b	26.71
mod-patron-b	24.29	mod-circulation-b	26.31
mod-source-record-storage-b	22.38	mod-patron-b	25.78
mod-authtoken-b	20.74	mod-source-record-storage-b	24.65
mod-users-bl-b	20.18	mod-authtoken-b	21.89
edge-patron-b	17.19	edge-patron-b	18.81
mod-source-record-manager-b	13.53	mod-source-record-manager-b	16.55
pub-okapi	4.97	pub-okapi	5.19
nginx-okapi	4.97	nginx-okapi	5.19
pub-edge	4.46	pub-edge	4.58

Tests #1, #2, #3, #4

Memory usage during 75 vUsers CI/CO test: mod-inventory - 78%, mod-oa - 78%, mod-dcb - 60%, mod-data-import - 48%, okapi - 46%, mod-pubsub - 45%, mod-users - 39%, mod-search - 38%. The memory trend doesn't reveal some problems.

Test #5

Memory consumption trends during longevity show steady growth for pubsub module which stopped after test completed with 68%.

RDS CPU Utilization

RDS CPU utilized:

8 vUsers - 12%, 20 vUsers - 20%, 30 vUsers - 25%, 75 vUsers - 56% During longevity test CPU grew from 25% to 40%. So it has growing trend during longevity test. The RDS CPU utilization is the same as in quesnelia release.

Tests #1, #2, #3, #4

Test #5

RDS Database Connections

For 45 minute and longevity tests RDS used max 885-920 connections. Without test it was 860 connections.

Tests #1, #2, #3, #4

Test #5

CPU (User) usage by broker

As MSK cluster is linked to all PTF clusters so the time range which can reflect only CI/CO during longevity test (test #5) - from midnight till 7 a.m. Max consumption rate here - 10%. Also we may observe impact of other CI/CO tests - the max consumption rate - 40% for all clusters (tests #1, #2, #3, #4).

Tests #1, #2, #3, #4

Test #5

Database load

Load by sqls (AAS) - SQL statements

Tests #1, #2, #3, #4
UPDATE fs09000000_mod_inventory_storage.item SET jsonb=$1 WHERE id=$2 RETURNING jsonb::text
INSERT INTO fs09000000_mod_pubsub.audit_message (id, event_id, event_type, tenant_id, audit_date, state, published_by, correlation_id, created_by, error_message) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
WITH deleted_rows AS ( delete from marc_indexers mi where exists( select ? from marc_records_tracking mrt where mrt.is_dirty = ? and mrt.marc_id = mi.marc_id and mrt.version > mi.version ) returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi where exists( select ? from records_lb where records_lb.id = mi.marc_id and records_lb.state = ? ) returning mi.marc_id) INSERT IN
SELECT fs09000000_mod_inventory_storage.count_estimate('SELECT * FROM fs09000000_mod_inventory_storage.material_type WHERE id=''025ba2c5-5e96-4667-a677-8186463aee69''')
UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
INSERT INTO fs09000000_mod_authtoken.refresh_tokens (id, user_id, is_revoked, expires_at) VALUES ($1, $2, $3, $4)
SELECT upsert('circulation_logs', $1::uuid, $2::jsonb)

Test #5
INSERT INTO fs09000000_mod_pubsub.audit_message (id, event_id, event_type, tenant_id, audit_date, state, published_by, correlation_id, created_by, error_message) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
SELECT fs09000000_mod_patron_blocks.count_estimate('SELECT jsonb FROM fs09000000_mod_patron_blocks.patron_block_limits WHERE (jsonb->>''patronGroupId'') = ''5fc96cbd-a860-42a7-8d2b-72af30206712''')
UPDATE fs09000000_mod_inventory_storage.item SET jsonb=$1 WHERE id=$2 RETURNING jsonb::text
SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>'userId') = '4cd01954-62da-46c5-8558-ebd222bc48eb'
SELECT fs09000000_mod_inventory_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_inventory_storage.service_point WHERE id=''7068e104-aa14-4f30-a8bf-71f71cc15e07''')
UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
INSERT INTO fs09000000_mod_authtoken.refresh_tokens (id, user_id, is_revoked, expires_at) VALUES ($1, $2, $3, $4)
SELECT upsert('circulation_logs', $1::uuid, $2::jsonb)
SELECT COUNT(*) FROM fs09000000_mod_users.users
SELECT fs09000000_mod_circulation_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_circulation_storage.loan_policy WHERE id=''2be97fb5-eb89-46b3-a8b4-776cea57a99e''')

During 45 minute tests (#1, #2, #3, #4) we see that the longest request is UPDATE fs09000000_mod_inventory_storage.item SET with 38 ms/request

During longevity test (#5) INSERT INTO fs09000000_mod_pubsub.audit_message - 41 ms and SELECT fs09000000_mod_inventory_storage.count_estimate - 107 ms.

Other observation is that we see a lot of UPDATE fs09000000_mod_login.auth_attempts and INSERT INTO fs09000000_mod_authtoken.refresh_tokens which is new. It may be connected to every 10 minutes token refresh.

Tests #1, #2, #3, #4

Test #5

Appendix

Infrastructure

PTF -environment rcp1
rcp1 9 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 1 instance of db.r6g.xlarge database instance: Writer instance MSK fse-tenant 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone) Apache Kafka version 3.7.x, metadata mode - KRaft EBS storage volume per broker 300 GiB auto.create.topics.enable=true log.retention.minutes=480 default.replication.factor=2 OpenSearch 2.13 ptf-test cluster r6g.2xlarge.search 4 data nodes r6g.large.search 3 dedicated master nodes

PTF -environment rcp1

rcp1 9 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 instance of db.r6g.xlarge database instance: Writer instance
MSK fse-tenant
- - 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
  - Apache Kafka version 3.7.x, metadata mode - KRaft
  - EBS storage volume per broker 300 GiB
  - auto.create.topics.enable=true
  - log.retention.minutes=480
  - default.replication.factor=2
OpenSearch 2.13 ptf-test cluster
- r6g.2xlarge.search 4 data nodes
- r6g.large.search 3 dedicated master nodes

DB table records size:

fs09000000
- instances - 27829992
- items - 28977562
- holding records - 27888818

Modules

All modules

Module	Task Definition Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft Limit	CPU Units	Xmx	Metaspace Size	Max Metaspace Size
mod-remote-storage	2	mod-remote-storage:3.3.0	2	4920	4472	1024	3960	512	512
mod-ncip	2	mod-ncip:1.15.4	2	1024	896	0	768	88	128
mod-finance-storage	2	mod-finance-storage:8.7.0	2	1024	896	1024	700	88	128
mod-agreements	2	mod-agreements:7.1.0	2	1592	1488	0	0	0	0
mod-ebsconet	2	mod-ebsconet:2.3.0	2	1248	1024	128	700	128	256
mod-organizations	2	mod-organizations:2.0.0	2	1024	896	0	700	88	128
edge-sip2	2	edge-sip2:3.3.0	2	1024	896	0	768	88	128
mod-settings	2	mod-settings:1.1.0	2	1024	896	200	768	88	128
mod-serials-management	2	mod-serials-management:1.1.0	0	2480	2312	0	1792	384	512
edge-dematic	2	edge-dematic:2.3.0	1	1024	896	0	768	88	128
mod-data-import	2	mod-data-import:3.2.1	1	2048	1844	256	1292	384	512
mod-search	11	mod-search:4.0.0	2	2592	2480	2048	1440	512	1024
mod-inn-reach	1	mod-inn-reach:3.2.1-SNAPSHOT.102	2	3600	3240	1024	2880	512	1024
mod-record-specifications	11	mod-record-specifications:1.0.0	2	1024	896	0	768	88	128
mod-tags	2	mod-tags:2.3.0	2	1024	896	0	768	88	128
mod-authtoken	2	mod-authtoken:2.16.1	2	1440	1152	512	922	88	128
edge-courses	2	edge-courses:1.5.0	2	1024	896	0	768	88	128
mod-notify	2	mod-notify:3.3.0	2	1024	896	0	768	88	128
mod-inventory-update	2	mod-inventory-update:4.0.0	2	1024	896	0	768	88	128
mod-configuration	2	mod-configuration:5.11.0	2	1024	896	0	768	88	128
mod-orders-storage	2	mod-orders-storage:13.8.0	2	1024	896	512	700	88	128
edge-caiasoft	2	edge-caiasoft:2.3.1	2	1024	896	0	768	88	128
mod-login-saml	2	mod-login-saml:2.9.1	2	1024	896	0	768	88	128
mod-erm-usage-harvester	2	mod-erm-usage-harvester:5.0.0	2	1024	896	0	768	88	128
mod-password-validator	2	mod-password-validator:3.3.0	2	1440	1298	0	768	384	512
mod-gobi	2	mod-gobi:2.9.0	2	1024	896	0	700	88	128
mod-licenses	2	mod-licenses:6.1.0	2	2480	2312	0	1792	384	512
edge-dcb	2	edge-dcb:1.2.0	2	1024	896	0	768	88	128
mod-bulk-operations	2	mod-bulk-operations:2.1.1	2	3072	2600	1024	1536	384	512
mod-fqm-manager	11	mod-fqm-manager:3.0.1	2	3000	2600	128	2048	384	512
mod-graphql	2	mod-graphql:1.12.2	2	1024	896	0	768	88	128
mod-finance	2	mod-finance:5.0.0	2	1024	896	0	700	88	128
mod-erm-usage	2	mod-erm-usage:5.0.0	2	1024	896	0	768	88	128
mod-batch-print	2	mod-batch-print:1.2.0	2	1024	896	0	768	88	128
mod-tlr	2	mod-tlr:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
mod-lists	6	mod-lists:3.0.1	2	6000	2600	128	768	88	128
mod-copycat	2	mod-copycat:1.7.0	2	1024	512	0	768	88	128
mod-entities-links	6	mod-entities-links:3.1.0	2	2592	2480	400	1440	0	1024
mod-permissions	4	mod-permissions:6.6.0	2	1684	1544	512	1024	384	512
pub-edge	2	pub-edge:2023.06.14	2	1024	896	0	768	0	0
mod-orders	2	mod-orders:12.9.1	2	2048	1440	1024	1024	384	512
edge-patron	2	edge-patron:5.2.0	2	1024	896	256	768	88	128
edge-ncip	2	edge-ncip:1.10.1	2	1024	896	0	768	88	128
mod-marc-migrations	26	mod-marc-migrations:1.0.0-SNAPSHOT.8	2	1024	896	0	768	88	128
edge-inn-reach	1	edge-inn-reach:3.3.0-SNAPSHOT.69	2	1024	896	0	768	88	128
mod-users-bl	2	mod-users-bl:7.9.2	2	1440	1152	512	922	88	128
mod-oa	1	mod-oa:2.1.0-SNAPSHOT.66	2	1024	896	0	768	88	128
mod-invoice	2	mod-invoice:5.9.0	2	1440	1152	512	922	88	128
mod-inventory-storage	5	mod-inventory-storage:28.0.1	2	4096	3690	2048	3076	512	1024
mod-user-import	2	mod-user-import:3.9.0	2	1024	896	0	768	88	128
mod-sender	2	mod-sender:1.13.0	2	1024	896	0	768	88	128
edge-oai-pmh	2	edge-oai-pmh:2.10.0	2	1512	1360	1024	1440	384	512
mod-data-export-worker	2	mod-data-export-worker:3.3.1	2	3072	2048	1024	2048	384	512
mod-rtac	2	mod-rtac:3.7.0	2	1024	896	0	768	88	128
mod-circulation-storage	2	mod-circulation-storage:17.3.0	2	2880	2592	1536	1814	384	512
mod-source-record-storage	4	mod-source-record-storage:5.9.0	2	5600	5000	2048	3500	384	512
mod-calendar	3	mod-calendar:3.2.0	2	1024	896	0	768	88	128
mod-event-config	2	mod-event-config:2.8.0	2	1024	896	0	768	88	128
mod-courses	2	mod-courses:1.4.11	2	1024	896	0	768	88	128
mod-circulation-item	2	mod-circulation-item:1.1.0	2	1024	896	0	0	0	0
mod-inventory	2	mod-inventory:21.0.0	2	2880	2592	1024	1814	384	512
mod-email	2	mod-email:1.18.0	2	1024	896	0	768	88	128
mod-requests-mediated	10	mod-requests-mediated:1.0.0-SNAPSHOT.4	2	1024	896	0	768	88	128
mod-di-converter-storage	2	mod-di-converter-storage:2.3.0	2	1024	896	0	768	88	128
mod-pubsub	2	mod-pubsub:2.15.0	2	1536	1440	1024	922	384	512
mod-circulation	2	mod-circulation:24.3.0	2	2880	2592	1536	1814	384	512
edge-orders	2	edge-orders:3.1.0	2	1024	896	0	768	88	128
edge-rtac	2	edge-rtac:2.8.0	2	1024	896	0	768	88	128
mod-template-engine	2	mod-template-engine:1.21.0	2	1024	896	0	768	88	128
mod-users	2	mod-users:19.4.0	2	1024	896	0	768	88	128
mod-patron-blocks	2	mod-patron-blocks:1.11.0	2	1024	896	1024	768	88	128
mod-audit	2	mod-audit:2.10.0	2	1024	896	0	768	88	128
edge-fqm	2	edge-fqm:3.0.0	2	1024	896	0	768	88	128
mod-source-record-manager	2	mod-source-record-manager:3.9.0	2	5600	5000	2048	3500	384	512
nginx-edge	2	nginx-edge:2023.06.14	2	1024	896	0	0	0	0
mod-quick-marc	2	mod-quick-marc:6.0.0	1	2288	2176	0	1664	384	512
nginx-okapi	2	nginx-okapi:2023.06.14	2	1024	896	0	0	0	0
okapi-b	2	okapi:5.3.0	3	1684	1440	1024	922	384	512
mod-feesfines	2	mod-feesfines:19.2.0	2	1024	896	0	768	88	128
mod-invoice-storage	2	mod-invoice-storage:5.9.0	2	1872	1536	1024	1024	384	512
mod-reading-room	3	mod-reading-room:1.0.0	2	1024	896	0	768	88	128
mod-service-interaction	2	mod-service-interaction:4.1.0	2	2048	1844	256	1290	384	512
mod-dcb	2	mod-dcb:1.2.0	2	1024	896	0	768	88	128
mod-data-export	2	mod-data-export:5.1.0	1	2048	1844	2048	0	0	0
mod-patron	2	mod-patron:6.2.0	2	1024	896	0	768	88	128
mod-oai-pmh	2	mod-oai-pmh:3.14.1	2	4096	3690	2048	3076	384	512
edge-connexion	2	edge-connexion:1.3.1	2	1024	896	0	768	88	128
mod-notes	2	mod-notes:6.0.0	2	1024	896	0	952	384	512
mod-kb-ebsco-java	2	mod-kb-ebsco-java:5.0.0	2	1024	896	0	768	88	128
mod-login	2	mod-login:7.12.1	2	1440	1298	1024	768	384	512
mod-organizations-storage	2	mod-organizations-storage:4.8.1	2	1024	896	0	700	88	128
mod-data-export-spring	2	mod-data-export-spring:3.4.0	1	2048	1844	256	1536	384	512
pub-okapi	2	pub-okapi:2023.06.14	2	1024	896	0	768	0	0
edge-erm	1	edge-erm:1.3.0	2	1024	896	0	768	88	128
mod-eusage-reports	2	mod-eusage-reports:3.0.0	2	1024	896	0	768	88	128

Methodology/Approach

Description

Testing includes data preparation step and testing itself

Data preparation for each test takes up to 20 minutes and consists of truncating involved in testing tables, populating data and updating statuses of items.
Test itself depends on duration and virtual users number creating necessary load.

In Ramsons token expiration set to 10 minutes by default so to run any tests use new login implementation from the script. Pay attention to Backend Listener. Replace value of application parameter to make the results visible in Grafana dashboard.

Module configuration recommended setup

Update revision in source-record-storage module to exclude every 30 minutes SQL statements - delete rows in marc_indexers (mi) WITH deleted_rows

{
"name": "srs.marcIndexers.delete.interval.seconds",
"value": "86400"
},

Update mod-serials module. Set number of task with 0 to exclude significant database connection growth.

DB trigger setup in Ramsons

Usual PTF CI/CO data preparation script won’t work in Ramsons. To solve that disable trigger updatecompleteupdateddate_item_insert_update before data preparation for the tenant and enable it before test start.

The sql file was updated to do that step from the script.

Data preparation

First step

To prepare data establish connection by AWS keys, then run from bash .sql script first (take from the title of code block and replace [PASSWORD] with correct password.

export PGPASSWORD='[PASSWORD]';psql -f checkin-checkout-db-restore.sql -a --echo-all -h 127.0.0.1 -U folio

-- Disable trigger
ALTER TABLE fs09000000_mod_inventory_storage.item
DISABLE TRIGGER updatecompleteupdateddate_item_insert_update;

TRUNCATE TABLE fs09000000_mod_patron_blocks.user_summary;
TRUNCATE TABLE fs09000000_mod_circulation_storage.loan;
TRUNCATE TABLE fs09000000_mod_circulation_storage.audit_loan;
TRUNCATE TABLE fs09000000_mod_circulation_storage.request;
TRUNCATE TABLE fs09000000_mod_circulation_storage.patron_action_session;
TRUNCATE TABLE fs09000000_mod_circulation_storage.scheduled_notice;
TRUNCATE TABLE fs09000000_mod_notify.notify_data;

UPDATE fs09000000_mod_inventory_storage.item 
SET jsonb = jsonb_set(jsonb, '{status, name}', '"Available"')
WHERE jsonb->'status'->>'name' != 'Available';

UPDATE fs09000000_mod_users.users 
SET jsonb = jsonb_set(jsonb, '{active}', '"true"') 
WHERE jsonb->'active' != 'true';

-- Enable trigger
ALTER TABLE fs09000000_mod_inventory_storage.item
ENABLE TRIGGER updatecompleteupdateddate_item_insert_update;

Second step

Run command from scripts folder uploaded to S3 bucket ./circ-data-load.sh psql_rcp1.conf [tenant] - where replace [tenant] with tenant Id, change parameters in psql_rcp1.conf file with valid data.
Troubleshooting:
- If the command executed from local machine you may encounter with too long query error message. To solve it use PGAdmin to run 2 long queries UPDATE ${TENANT}_mod_inventory_storage.item SET jsonb = jsonb_set(jsonb, '{status, name}', '\"Checked out\"') where id IN.
- Other possible issue - incorrect encoding (on Windows machine). To solve it just add ENCODING 'UTF8'
  - Use pattern: copy ${TENANT}_mod_circulation_storage.loan(id, jsonb) FROM '${LOANS}' DELIMITER E'\t' ENCODING 'UTF8'

Use .jmx file script for Ramson release. If any changes were made then upload the artefacts to S3 bucket and AWS instance load generator.

To start test from AWS instance (load generator) use template for the command. Test locally before start.

8 vUsers - 
nohup jmeter -n -t /home/ptf/testdata/RCP1/CICO/circulation_checkInCheckOut_rcp1.jmx -l rcp1_8vUsers.jtl -e -o /home/ptf/testdata/RCP1/CICO/results/8vUsers -JGlobal_duration=2700 -JCICO_vusers=8 -JCICO_rampup=80

20vUsers - 
nohup jmeter -n -t /home/ptf/testdata/RCP1/CICO/circulation_checkInCheckOut_rcp1.jmx -l rcp1_20vUsers.jtl -e -o /home/ptf/testdata/RCP1/CICO/results/20vUsers -JGlobal_duration=2700 -JCICO_vusers=20 -JCICO_rampup=200

30vUsers - 
nohup jmeter -n -t /home/ptf/testdata/RCP1/CICO/circulation_checkInCheckOut_rcp1.jmx -l rcp1_30vUsers.jtl -e -o /home/ptf/testdata/RCP1/CICO/results/30vUsers -JGlobal_duration=2700 -JCICO_vusers=30 -JCICO_rampup=300


75vUsers - 
nohup jmeter -n -t /home/ptf/testdata/RCP1/CICO/circulation_checkInCheckOut_rcp1.jmx -l rcp1_75vUsers.jtl -e -o /home/ptf/testdata/RCP1/CICO/results/75vUsers -JGlobal_duration=2700 -JCICO_vusers=75 -JCICO_rampup=750

Test CI/CO with 8, 20, 30, 75 concurrent users for 45 minutes each.

Test CI/CO with 30 users for 24 hours to detect any trends in memory.

To create widgets in AWS dashboard to monitor and collect CI/CO related modules parameters (service CPU and Memory) use these json:

CPU and Memory widget jsons

JSON widgets to monitor service CPU consumption

{
"metrics": [
[ "AWS/ECS", "CPUUtilization", "ServiceName", "pub-edge", "ClusterName", "rcp1-pvt", { "region": "us-east-1" } ],
[ "...", "mod-eusage-reports-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-pubsub-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-data-export-spring-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-data-import-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-rtac-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-users-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-organizations-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-graphql-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-event-config-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-audit-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-organizations-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-tags-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-ncip-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-feesfines-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-user-import-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "pub-okapi", ".", ".", { "region": "us-east-1" } ],
[ "...", "okapi-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-entities-links-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-sender-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "nginx-okapi", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-circulation-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-kb-ebsco-java-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-dcb-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-dematic-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-oai-pmh-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inventory-update-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-search-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-orders-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-orders-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-invoice-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-circulation-item-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-batch-print-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-sip2-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inventory-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-licenses-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "nginx-edge", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-invoice-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-login-saml-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-agreements-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-di-converter-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-erm-usage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-copycat-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-courses-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-oa-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-email-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-serials-management-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-patron-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-permissions-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-gobi-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-configuration-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-calendar-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-patron-blocks-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-fqm-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-service-interaction-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-circulation-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-caiasoft-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-finance-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-authtoken-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-connexion-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-ncip-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-notes-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-bulk-operations-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-oai-pmh-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-erm-usage-harvester-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-settings-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-notify-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-source-record-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-quick-marc-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-login-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inventory-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-rtac-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inn-reach-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-lists-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-finance-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-password-validator-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-orders-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-fqm-manager-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-dcb-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-source-record-manager-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-users-bl-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-template-engine-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-courses-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-patron-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-ebsconet-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-data-export-worker-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-remote-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-inn-reach-b", ".", ".", { "region": "us-east-1", "visible": false } ]
],
"view": "timeSeries",
"period": 60,
"yAxis": {
"left": {
"min": 0
}
},
"stacked": false,
"title": "CI/CO Service CPU Utilization",
"region": "us-east-1",
"stat": "Average"
}

JSON widgets to monitor service Memory consumption

{
"metrics": [
[ "AWS/ECS", "MemoryUtilization", "ServiceName", "pub-edge", "ClusterName", "rcp1-pvt", { "region": "us-east-1" } ],
[ "...", "mod-eusage-reports-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-pubsub-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-data-export-spring-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-data-import-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-rtac-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-users-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-organizations-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-graphql-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-event-config-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-audit-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-organizations-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-tags-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-ncip-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-feesfines-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-user-import-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "pub-okapi", ".", ".", { "region": "us-east-1" } ],
[ "...", "okapi-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-entities-links-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-sender-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "nginx-okapi", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-circulation-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-kb-ebsco-java-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-dcb-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-dematic-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-oai-pmh-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inventory-update-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-search-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-orders-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-orders-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-invoice-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-circulation-item-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-batch-print-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-sip2-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inventory-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-licenses-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "nginx-edge", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-invoice-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-login-saml-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-agreements-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-di-converter-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-erm-usage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-copycat-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-courses-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-oa-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-email-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-serials-management-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-patron-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-permissions-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-gobi-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-configuration-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-calendar-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-patron-blocks-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-fqm-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-service-interaction-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-circulation-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-caiasoft-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-finance-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-authtoken-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "edge-connexion-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-ncip-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-notes-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-bulk-operations-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-oai-pmh-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-erm-usage-harvester-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-settings-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-notify-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-source-record-storage-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-quick-marc-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-login-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inventory-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-rtac-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-inn-reach-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-lists-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-finance-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-password-validator-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-orders-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-fqm-manager-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-dcb-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-source-record-manager-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-users-bl-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-template-engine-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-courses-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-patron-b", ".", ".", { "region": "us-east-1" } ],
[ "...", "mod-ebsconet-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-data-export-worker-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "mod-remote-storage-b", ".", ".", { "region": "us-east-1", "visible": false } ],
[ "...", "edge-inn-reach-b", ".", ".", { "region": "us-east-1", "visible": false } ]
],
"view": "timeSeries",
"period": 60,
"yAxis": {
"left": {
"min": 0
}
},
"stacked": false,
"title": "CI/CO Service Memory Usage",
"region": "us-east-1",
"stat": "Average"
}

File with raw data
Use the file to get raw data and comparison tables

Browser not supported