[Quesnelia] [non-ECS] [Data import] Create MARC authority Records

Overview

  • This document contains the results of testing Data Import for MARC Bibliographic records with an update job in the Quesnelia release on qcp1 environments with Kafka consolidated topics and file splitting features enabled on a non-ecs environment.

PERF-832 - Getting issue details... STATUS  

Summary

  • Data import tests finished successfully, only Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files. Duration of DI grew in correspondence with the number of records in files.
  • Check-in and Check-out with 5 virtual users was performed during DI Create new MARC authority records jobs for non-matches No issues.
  • Data Import in Quesnelia without CICO perform faster than with it.
  • Comparing Poppy and Quesnelia releases
    • Check-in / Check-out perform better in Quesnelia.  Response time improved during Create jobs for long period of work time on 15% in Average.
    • DI durations improved  - 11%-14% in Average.
  • During testing, we noticed spikes in the mod permissions module. To mitigate this issue and prevent system slowdowns, we adjusted the order of loading files, starting with Tenant 3 (qcp1-02), followed by Tenant 2 (qcp1-01), and finally Tenant 1 (qcp1-00).

Test Results and Comparison

Test №1

Test with 1k, 10k, 25k and 50k records files DI started on one tenant only(qcp1-00), and comparative results between Poppy and Quesnelia.

# of records 

% creates

File

DI duration 
Morning Glory

DI duration
Nolana

DI duration 
Orchid

DI duration 
Poppy

DI duration 
Quesnelia
1,0001001k_marc_authority.mrc?api=v2 24 s27 s41 sec29 sec22 sec
-24%
5,000100 LC_SUBJ_msplit00000000.mrc?api=v21 min 21 s1 min 15 s1min 21s1 min 38 sec1 min 19 sec
-19%
10,000100msplit00000000.mrc?api=v2 2 min 32 s2 min 31 s2min 53s2 min 53 sec2 min 36 sec
-9.8%
22778
(for Poppy test)
25000
(for Quesnelia test)
100 msplit00000013.mrc?api=v211 min 14 s7 min 7 s5 min 42s6 min 24 sec6 min 19 sec
-1.3%
50,00010050000_authorityrecords.mrc?api=v222 min11 min 24 s11 min 11s13 min 48 sec11 min 59 sec
-13%

Test №2

Test with CICO 5 concurrent users and DI 1K, 5K, 10K, 25K and 50K started on one tenant only.

  • Сomparative Baseline Check-In\Check-Out results without Data Import between Poppy and Quesnelia.

CICO, Median time without
DI
(Poppy)

CICO, 95% time without
DI
(Poppy)
CICO, Median time without
DI
(Quesnelia)
CICO, 95% time without
DI
(Quesnelia)
CICO, Avg time without
DI
(Quesnelia)
Check-In516 ms567 ms503 ms
-2.5%

593 ms
+4.5%

511 ms

Check-Out910 ms2094 ms 836 ms
-8%
1117 ms
-46%
876 ms
  • Сomparative  Check-In\Check-Out results between Baseline (Quesnelia) and  Check-In\Check-Out plus Data Import (Quesnelia.)
# of records
(Quesnelia)

DI Duration with CICO
(Quesnelia)

CI time Avg
(Quesnelia)
CI time 95th pct
(Quesnelia)
CO time Avg
(Quesnelia)
CO time 95th pct
(Quesnelia)
Baseline CI
Avg delta
Baseline CI 
95th pct delta
Baseline CO
Avg delta
Baseline CO 
95th pct delta
1,000

20 sec

0.5600.7541.1641.313+9%+27%+32%+17%
5,0001 min 19 sec0.7011.1711.1411.790+37%+97%+30%+60%
10,0002 min 35 se0.7231.0241.1791.494+41%+72%+34%+34%
25,0006 min 26 sec0.7221.0241.1801.494+41%+72%+35%+34%
50,000

12 min 16 sec

0.7771.045

1.265

1.550+52%+76%+44%+39%
  • Сomparative Data Import and Check-In\Check-Out results between Poppy and Quesnelia.

# of records 
(Poppy)

DI Duration with CICO
(Poppy)

CI time Avg
(Poppy)

CI time 95th pct
(Poppy)

CO time Avg
(Poppy)

CO time 95th pct
(Poppy)

# of records
(Quesnelia)

DI Duration with CICO
(Quesnelia)

CI time Avg
(Quesnelia)
CI time 95th pct
(Quesnelia)
CO time Avg
(Quesnelia)
CO time 95th pct
(Quesnelia)
1,00035 sec0.5250.5761.0781.3261,000

20 sec
-42.8%

0.560
+6%
0.754
+30%
1.164
+8%
1.313
-1%
5,0001 min 41 sec0.5130.6120.91.0195,0001 min 19 sec
-21.7%
0.701
+36%
1.171
+91%
1.141
+26%
1.790
+75%
10,0003 min 4 sec0.5810.6851.0161.32110,0002 min 35 sec
-15.7%
0.723
+24%
1.024
+49%
1.179
+16%
1.494
+13%
22,7786 min 32 sec0.5981.5421.2441.72925,0006 min 26 sec
-1.5%
0.722
+20%
1.024
-33%
1.180
-5%
1.494
-13%
50,00013 min 48 sec0.6711.9531.512.0950,000

12 min 16 sec
-11%

0.777
+15%
1.045
-46%

1.265
-16%

1.550
-25%

Test №3

Multitenant testing

File with 50K, 25K, 10K, 5K, 1K loaded sequences and launched on Tenant1, Tenant2, and Tenant3 for Poppy 

File with 50K, 25K, 10K, 5K, 1K loaded sequences and launched on Tenant3, Tenant2 and Tenant1 for Quesnelia

Num of recordsTenant 1
pcp1-00
duration
(Poppy)
Tenant 2
pcp1-01
duration
(Poppy)
Tenant 3
pcp1-02
duration
(Poppy)
Num of recordsTenant 3
qcp1-02
duration
(Quesnelia)
Tenant 2
qcp1-01
duration
(Quesnelia)
Tenant 1
qcp1-00
duration
(Quesnelia)
100066 min75 min74 min100015 min 15 sec15 min 46 sec16 min
500067 min76 min75 min500022 min 9 sec22 min 10 sec21 min 33 sec
10,00067 min76 min74 min10,00027 min 38 sec27 min 51 sec29 min 42 sec
22,77867 min*73 min*71 min*2500046 min 42 sec47 min47 min 14 sec
50,00062 min56 min54 min50,00067 min 29 sec67 min 27 sec67 min 22 sec

Test №4

Order for load files with pause between files: 50k, 25k, 10k, 5k, and 1k  for order tenants Tenant 3(qcp1-02), Tenant 1(qcp1-00) and Tenant 2(qcp1-01)

Job ProfileNum of recordsTenant 1
qcp1-00
duration
Tenant 2
qcp1-01
duration
Tenant 3
qcp1-02
duration
KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q100000:2800:3200:18
500003:3803:3103:29
10,00006:5807:0206:47
2500017:5417:3517:53
50,00034:0935:0829:49

Test №5

Order for load files without pause between files: 1k, 5k, 10k, 25k and 50k for order tenants : Tenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)


Job ProfileNum of recordsTenant 1
qcp1-00
duration
Tenant 2
qcp1-01
duration
Tenant 3
qcp1-02
duration
KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q




10000:09:370:02:430:00:53
50000:10:540:28:430:02:17
10,0000:19:130:31:450:05:53
250000:35:540:44:130:22:44
50,0001:03:551:10:160:52:38


Resource utilization for Test #1

 Resource utilization table
CPURAM
mod-inventory-b49.90%mod-data-import-b64.80%
mod-source-record-storage-b44.70%mod-permissions-b63.50%
nginx-okapi38.10%mod-source-record-manager-b49.90%
mod-di-converter-storage-b28.40%mod-inventory-b41.80%
mod-source-record-manager-b24.80%mod-di-converter-storage-b41.30%
okapi-b16.40%okapi-b36.20%
mod-data-import-b16.10%mod-source-record-storage-b27.70%
mod-permissions-b5.60%mod-inventory-storage-b16.20%
mod-inventory-storage-b0.50%nginx-okapi4.90%
pub-okapi0.20%pub-okapi4.70%

Service CPU Utilization

Here we can see that mod-inventory-b module used 50% CPU and mod-source-record-storage-b 46% CPU 

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU in the average was 90%.

DB Connections

DB connections was 1151.

DB load

                                                                                                                     

Top SQL-queries


#TOP 5 SQL statements
1
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
2
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
3
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
4
WITH input_rows(record_id, authority_id) AS (
   VALUES ($1::uuid,$2::uuid)
)
, ins AS (
   INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id)
   SELECT * FROM input_rows
   ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id
   RETURNING record_id::uuid, authority_id::uuid
   )
SELECT record_id, authority_id
FROM   ins
UNION  ALL
SELECT c.record_id, c.authority_id 
FROM   input_rows
JOIN   fs09000000_mod_inventory.records_authorities c USING (record_id);
5
UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *


Resource utilization for Test #2

 Resource utilization table
CPURAM
nginx-okapi49%mod-inventory-b75%
mod-inventory-b47%mod-permissions-b70%
mod-source-record-storage-b42%mod-data-import-b61%
mod-di-converter-storage-b29%mod-source-record-manager-b48%
mod-source-record-manager-b26%okapi-b36%
okapi-b22%mod-inventory-storage-b35%
mod-permissions-b7%mod-di-converter-storage-b34%
pub-okapi2%mod-source-record-storage-b28%
mod-inventory-storage-b2%nginx-okapi5%
mod-data-import-b2%pub-okapi4%

Service CPU Utilization

Here we can see that mod-inventory-b module and nginx-okapi used 50% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 93%.

DB Connections

DB connections was 1580.

DB load

                                                                                                                     

Top SQL-queries

#TOP 5 SQL statements
1
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
2
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
3
UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *
4
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
5
WITH input_rows(record_id, authority_id) AS (
   VALUES ($1::uuid,$2::uuid)
)
, ins AS (
   INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id)
   SELECT * FROM input_rows
   ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id
   RETURNING record_id::uuid, authority_id::uuid
   )
SELECT record_id, authority_id
FROM   ins
UNION  ALL
SELECT c.record_id, c.authority_id 
FROM   input_rows
JOIN   fs09000000_mod_inventory.records_authorities c USING (record_id);

Resource utilization for Test #3

 Resource utilization table
CPURAM
mod-permissions-b146%mod-permissions-b79%
mod-inventory-b56%mod-inventory-b72%
nginx-okapi42%mod-data-import-b61%
mod-source-record-storage-b41%mod-source-record-manager-b44%
mod-di-converter-storage-b37%okapi-b37%
mod-source-record-manager-b27%mod-source-record-storage-b36%
okapi-b20%mod-inventory-storage-b36%
mod-data-import-b2%mod-di-converter-storage-b35%
mod-inventory-storage-b1%nginx-okapi5%
pub-okapi1%pub-okapi4%

Service CPU Utilization

Here we can see that mod-inventory-b module used 57% CPU, mod-inventory-b used 43% CPU and mod-permissions-b spiked up to 150% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 95%.

DB Connections

DB connections was 1823.

DB load

                                                                                                                     

Top SQL-queries

#TOP 5 SQL statements
1
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
2
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
3
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
4
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
5
INSERT INTO fs07000001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)


Resource utilization for Test #4

 Resource utilization table
CPURAM
mod-inventory-b91.30%mod-permissions-b80.00%
nginx-okapi43.60%mod-data-import-b65.10%
mod-source-record-storage-b39.90%mod-inventory-b60.40%
mod-di-converter-storage-b26.60%mod-source-record-storage-b42.20%
okapi-b25.00%mod-di-converter-storage-b42.00%
mod-source-record-manager-b18.90%mod-source-record-manager-b41.90%
mod-data-import-b2.40%okapi-b37.60%
mod-permissions-b1.60%mod-inventory-storage-b16.30%
mod-inventory-storage-b0.30%nginx-okapi5.60%
pub-okapi0.20%pub-okapi4.80%

Service CPU Utilization

Here we can see that mod-inventory-b module used 91% CPU, nginx-okapiused 44% CPU and mod-permissions-b spiked up to used 170% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 90%.

DB Connections

DB connections was 1450.

DB load

                                                                                                                     

Top SQL-queries

#TOP 5 SQL statements
1
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
2
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
3
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
4
INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
5
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)

Resource utilization for Test #5

 Resource utilization table
CPURAM
mod-permissions-b77%mod-permissions-b77%
mod-inventory-b63%mod-inventory-b75%
mod-di-converter-storage-b35%mod-data-import-b59%
nginx-okapi35%mod-source-record-manager-b53%
mod-source-record-storage-b35%mod-source-record-storage-b44%
mod-source-record-manager-b23%mod-di-converter-storage-b34%
mod-data-import-b20%mod-inventory-storage-b34%
okapi-b19%okapi-b33%
mod-inventory-storage-b1%nginx-okapi4%
pub-okapi1%pub-okapi4%

Service CPU Utilization

Here we can see that mod-inventory-b module used 63% CPU in avarage, and mod-permissions-b spiked up to 130% CPU

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 90%.

DB Connections

DB connections was 1757.

DB load

                                                                                                                     

Top SQL-queries

#TOP 5 SQL statements
1
INSERT INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
2
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
3
INSERT INTO fs07000002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
4
INSERT INTO fs07000001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
5
INSERT INTO fs07000002_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)


Appendix

Infrastructure

PTF - environment Quesnelia (qcp1)

  • 10 db.r6g.xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instances, writer

    NameMemory GIBvCPUs

    db.r6g.xlarge

    32 GiBvCPUs
  • MSK ptf-mobius-testing2
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


 Quesnelia modules memory and CPU parameters
#277
Apr 26, 2024, 10:55 AM UTC










ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSizeR/W split enabled
qcp1-pvt









Fri Apr 26 10:56:03 UTC 2024









mod-remote-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.024920447210243960512512FALSE
mod-ncip3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.42102489612876888128FALSE
mod-finance-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.021024896102470088128FALSE
mod-agreements3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0215921488128000FALSE
mod-ebsconet3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.2.0212481024128700128256FALSE
mod-organizations3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.9.02102489612870088128FALSE
mod-consortia2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.1.023072204812820485121024FALSE
edge-sip22579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-sip2:3.2.0-SNAPSHOT.2092102489612876888128FALSE
mod-serials-management3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.0.02248023121281792384512FALSE
mod-settings3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.32102489620076888128FALSE
mod-data-import6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.1.01204818442561292384512FALSE
edge-dematic3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.2.01102489612876888128FALSE
mod-search3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.2.0225922480204814405121024FALSE
mod-inn-reach2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inn-reach:3.2.0-SNAPSHOT.86236003240102428805121024FALSE
mod-tags3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.2.02102489612876888128FALSE
edge-courses3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.4.02102489612876888128FALSE
mod-authtoken4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.15.121440115251292288128FALSE
mod-inventory-update3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.3.02102489612876888128FALSE
mod-notify3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.2.02102489612876888128FALSE
mod-configuration3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.10.02102489612876888128FALSE
mod-orders-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.7.02102489651270088128FALSE
edge-caiasoft3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.2.02102489612876888128FALSE
mod-login-saml3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.8.02102489612876888128FALSE
mod-erm-usage-harvester3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.5.02102489612876888128FALSE
mod-licenses3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.0.02248023121281792384512FALSE
mod-gobi3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.8.02102489612870088128FALSE
mod-password-validator3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.2.0214401298128768384512FALSE
mod-bulk-operations3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.0.023072260010241536384512FALSE
mod-fqm-manager3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:2.0.12300026001282048384512TRUE
edge-dcb3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.1.02102489612876888128FALSE
mod-graphql4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.12102489612876888128FALSE
mod-finance3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.9.02102489612870088128FALSE
mod-erm-usage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.7.02102489612876888128FALSE
mod-batch-print4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.1.02102489612876888128FALSE
mod-copycat3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.6.02102451212876888128FALSE
mod-lists3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:2.0.02300026001282048384512FALSE
mod-entities-links4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.0.0225922480400144001024FALSE
mod-permissions6579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.5.02168415445121024384512FALSE
pub-edge3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.142102489612876800FALSE
mod-orders3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.8.022048144010241024384512FALSE
edge-patron3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.1.02102489625676888128FALSE
edge-ncip3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.22102489612876888128FALSE
edge-inn-reach2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-inn-reach:3.1.1-SNAPSHOT.452102489612876888128FALSE
mod-users-bl3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.7.021440115251292288128FALSE
mod-oa2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oa:2.1.0-SNAPSHOT.622102489612876888128FALSE
mod-inventory-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.1.024096369020483076384512FALSE
mod-invoice4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.8.021440115251292288128FALSE
mod-user-import3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.02102489612876888128FALSE
mod-sender4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.12.02102489612876888128FALSE
edge-oai-pmh3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.9.021512136010241440384512FALSE
mod-data-export-worker3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.2.123072204810242048384512FALSE
mod-rtac3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.6.02102489612876888128FALSE
mod-circulation-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.2.022880259215361814384512FALSE
mod-calendar3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.1.02102489612876888128FALSE
mod-source-record-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.8.025600500020483500384512FALSE
mod-event-config3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.7.02102489612876888128FALSE
mod-courses3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.102102489612876888128FALSE
mod-circulation-item3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.0.021024896128000FALSE
mod-inventory3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.2.022880259210241814384512FALSE
mod-email3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.17.02102489612876888128FALSE
mod-pubsub3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.13.02153614401024922384512FALSE
mod-circulation3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.2.022880259215361814384512FALSE
mod-di-converter-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.02102489612876888128FALSE
edge-rtac3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.7.12102489612876888128FALSE
edge-orders3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.0.02102489612876888128FALSE
mod-users4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.3.12102489612876888128FALSE
mod-template-engine3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.20.02102489612876888128FALSE
mod-patron-blocks3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.10.021024896102476888128FALSE
mod-audit3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.9.02102489612876888128FALSE
edge-fqm3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:2.0.02102489612876888128FALSE
mod-source-record-manager3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.8.025600500020483500384512FALSE
nginx-edge3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.1421024896128000FALSE
mod-quick-marc3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.1.01228821761281664384512FALSE
nginx-okapi3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.1421024896128000FALSE
okapi-b4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.3.03168414401024922384512FALSE
mod-feesfines3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.1.02102489612876888128FALSE
mod-invoice-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.8.021872153610241024384512FALSE
mod-dcb4579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.1.02102489612876888128FALSE
mod-service-interaction3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.0.12204818442561290384512FALSE
mod-data-export10579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.0.41204815241024000FALSE
mod-patron3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.1.02102489612876888128FALSE
mod-oai-pmh3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.13.024096369020483076384512FALSE
edge-connexion3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.2.02102489612876888128FALSE
mod-kb-ebsco-java3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.02102489612876888128FALSE
mod-notes3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.2.021024896128952384512FALSE
mod-data-export-spring3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.01204818442561536384512FALSE
mod-organizations-storage3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.7.02102489612870088128FALSE
mod-login3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.11.02144012981024768384512FALSE
pub-okapi3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.142102489612876800FALSE
mod-eusage-reports3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.12102489612876888128FALSE


Additional links and Errors


Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files.

  • 09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR Api Access for user 'folio' (9eb67301-6f6e-468f-9b1a-6134dc39a684) requires permission: metadata-provider.incomingrecords.get
  • 09:55:16 [815600/metadata-provider] [fs07000001] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_source_record_manager] ERROR PostgresClient queryAndAnalyze: ERROR: invalid input syntax for type uuid: "undefined" (22P02) - SELECT * FROM get_record_processing_log('3e63f944-40ea-477c-ac21-79bb24780bc5', 'undefined')
  • 09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR FilterApi Permission missing in []

Also we used different order for Tenants when load files, we decided started load files from Tenant 3(qcp1-02) → Tenant 2(qcp1-01) → Tenant 1(qcp1-00) to avoid problem when mod-permissions spiked and  system stacked.

CPU Utilization when mod-permissions spiked and  system stacked.


CPU Utilization when mod-permissions spiked and  system stacked.


Recommendations & Jiras (Optional)

Link to Jira ticket: https://folio-org.atlassian.net/browse/PERF-801


Methodology/Approach

DI tests scenario a data import job profile that creates new MARC authority records for non-matches (Job Profile: KG - Create SRS MARC Authority on nonmatches to 010 $a DUBLICATE for Q) were started from UI on Quesnelia (qcp1) env with  file splitting features enabled on a non-ecs environment.

  • Action for non-matches:  Create MARC authority record 

  • The above files are all stored here - MARC Resources
    - 22k file what was provided from MARC Resources  does nor work, so 50k file was split to file with 25k records and used instead of 22k file.
  • At the time of the test run, Grafana was not available. As a result, response times for Check-In/Check-Out were parsed manually from a .jtl file, using the start and finish dates of the data import tests. These results were visualized in JMeter using a Listener (Response Times Over Time).

Test set

  • Test 1: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant(qcp1-00) only.
  • Test 2: Manually tested 1k, 10k, 25k and 50k records files DI started on one tenant(qcp1-00) only plus Check-in and Checkout (CICO) for 5 concurrent users.
  • Test 3: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrentlyOrder for load file without pause between files: 50k, 25k, 10k, 5k, and 1k for order tenantsTenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)
  • Test 4: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrently. Order for load file with pause between files: 50k, 25k, 10k, 5k, and 1k for order tenantsTenant 3(qcp1-02), Tenant 1(qcp1-00) and Tenant 2(qcp1-01)
  • Test 5: Manually tested 1k, 10k, 25k and 50k records files DI started on 3 tenants concurrentlyOrder for load file without pause between files: 1k, 5k, 10k, 25k and 50k for order tenantsTenant 3(qcp1-02), Tenant 2(qcp1-01) and Tenant 1(qcp1-00)