Versions Compared

Version Old Version 15 New Version Current
Changes made by

Stanislav Nehrii

Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

Overview

  • This document contains the results of testing Data Export (MARC BIB) on the Quesnelia [ECS] release on qcon environment.

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-844
 

Summary

  • Data import tests finished successfully, only Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files. Duration of DI grew in correspondence with the number of records in files.
  • Check-in and Check-out with 5 virtual users was performed during DI Create new MARC authority records jobs for non-matches No issues.
  • Data Import in Quesnelia without CICO perform faster than with it.
  • Comparing Poppy and Quesnelia releases
    • Check-in / Check-out perform better in Quesnelia.  Response time improved during Create jobs for long period of work time on 15% in Average.
    • DI durations improved  - 11%-14% in Average.
  • During testing, we noticed spikes in the mod permissions module. To mitigate this issue and prevent system slowdowns, we adjusted the order of loading files, starting with Tenant 3 (qcp1-02), followed by Tenant 2 (qcp1-01), and finally Tenant 1 (qcp1-00).

Test Results

This table contains durations for jobs with 2 job profiles. 

...

Profile

...

Comparison

Test №1

Test with 1k, 10k, 25k and 50k records files DI started on one tenant only(qcp1-00), and comparative results between Poppy and Quesnelia.

...

% creates

...

File

...

DI duration 
Morning Glory

...

DI duration
Nolana

...

DI duration 
Orchid

...

DI duration 
Poppy

...

Test №2

Test with CICO 5 concurrent users and DI 1K, 5K, 10K, 25K and 50K started on one tenant only.

  • Сomparative Baseline Check-In\Check-Out results without Data Import between Poppy and Quesnelia.

...

CICO, Median time without
DI
(Poppy)

...

593 ms
+4.5%

...

  • Сomparative  Check-In\Check-Out results between Baseline (Quesnelia) and  Check-In\Check-Out plus Data Import (Quesnelia.)

...

DI Duration with CICO
(Quesnelia)

...

20 sec

...

12 min 16 sec

...

1.265

...

  • Сomparative Data Import and Check-In\Check-Out results between Poppy and Quesnelia.

...

# of records 
(Poppy)

...

DI Duration with CICO
(Poppy)

...

CI time 95th pct
(Poppy)

...

CO time Avg
(Poppy)

...

CO time 95th pct
(Poppy)

...

DI Duration with CICO
(Quesnelia)

...

20 sec
-42.8%

...

12 min 16 sec
-11%

...

1.265
-16%

...

Table of Contents
outlinetrue

Overview

  • This document contains the results of testing Data Export (MARC BIB) on the Quesnelia [ECS] release on qcon environment.

Jira Legacy
serverSystem Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-844
 

Summary

  • Data Export tests finished successfully on qcon environment using the profiles Default instances export job profile and srs - holdings and items job profile.
  • Data Export test were run on College and Central tenants, but results for comparing between environment releases were taken from College tenant.
  • Comparing with previous testing results Poppy and Quesnelia releases
    • Data Export processed all files including file with 500k records without errors for Quesnelia releases.
    • Data Export durations improved  - 80% in Average for Quesnelia releases.
  • During testing, we noticed spikes in the mod-data-export up to 593% CPU. 
  • For Test №5 Data Export started on College tenant(cs00000int_0001), Central tenant(cs00000int) and Professional tenant(cs00000int_0002) concurrently using the Default instances export job profile, we observed that the CPU usage of the mod-data-export module was initially at 44% before the test began then it spiked to 109% during the test and remained elevated without returning to the initial state. 

Test Results

This table contains durations for Data Export with 2 job profiles. 

Profile

CSV  FileTenant College (cs00000int_0001)Central Tenant (cs00000int)
ResultStatusResultStatus
DE MARC Bib (Default instances export job profile)

1k.csv0:00:02COMPLETED0:00:05COMPLETED
100k.csv0:02:39COMPLETED0:04:24COMPLETED
500k.csv0:05:21COMPLETED0:06:17COMPLETED
DE MARC Bib (srs - holdings and items)

1k.csv0:00:05COMPLETED0:00:05COMPLETED
100k.csv0:08:15COMPLETED0:05:58COMPLETED
500k.csv0:09:22COMPLETED0:08:28COMPLETED


This table contains durations for Test №5 Data Export for 3 tenants concurrently.

TenantCSV  FileResultStatus

Tenant College

(cs00000int_0001)

500k.csv0:10:24COMPLETED

Tenant Professional

(cs00000int_0002)

500k.csv

0:06:47COMPLETED

Central Tenant

(cs00000int)

500k.csv

0:07:56COMPLETED


Comparison

This table contains durations comparison between Poppy and Quesnelia releases.


ProfileCSV  FileDE Duration/Status OrchidDE Duration/Status Poppy 1 setDE Duration/Status Quesnelia
Tenant College (cs00000int_0001)
DE Duration, DELTA Poppy/Quesnelia
ResultStatusResultStatusResultStatushh:mm:ss / percent
DE MARC Bib (Default instances export job profile)1k.csv

00:00:08COMPLETED0:00:02COMPLETED

-00:00:06
-75%

100k.csv

00:15:36COMPLETED0:02:39COMPLETED

-00:12:57
-83.02%

500k.csv

00:57:25FAIL0:05:21COMPLETED

-00:52:04
-90.68%

DE MARC Bib (srs - holdings and items)1k.csv00:00:27COMPLETED00:00:29COMPLETED0:00:05COMPLETED

-00:00:24
-82.76%

100k.csv00:47:51COMPLETED00:47:23COMPLETED0:08:15COMPLETED

-00:39:08
-82.59%

500k.csv04:00:26COMPLETED04:11:09FAIL0:09:22COMPLETED

-04:01:47
-96.27%



Resource utilization for Test #1 and Test #2

...

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU spike was 32%.

DB Connections

DB connections was 1470.

DB load

                                                                                                                     

...

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 35%.

DB Connections

DB connections was 1377.

DB load

                                                                                                                     

...

#TOP 5 SQL statements
1
select mre1_0.id,mre1_0.content,mre1_0.external_id,mre1_0.leader_record_status,mre1_0.record_type,mre1_0.state,mre1_0.suppress_discovery from v_marc_records_lb mre1_0 where mre1_0.external_id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$
2
INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING
3
select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)
4
select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1
5
select eie1_0.id,eie1_0.instance_id,eie1_0.job_execution_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.instance_id offset $4 rows fetch first $5 rows only

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: fse

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled

...

titleQuesnelia modules memory and CPU parameters

...

,$76,$77,$78,$
2
INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING
3
select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)
4
select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1
5
select eie1_0.id,eie1_0.instance_id,eie1_0.job_execution_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.instance_id offset $4 rows fetch first $5 rows only


Resource utilization for Test #5

Expand
titleResource utilization table


CPURAM
mod-data-export-b592%mod-data-export-b108%
mod-inventory-b10%mod-inventory-b78%
mod-source-record-storage-b1.80%mod-source-record-storage-b40%
mod-authtoken-b1.70%mod-source-record-manager-b39%
mod-source-record-manager-b1.50%okapi-b32%
okapi-b1.50%mod-users-bl-b24%
mod-inventory-storage-b0.60%mod-authtoken-b18%
mod-users-bl-b0.60%mod-inventory-storage-b13%
nginx-okapi0.40%nginx-okapi4%


Service CPU Utilization

Here we can see that mod-data-export used 593% CPU in spike.

Image Added

Service Memory Utilization

We observed that the CPU usage of the mod-data-export module was initially at 44% before the test began. It spiked to 109% during the test and remained elevated without returning to the initial state.

Image Added

DB CPU Utilization

DB CPU was 50%.

Image Added

DB Connections

DB connections was 1368.

Image Added

DB load

 Image Added                                                                                                                    

Top SQL-queries

Image Added

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: fse

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled


Expand
titleQuesnelia modules memory and CPU parameters


ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
qcon-pvt








Thu May 23 10:47:25 UTC 2024








mod-remote-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-agreementsstorage:73.02.02159249201488447212810240396005120512
mod-finance-ebsconetstorage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-ebsconetstorage:28.26.0210241248896102412870070088128256
mod-consortiancip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortiancip:1.114.0423072102420488961282048768512881024128
mod-organizationsagreements1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizationsagreements:17.90.0210241592896148812870008801280
mod-serials-managementebsconet1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-managementebsconet:12.02.0224801248231210241281792700384128512256
mod-settingsconsortia1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settingsconsortia:1.1.0.321024307289620482001287682048885121281024
mod-searchorganizations91579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-searchorganizations:31.39.0-SNAPSHOT.224225921024248089620481281440700512881024128
edgemod-serials-dematicmanagement1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-serials-dematicmanagement:21.20.012102424808962312128768179288384128512
mod-data-importsettings1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-importsettings:3.1.0.312204810241844896256200129276838488512128
mod-tagssearch19579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tagssearch:23.23.0-SNAPSHOT.224210242592896248012820487681440885121281024
modedge-authtokendematic31579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-authtokendematic:2.152.01210241440896115212851292276888128
edgemod-data-coursesimport1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-data-coursesimport:3.1.4.021102420488961844128256768129288384128512
mod-inventory-updatetags1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-updatetags:32.32.02102489612876888128
mod-notifyauthtoken13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notifyauthtoken:32.215.01210241440896115212851276892288128
modedge-configurationcourses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-configurationcourses:51.104.02102489612876888128
mod-ordersinventory-storageupdate1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ordersinventory-storageupdate:133.73.02102489651212870076888128
edgemod-caiasoftnotify1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-caiasoftnotify:23.2.02102489612876888128
mod-login-samlconfiguration1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-samlconfiguration:25.810.02102489612876888128
mod-ermorders-usage-harvesterstorage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ermorders-usage-harvesterstorage:413.57.02102489612851276870088128
modedge-password-validatorcaiasoft1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-password-validatorcaiasoft:32.2.0214401024129889612876838488512128
mod-login-licensessaml1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-licensessaml:62.08.02248010242312896128179276838488512128
mod-erm-usage-gobiharvester1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-gobiharvester:24.85.02102489612870076888128
mod-bulkpassword-operationsvalidator1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulkpassword-operationsvalidator:3.2.0.02307214402600129810241281536768384512
mod-fqm-managerlicenses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-managerlicenses:26.0.102300024802600231212820481792384512
edgemod-dcbgobi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-dcbgobi:12.18.02102489612876870088128
mod-bulk-graphqloperations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-graphqloperations:12.120.1021024307289626001281024768153688384128512
mod-fqm-financemanager1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-financemanager:42.90.012102430008962600128700204888384128512
modedge-erm-usagedcb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-erm-usagedcb:41.71.02102489612876888128
mod-batch-printgraphql1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-printgraphql:1.12.1.02102489612876888128
mod-copycatfinance1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycatfinance:14.69.02102451289612876870088128
mod-listserm-usage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-listsusage:24.07.02300010242600896128204876838488512128
mod-entitiesbatch-linksprint1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entitiesbatch-linksprint:31.01.0225921024248089640012814407680881024128
mod-permissionscopycat21579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissionscopycat:1.6.5.02168410241544512512128102476838488512128
pubmod-edgelists1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pubmod-edgelists:20232.060.1402102430008962600128768204803840512
mod-entities-orderslinks1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-orderslinks:123.80.022592204824804001440102401024384512
edge-patron1mod-permissions2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-patronpermissions:6.5.1.02102416848961544256512768102488384128512
pub-edge-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge-ncip:12023.906.214210248961287688801280
mod-users-blorders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-blorders:712.78.022048144011521024512102492238488512128
modedge-invoicepatron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-invoicepatron:5.81.0214401024115289651225692276888128
modedge-inventory-storagencip21579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventoryedge-storagencip:271.9.2.0-SNAPSHOT.73824096102436908962048128307676838488512128
mod-userusers-importbl1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-userusers-importbl:37.87.0210241440896115212851276892288128
mod-senderinvoice1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-senderinvoice:15.128.0210241440896115212851276892288128
edgemod-oaiinventory-pmhstorage12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-oaiinventory-pmhstorage:27.2.90-SNAPSHOT.0738215124096136036901024204814403076384512
mod-data-exportuser-workerimport1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-datauser-export-workerimport:3.28.1023072102420488961024128204876838488512128
mod-rtacsender1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtacsender:31.612.02102489612876888128
modedge-circulationoai-storagepmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-circulationoai-storagepmh:172.29.0228801512259213601536102418141440384512
mod-sourcedata-recordexport-storageworker1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sourcedata-recordexport-storageworker:53.82.0123072560020485000102420483500384512
mod-calendarrtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendarrtac:3.16.02102489612876888128
mod-eventcirculation-configstorage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eventcirculation-configstorage:17.2.7.021024288089625921281536768181488384128512
mod-source-coursesrecord-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-coursesstorage:15.48.10021024560089650001282048768350088384128512
mod-circulation-itemcalendar1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-itemcalendar:3.1.0.02102489612807680880128
mod-event-inventoryconfig31579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-inventoryconfig:202.27.022880102425928961024128181476838488512128
mod-emailcourses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-emailcourses:1.174.0102102489612876888128
mod-circulation-item21579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:241.20.10228801024259289615361281814038405120
mod-pubsubinventory13579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsubinventory:20.2.13.02153628801440259210249221814384512
mod-di-converter-storage2email1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.2email:1.17.02102489612876888128
edgemod-orderscirculation12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-orderscirculation:324.02.0121024288089625921281536768181488384128512
edgemod-rtacpubsub1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-rtacpubsub:2.713.102102415368961440128102476892288384128512
mod-di-converter-usersstorage12579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.3.1di-converter-storage:2.2.22102489612876888128
modedge-template-engineorders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-template-engineorders:13.200.02102489612876888128
modedge-patron-blocksrtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-patron-blocksrtac:12.107.0121024896102412876888128
mod-auditusers1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-auditusers:219.93.012102489612876888128
edgemod-template-fqmengine1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-template-fqmengine:21.020.02102489612876888128
mod-sourcepatron-record-managerblocks1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sourcepatron-record-managerblocks:31.810.0256001024500089620481024350076838488512128
nginxmod-edgeaudit1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginxmod-edgeaudit:20232.069.1402102489612807680880128
modedge-quick-marcfqm1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-quick-marcfqm:52.10.012228810242176896128166476838488512nginx-okapi128
mod-source-record-manager1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14mod-source-record-manager:3.8.0210245600896500012820480350003840512
okapinginx-bedge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapinginx-edge:52023.306.014321684102414408961024128922038405120
mod-quick-feesfinesmarc1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-feesfinesmarc:195.1.021102422888962176128768166488384128512
modnginx-invoice-storageokapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modnginx-invoice-storageokapi:52023.806.014218721024153689610241281024038405120
modokapi-service-interactionb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4okapi:5.3.0.123204816841844144025610241290922384512
mod-dcbfeesfines1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcbfeesfines:119.1.02102489612876888128
mod-invoice-patronstorage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-patronstorage:65.18.021024187289615361281024768102488384128512
mod-dataservice-exportinteraction1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dataservice-exportinteraction:54.0.011220481524184410242560129003840512
mod-oai-pmhdcb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmhdcb:31.131.024096102436908962048128307676838488512128
edgemod-connexionpatron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edgemod-connexionpatron:6.1.2.02102489612876888128
mod-notesdata-export1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notesdata-export:5.20.0211024204889615241281024952038405120
mod-kboai-ebsco-javapmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kboai-ebsco-javapmh:43.013.021024409689636901282048768307688384128512
modedge-loginconnexion1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/modedge-loginconnexion:71.112.02144010241298896102412876838488512128
mod-organizations-storagenotes1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storagenotes:45.72.02102489612870095288384128512
mod-datakb-exportebsco-springjava1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-datakb-exportebsco-springjava:34.20.012204810241844896256128153676838488512128
pubmod-okapilogin1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pubmod-okapilogin:20237.0611.1402102414408961298128102476803840512
mod-eusageorganizations-reportsstorage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.12102489612876888128

Additional links and Errors

Test №5 had one failed record for Tenant 2(qcp1-01) when processed 50k files.

  • 09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR Api Access for user 'folio' (9eb67301-6f6e-468f-9b1a-6134dc39a684) requires permission: metadata-provider.incomingrecords.get
  • 09:55:16 [815600/metadata-provider] [fs07000001] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_source_record_manager] ERROR PostgresClient queryAndAnalyze: ERROR: invalid input syntax for type uuid: "undefined" (22P02) - SELECT * FROM get_record_processing_log('3e63f944-40ea-477c-ac21-79bb24780bc5', 'undefined')
  • 09:55:16 [526300/metadata-provider] [fs07000001] [] [mod-authtoken] ERROR FilterApi Permission missing in []

Also we used different order for Tenants when load files, we decided started load files from Tenant 3(qcp1-02) → Tenant 2(qcp1-01) → Tenant 1(qcp1-00) to avoid problem when mod-permissions spiked and  system stacked.

CPU Utilization when mod-permissions spiked and  system stacked.

CPU Utilization when mod-permissions spiked and  system stacked.

organizations-storage:4.7.02102489612870088128
mod-data-export-spring1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.01204818442561536384512
pub-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.142102489612876800
mod-eusage-reports1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.12102489612876888128



Methodology/Approach

Data Export tests scenario using the profiles Default instances export job profile and srs - holdings and items were started from UI on Quesnelia (qcon) ecs environment.

...

  • Test 1: Manually tested 1k, 100k and 500k records files Data Export started on one College tenant(cs00000int_0001) only using Default instances export job profile.
  • Test 2: Manually tested 1k, 100k and 500k records files Data Export started on one College tenant(cs00000int_0001) only using srs - holdings and items job profile.
  • Test 3: Manually tested 1k, 100k and 500k records files Data Export started on central Central tenant(cs00000int) only using Default instances export job profile.
  • Test 4: Manually tested 1k, 100k and 500k records files Data Export started on central Central tenant(cs00000int) only using srs - holdings and items job profile.
  • Test 5: Manually tested 500k records file Data Export started on College tenant(cs00000int_0001), Central tenant(cs00000int) and Professional tenant(cs00000int_0002) concurrently using Default instances export job profile. 

To get status and time range for export jobs the query used: 

...