Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  • Сomparative Baseline Check-In\Check-Out results without Data Import between Poppy and Quesnelia.

CICO, Median time without
DI
(Poppy)

CICO, 95% time without
DI
(Poppy)
CICO, Median time without
DI
(Quesnelia)
CICO, 95% time without
DI
(Quesnelia)
CICO, Avg time without
DI
(Quesnelia)
Check-In516 ms567 ms503 ms
-2.5%

593 ms
+4.5%

511 ms

Check-Out910 ms2094 ms 836 ms
-8%
1117 ms
-46%
876 ms


...

  • Сomparative Data Import and Check-In\Check-Out results between Poppy and Quesnelia.

# of records 
(Poppy)

DI Duration with CICO
(Poppy)

CI time Avg
(Poppy)

CI time 95th pct
(Poppy)

CO time Avg
(Poppy)

CO time 95th pct
(Poppy)

# of records
(Quesnelia)

DI Duration with CICO
(Quesnelia)

CI time Avg
(Quesnelia)
CI time 95th pct
(Quesnelia)
CO time Avg
(Quesnelia)
CO time 95th pct
(Quesnelia)
1,00035 sec0.5250.5761.0781.3261,000

20 sec
-42.8%

0.560
+6%
0.754
+30%
1.164
+8%
1.313
-1%
5,0001 min 41 sec0.5130.6120.91.0195,0001 min 19 sec
-21.7%
0.701
+36%
1.171
+91%
1.141
+26%
1.790
+75%
10,0003 min 4 sec0.5810.6851.0161.32110,0002 min 35 sec
-15.7%
0.723
+24%
1.024
+49%
1.179
+16%
1.494
+13%
22,7786 min 32 sec0.5981.5421.2441.72925,0006 min 26 sec
-1.5%
0.722
+20%
1.024
-33%
1.180
-5%
1.494
-13%
50,00013 min 48 sec0.6711.9531.512.0950,000

12 min 16 sec
-11%

0.777
+15%
1.045
-46%

1.265
-16%

1.550
-25%


...

DB connections was 1377.

DB load

 Image Added                                                                                                                    

Top SQL-queries

Image Added


#TOP 5 SQL statements
1
INSERTselect INTO fs09000000_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
2
INSERT INTO fs09000000_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
3
UPDATE fs09000000_mod_source_record_manager.job_execution_progress SET succeeded_records_count = succeeded_records_count + $2, error_records_count = error_records_count + $3 WHERE job_execution_id = $1 Returning *
4
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
5
WITH input_rows(record_id, authority_id) AS (
   VALUES ($1::uuid,$2::uuid)
)
, ins AS (
   INSERT INTO fs09000000_mod_inventory.records_authorities(record_id, authority_id)
   SELECT * FROM input_rows
   ON CONFLICT (record_id) DO UPDATE SET record_id=EXCLUDED.record_id
   RETURNING record_id::uuid, authority_id::uuid
   )
SELECT record_id, authority_id
FROM   ins
UNION  ALL
SELECT c.record_id, c.authority_id 
FROM   input_rows
JOIN   fs09000000_mod_inventory.records_authorities c USING (record_id);

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: fse

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled

...

mre1_0.id,mre1_0.content,mre1_0.external_id,mre1_0.leader_record_status,mre1_0.record_type,mre1_0.state,mre1_0.suppress_discovery from v_marc_records_lb mre1_0 where mre1_0.external_id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$
2
INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING
3
select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)
4
select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1
5
select eie1_0.id,eie1_0.instance_id,eie1_0.job_execution_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.instance_id offset $4 rows fetch first $5 rows only



Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: fse

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled


Expand
titleQuesnelia modules memory and CPU parameters


ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
qcon-pvt








Thu May 23 10:47:25 UTC 2024








mod-remote-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.024920447210243960512512
mod-finance-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.021024896102470088128
mod-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.42102489612876888128
mod-agreements1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0215921488128000
mod-ebsconet1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.2.0212481024128700128256
mod-consortia1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.1.023072204812820485121024
mod-organizations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.9.02102489612870088128
mod-serials-management1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.0.02248023121281792384512
mod-settings1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.32102489620076888128
mod-search9579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.3.0-SNAPSHOT.224225922480204814405121024
edge-dematic1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.2.01102489612876888128
mod-data-import1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.1.01204818442561292384512
mod-tags1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.2.02102489612876888128
mod-authtoken3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.15.121440115251292288128
edge-courses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.4.02102489612876888128
mod-inventory-update1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.3.02102489612876888128
mod-notify1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.2.02102489612876888128
mod-configuration1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.10.02102489612876888128
mod-orders-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.7.02102489651270088128
edge-caiasoft1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.2.02102489612876888128
mod-login-saml1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.8.02102489612876888128
mod-erm-usage-harvester1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.5.02102489612876888128
mod-password-validator1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.2.0214401298128768384512
mod-licenses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.0.02248023121281792384512
mod-gobi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.8.02102489612870088128
mod-bulk-operations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.0.023072260010241536384512
mod-fqm-manager1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:2.0.12300026001282048384512
edge-dcb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.1.02102489612876888128
mod-graphql1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.12102489612876888128
mod-finance1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.9.02102489612870088128
mod-erm-usage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.7.02102489612876888128
mod-batch-print1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.1.02102489612876888128
mod-copycat1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.6.02102451212876888128
mod-lists1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:2.0.02300026001282048384512
mod-entities-links1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.0.0225922480400144001024
mod-permissions2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.5.02168415445121024384512
pub-edge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.142102489612876800
mod-orders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.8.022048144010241024384512
edge-patron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.1.02102489625676888128
edge-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.22102489612876888128
mod-users-bl1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.7.021440115251292288128
mod-invoice1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.8.021440115251292288128
mod-inventory-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.2.0-SNAPSHOT.73824096369020483076384512
mod-user-import1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.02102489612876888128
mod-sender1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.12.02102489612876888128
edge-oai-pmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.9.021512136010241440384512
mod-data-export-worker1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.2.123072204810242048384512
mod-rtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.6.02102489612876888128
mod-circulation-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.2.022880259215361814384512
mod-source-record-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.8.025600500020483500384512
mod-calendar1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.1.02102489612876888128
mod-event-config1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.7.02102489612876888128
mod-courses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.102102489612876888128
mod-circulation-item1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.0.021024896128000
mod-inventory3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.2.022880259210241814384512
mod-email1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.17.02102489612876888128
mod-circulation2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.2.122880259215361814384512
mod-pubsub1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.13.02153614401024922384512
mod-di-converter-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.22102489612876888128
edge-orders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.0.02102489612876888128
edge-rtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.7.12102489612876888128
mod-users1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.3.12102489612876888128
mod-template-engine1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.20.02102489612876888128
mod-patron-blocks1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.10.021024896102476888128
mod-audit1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.9.02102489612876888128
edge-fqm1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:2.0.02102489612876888128
mod-source-record-manager1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.8.025600500020483500384512
nginx-edge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.1421024896128000
mod-quick-marc1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.1.01228821761281664384512
nginx-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.1421024896128000
okapi-b1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.3.03168414401024922384512
mod-feesfines1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.1.02102489612876888128
mod-invoice-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.8.021872153610241024384512
mod-service-interaction1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.0.12204818442561290384512
mod-dcb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.1.02102489612876888128
mod-patron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.1.02102489612876888128
mod-data-export1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.0.01204815241024000
mod-oai-pmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.13.024096369020483076384512
edge-connexion1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.2.02102489612876888128
mod-notes1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.2.021024896128952384512
mod-kb-ebsco-java1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.02102489612876888128
mod-login1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.11.02144012981024768384512
mod-organizations-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.7.02102489612870088128
mod-data-export-spring1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.01204818442561536384512
pub-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.142102489612876800
mod-eusage-reports1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.12102489612876888128



Additional links and Errors

...