Skip to end of banner
Go to start of banner

PTF - Data Import MARC BIB + CI/CO (Quesnelia) [ECS]

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Overview

  • This document contains the results of testing  Data Import for MARC Bibliographic records with an update job on the Quesnelia [ECS] release on qcon environment. 

PERF-846 - Getting issue details... STATUS  

Summary

  • Data Import tests finished successfully on qcon environment using the PTF - Updates Success - 2 profile and file with 25k records.
  • Comparing with previous testing results Poppy and Quesnelia releases
    • Data Import processed all jobs including test on 3 tenants concurrently without errors for Quesnelia releases.
    • Data Import durations stayed in the same time range in Average for Quesnelia releases but it works stable and without errors.
  • During testing, we noticed that mod-permission did not have any spikes and used 12% CPU for Quesnelia releases. For Poppy releases we had error.

Test Runs 

Test â„–ScenarioTest ConditionsResults

1
DI MARC Bib Create5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)
Completed
CICO 8 users

2
DI MARC Bib Update5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause)


Completed

CICO8 users

Test Results

This table contains durations for Data Import. 

Comparison

This table contains durations comparison between Poppy and Quesnelia releases.


Resource utilization for Test â„–1

 Resource utilization table
CPURAM
mod-data-import-b56%mod-inventory-b65%
nginx-okapi56%mod-data-import-b53%
mod-di-converter-storage-b38%mod-source-record-manager-b48%
okapi-b36%mod-source-record-storage-b43%
mod-inventory-storage-b23%okapi-b34%
mod-source-record-storage-b13%mod-di-converter-storage-b33%
mod-source-record-manager-b11%mod-feesfines-b33%
mod-feesfines-b10%mod-patron-blocks-b31%
mod-quick-marc-b8%mod-quick-marc-b31%
mod-pubsub-b8%mod-pubsub-b30%
mod-authtoken-b7%mod-configuration-b28%
mod-configuration-b6%mod-users-bl-b26%
pub-okapi4%mod-circulation-b25%
mod-remote-storage-b3%mod-authtoken-b20%
mod-circulation-storage-b3%mod-circulation-storage-b20%
mod-inventory-update-b2%mod-inventory-storage-b18%
mod-circulation-b2%mod-remote-storage-b17%
mod-patron-blocks-b1%nginx-okapi4%
mod-users-bl-b1%pub-okapi4%

Service CPU Utilization

Here we can see that mod-data-import used 150% CPU in spikes.

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 92%.

DB Connections

Max number of DB connections was 1690.

DB load

                                                                                                                     

Top SQL-queries


#TOP 5 SQL statements
1
INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
2
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
3
INSERT INTO cs00000int_0001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) 
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
4
  INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date
5
INSERT INTO cs00000int_0001_mod_inventory_storage.holdings_record (id, jsonb) VALUES ($1, $2) RETURNING jsonb

Resource utilization for Test â„–2

 Resource utilization table
RAMCPU
nginx-okapi67%mod-data-import-b72%
mod-data-import-b50%mod-inventory-b66%
okapi-b42%mod-source-record-manager-b52%
mod-di-converter-storage-b41%mod-source-record-storage-b45%
mod-source-record-storage-b21%mod-pubsub-b35%
mod-inventory-storage-b20%okapi-b35%
mod-source-record-manager-b9%mod-di-converter-storage-b34%
mod-quick-marc-b9%mod-feesfines-b33%
mod-feesfines-b9%mod-patron-blocks-b32%
mod-authtoken-b9%mod-quick-marc-b31%
mod-pubsub-b7%mod-circulation-storage-b30%
mod-configuration-b6%mod-configuration-b29%
pub-okapi4%mod-users-bl-b28%
mod-circulation-storage-b2%mod-circulation-b28%
mod-remote-storage-b2%mod-inventory-storage-b22%
mod-inventory-update-b2%mod-authtoken-b21%
mod-circulation-b1%mod-remote-storage-b18%
mod-patron-blocks-b1%nginx-okapi4%
mod-users-bl-b0.8%pub-okapi4%

Service CPU Utilization

Here we can see that mod-data-import used 130% CPU in spikes.

Service Memory Utilization

Here we can see that all modules show a stable trend.

DB CPU Utilization

DB CPU was 92%.

DB Connections

Max number of DB connections was 1685.

DB load

                                                                                                                     

Top SQL-queries


#TOP 5 SQL statements
1
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
2
INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
3
INSERT INTO cs00000int_0001_mod_source_record_manager.journal_records (id, job_execution_id, source_id, source_record_order, entity_type, entity_id, entity_hrid, action_type, action_status, error, action_date, title, instance_id, holdings_id, order_id, permanent_location_id, tenant_id) 
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
4
INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date
5
UPDATE cs00000int_0001_mod_inventory_storage.instance SET jsonb = $1::jsonb WHERE id=?

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • OpenSearch

    • domain: fse

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled


 Quesnelia modules memory and CPU parameters
ModuleTask Def. RevisionModule VersionTask CountMem Hard LimitMem Soft limitCPU unitsXmxMetaspaceSizeMaxMetaspaceSize
qcon-pvt








Thu May 23 10:47:25 UTC 2024








mod-remote-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.024920447210243960512512
mod-finance-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.021024896102470088128
mod-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.42102489612876888128
mod-agreements1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0215921488128000
mod-ebsconet1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.2.0212481024128700128256
mod-consortia1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.1.023072204812820485121024
mod-organizations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.9.02102489612870088128
mod-serials-management1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.0.02248023121281792384512
mod-settings1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.32102489620076888128
mod-search9579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.3.0-SNAPSHOT.224225922480204814405121024
edge-dematic1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.2.01102489612876888128
mod-data-import1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.1.01204818442561292384512
mod-tags1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.2.02102489612876888128
mod-authtoken3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.15.121440115251292288128
edge-courses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.4.02102489612876888128
mod-inventory-update1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.3.02102489612876888128
mod-notify1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.2.02102489612876888128
mod-configuration1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.10.02102489612876888128
mod-orders-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.7.02102489651270088128
edge-caiasoft1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.2.02102489612876888128
mod-login-saml1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.8.02102489612876888128
mod-erm-usage-harvester1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.5.02102489612876888128
mod-password-validator1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.2.0214401298128768384512
mod-licenses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.0.02248023121281792384512
mod-gobi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.8.02102489612870088128
mod-bulk-operations1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.0.023072260010241536384512
mod-fqm-manager1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:2.0.12300026001282048384512
edge-dcb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.1.02102489612876888128
mod-graphql1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.12102489612876888128
mod-finance1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.9.02102489612870088128
mod-erm-usage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.7.02102489612876888128
mod-batch-print1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.1.02102489612876888128
mod-copycat1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.6.02102451212876888128
mod-lists1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:2.0.02300026001282048384512
mod-entities-links1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.0.0225922480400144001024
mod-permissions2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.5.02168415445121024384512
pub-edge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.142102489612876800
mod-orders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.8.022048144010241024384512
edge-patron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.1.02102489625676888128
edge-ncip1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.22102489612876888128
mod-users-bl1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.7.021440115251292288128
mod-invoice1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.8.021440115251292288128
mod-inventory-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.2.0-SNAPSHOT.73824096369020483076384512
mod-user-import1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.02102489612876888128
mod-sender1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.12.02102489612876888128
edge-oai-pmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.9.021512136010241440384512
mod-data-export-worker1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.2.123072204810242048384512
mod-rtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.6.02102489612876888128
mod-circulation-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.2.022880259215361814384512
mod-source-record-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.8.025600500020483500384512
mod-calendar1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.1.02102489612876888128
mod-event-config1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.7.02102489612876888128
mod-courses1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.102102489612876888128
mod-circulation-item1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.0.021024896128000
mod-inventory3579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.2.022880259210241814384512
mod-email1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.17.02102489612876888128
mod-circulation2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.2.122880259215361814384512
mod-pubsub1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.13.02153614401024922384512
mod-di-converter-storage2579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.22102489612876888128
edge-orders1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.0.02102489612876888128
edge-rtac1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.7.12102489612876888128
mod-users1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.3.12102489612876888128
mod-template-engine1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.20.02102489612876888128
mod-patron-blocks1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.10.021024896102476888128
mod-audit1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.9.02102489612876888128
edge-fqm1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:2.0.02102489612876888128
mod-source-record-manager1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.8.025600500020483500384512
nginx-edge1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.1421024896128000
mod-quick-marc1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.1.01228821761281664384512
nginx-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.1421024896128000
okapi-b1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.3.03168414401024922384512
mod-feesfines1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.1.02102489612876888128
mod-invoice-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.8.021872153610241024384512
mod-service-interaction1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.0.12204818442561290384512
mod-dcb1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.1.02102489612876888128
mod-patron1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.1.02102489612876888128
mod-data-export1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.0.01204815241024000
mod-oai-pmh1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.13.024096369020483076384512
edge-connexion1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.2.02102489612876888128
mod-notes1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.2.021024896128952384512
mod-kb-ebsco-java1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.02102489612876888128
mod-login1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.11.02144012981024768384512
mod-organizations-storage1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.7.02102489612870088128
mod-data-export-spring1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.01204818442561536384512
pub-okapi1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.142102489612876800
mod-eusage-reports1579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.12102489612876888128


Methodology/Approach

DI tests scenario (DI MARC Bib Create and Update) were started from UI on Quesnelia (qcon) env with  file splitting features enabled on a ecs environment..

Test runs:

  • Test 1: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Create ) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.
  • Test 2: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Update) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.

At the time of the test run, Grafana was not available. As a result, response times for Check-In/Check-Out were parsed manually from a .jtl files, using the start and finish dates of the data import tests. These results were visualized in JMeter using a Listener (Response Times Over Time).



  • No labels