Overview https://folio-org.atlassian.net/browse/PERF-841
Summary Data Import tests finished successfully on qcon environment using the PTF - Updates Success - 2 profile and file with 25k records .
Comparing with previous testing results Poppy and Quesnelia releases
Data Import processed all jobs including test on 3 tenants concurrently without errors for Quesnelia releases.
Data Import durations stayed in the same time range in Average for Quesnelia releases but it works stable and without errors.
During testing, we noticed that mod-permission did not have any spikes and used 12% CPU for Quesnelia releases. For Poppy releases we had error .
Test Results and Test Runs This table contains durations for Data Import.
DI Duration Quesnelia (hh:mm:ss)
DI MARC Bib Update (PTF - Updates Success - 2 )
Professional (cs00000int_0002)
Professional (cs00000int_0002)
Completed
Comparison This table contains durations comparison between Poppy and Quesnelia releases.
DI MARC Bib Update (PTF - Updates Success - 2 )
Resource utilization for Test Set Resource utilization table CPU RAM mod-inventory-b 125% mod-inventory-b 67% mod-di-converter-storage-b 69% mod-data-import-b 55% nginx-okapi 64% mod-source-record-storage-b 50% okapi-b 41% mod-source-record-manager-b 45% mod-inventory-storage-b 29% mod-di-converter-storage-b 34% mod-source-record-storage-b 24% okapi-b 31% mod-source-record-manager-b 12% mod-inventory-storage-b 15% mod-data-import-b 1.80% nginx-okapi 4.60% pub-okapi 0.20% pub-okapi 4.40%
Service CPU Utilization Here we can see that mod-inventory used 125% CPU, mod-di-converter-storage and nginx-okapi used 65%-70% CPU.
Service Memory Utilization Here we can see that all modules show a stable trend.
DB CPU Utilization DB CPU was 95%.
DB Connections Max number of DB connections was 1650 .
DB load Top SQL-queries
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
VALUES ($1, $2, $3::json, $4, $5)
ON CONFLICT (tenant_id, instance_id)
DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date
WITH deleted_rows AS ( delete from marc_indexers mi where exists( select 1 from marc_records_tracking mrt where mrt.is_dirty = true and mrt.marc_id = mi.marc_id and mrt.version > mi.version )returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi where exists( select 1 from records_lb where records_lb.id = mi.marc_id and records_lb.state = 'OLD' )returning mi.marc_id) INSERT INTO marc_indexers_deleted_ids SELECT DISTINCT marc_id FROM deleted_rows UNION SELECT marc_id FROM deleted_rows2
INSERT INTO cs00000int_0002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
Appendix Infrastructure PTF - environment Quesnelia (qcon)
11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
1 instance of db.r6.xlarge database instance: Writer instance
OpenSearch
MSK - tenat
4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Kafka consolidated topics enabled
Quesnelia modules memory and CPU parameters Module Task Def. Revision Module Version Task Count Mem Hard Limit Mem Soft limit CPU units Xmx MetaspaceSize MaxMetaspaceSize qcon-pvt Thu May 23 10:47:25 UTC 2024 mod-remote-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:3.2.0 2 4920 4472 1024 3960 512 512 mod-finance-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance-storage:8.6.0 2 1024 896 1024 700 88 128 mod-ncip 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ncip:1.14.4 2 1024 896 128 768 88 128 mod-agreements 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:7.0.0 2 1592 1488 128 0 0 0 mod-ebsconet 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-ebsconet:2.2.0 2 1248 1024 128 700 128 256 mod-consortia 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-consortia:1.1.0 2 3072 2048 128 2048 512 1024 mod-organizations 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations:1.9.0 2 1024 896 128 700 88 128 mod-serials-management 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-serials-management:1.0.0 2 2480 2312 128 1792 384 512 mod-settings 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-settings:1.0.3 2 1024 896 200 768 88 128 mod-search 9 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:3.3.0-SNAPSHOT.224 2 2592 2480 2048 1440 512 1024 edge-dematic 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dematic:2.2.0 1 1024 896 128 768 88 128 mod-data-import 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-import:3.1.0 1 2048 1844 256 1292 384 512 mod-tags 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-tags:2.2.0 2 1024 896 128 768 88 128 mod-authtoken 3 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.15.1 2 1440 1152 512 922 88 128 edge-courses 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-courses:1.4.0 2 1024 896 128 768 88 128 mod-inventory-update 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-update:3.3.0 2 1024 896 128 768 88 128 mod-notify 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notify:3.2.0 2 1024 896 128 768 88 128 mod-configuration 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-configuration:5.10.0 2 1024 896 128 768 88 128 mod-orders-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders-storage:13.7.0 2 1024 896 512 700 88 128 edge-caiasoft 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-caiasoft:2.2.0 2 1024 896 128 768 88 128 mod-login-saml 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login-saml:2.8.0 2 1024 896 128 768 88 128 mod-erm-usage-harvester 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage-harvester:4.5.0 2 1024 896 128 768 88 128 mod-password-validator 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-password-validator:3.2.0 2 1440 1298 128 768 384 512 mod-licenses 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-licenses:6.0.0 2 2480 2312 128 1792 384 512 mod-gobi 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-gobi:2.8.0 2 1024 896 128 700 88 128 mod-bulk-operations 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-bulk-operations:2.0.0 2 3072 2600 1024 1536 384 512 mod-fqm-manager 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-fqm-manager:2.0.1 2 3000 2600 128 2048 384 512 edge-dcb 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-dcb:1.1.0 2 1024 896 128 768 88 128 mod-graphql 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-graphql:1.12.1 2 1024 896 128 768 88 128 mod-finance 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-finance:4.9.0 2 1024 896 128 700 88 128 mod-erm-usage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-erm-usage:4.7.0 2 1024 896 128 768 88 128 mod-batch-print 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-batch-print:1.1.0 2 1024 896 128 768 88 128 mod-copycat 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-copycat:1.6.0 2 1024 512 128 768 88 128 mod-lists 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-lists:2.0.0 2 3000 2600 128 2048 384 512 mod-entities-links 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-entities-links:3.0.0 2 2592 2480 400 1440 0 1024 mod-permissions 2 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-permissions:6.5.0 2 1684 1544 512 1024 384 512 pub-edge 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-edge:2023.06.14 2 1024 896 128 768 0 0 mod-orders 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-orders:12.8.0 2 2048 1440 1024 1024 384 512 edge-patron 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-patron:5.1.0 2 1024 896 256 768 88 128 edge-ncip 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-ncip:1.9.2 2 1024 896 128 768 88 128 mod-users-bl 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users-bl:7.7.0 2 1440 1152 512 922 88 128 mod-invoice 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice:5.8.0 2 1440 1152 512 922 88 128 mod-inventory-storage 2 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:27.2.0-SNAPSHOT.738 2 4096 3690 2048 3076 384 512 mod-user-import 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-user-import:3.8.0 2 1024 896 128 768 88 128 mod-sender 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-sender:1.12.0 2 1024 896 128 768 88 128 edge-oai-pmh 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-oai-pmh:2.9.0 2 1512 1360 1024 1440 384 512 mod-data-export-worker 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-worker:3.2.1 2 3072 2048 1024 2048 384 512 mod-rtac 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-rtac:3.6.0 2 1024 896 128 768 88 128 mod-circulation-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:17.2.0 2 2880 2592 1536 1814 384 512 mod-source-record-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-storage:5.8.0 2 5600 5000 2048 3500 384 512 mod-calendar 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:3.1.0 2 1024 896 128 768 88 128 mod-event-config 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-event-config:2.7.0 2 1024 896 128 768 88 128 mod-courses 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-courses:1.4.10 2 1024 896 128 768 88 128 mod-circulation-item 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-item:1.0.0 2 1024 896 128 0 0 0 mod-inventory 3 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.2.0 2 2880 2592 1024 1814 384 512 mod-email 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-email:1.17.0 2 1024 896 128 768 88 128 mod-circulation 2 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation:24.2.1 2 2880 2592 1536 1814 384 512 mod-pubsub 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.13.0 2 1536 1440 1024 922 384 512 mod-di-converter-storage 2 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.2.2 2 1024 896 128 768 88 128 edge-orders 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-orders:3.0.0 2 1024 896 128 768 88 128 edge-rtac 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-rtac:2.7.1 2 1024 896 128 768 88 128 mod-users 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-users:19.3.1 2 1024 896 128 768 88 128 mod-template-engine 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-template-engine:1.20.0 2 1024 896 128 768 88 128 mod-patron-blocks 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.10.0 2 1024 896 1024 768 88 128 mod-audit 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-audit:2.9.0 2 1024 896 128 768 88 128 edge-fqm 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-fqm:2.0.0 2 1024 896 128 768 88 128 mod-source-record-manager 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.8.0 2 5600 5000 2048 3500 384 512 nginx-edge 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-edge:2023.06.14 2 1024 896 128 0 0 0 mod-quick-marc 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:5.1.0 1 2288 2176 128 1664 384 512 nginx-okapi 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14 2 1024 896 128 0 0 0 okapi-b 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.3.0 3 1684 1440 1024 922 384 512 mod-feesfines 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:19.1.0 2 1024 896 128 768 88 128 mod-invoice-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-invoice-storage:5.8.0 2 1872 1536 1024 1024 384 512 mod-service-interaction 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-service-interaction:4.0.1 2 2048 1844 256 1290 384 512 mod-dcb 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-dcb:1.1.0 2 1024 896 128 768 88 128 mod-patron 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:6.1.0 2 1024 896 128 768 88 128 mod-data-export 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export:5.0.0 1 2048 1524 1024 0 0 0 mod-oai-pmh 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-oai-pmh:3.13.0 2 4096 3690 2048 3076 384 512 edge-connexion 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/edge-connexion:1.2.0 2 1024 896 128 768 88 128 mod-notes 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.2.0 2 1024 896 128 952 384 512 mod-kb-ebsco-java 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-kb-ebsco-java:4.0.0 2 1024 896 128 768 88 128 mod-login 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-login:7.11.0 2 1440 1298 1024 768 384 512 mod-organizations-storage 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-organizations-storage:4.7.0 2 1024 896 128 700 88 128 mod-data-export-spring 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-data-export-spring:3.2.0 1 2048 1844 256 1536 384 512 pub-okapi 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14 2 1024 896 128 768 0 0 mod-eusage-reports 1 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-eusage-reports:2.1.1 2 1024 896 128 768 88 128
Methodology/Approach DI tests scenario(DI MARC Bib Update) were started from UI.
Test set:
Test 1: Manually tested 25k records files DI started on College tenant( cs00000int_0001 ) only.
Test 2: Manually tested 25k records files DI started on College tenant( cs00000int_0001 ) and Professional tenant( cs00000int_0002 ) concurrently, step 30% .
Test 3: Manually tested 25k records files DI started on College tenant( cs00000int_0001 ), Professional tenant( cs00000int_0002 ) and School tenant( cs00000int_0003 ) concurrently, step 30% .
To get status and time range for import jobs the query used:
SQL Query select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0001_mod_source_record_manager.job_execution order by started_date desc limit 1000;
select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0002_mod_source_record_manager.job_execution order by started_date desc limit 1000;
select file_name,started_date,completed_date, completed_date - started_date as duration ,status
from cs00000int_0003_mod_source_record_manager.job_execution order by started_date desc limit 1000;
