Versions Compared

Version Old Version 1 New Version 2
Changes made by

Stanislav Nehrii

Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  • Data Import tests finished successfully on qcon environment using the PTF - Updates Success - 2 profile and file with 25k records.
  • Comparing with previous testing results Poppy and Quesnelia releases
    • Data Import processed all jobs including test on 3 tenants concurrently without errors for Quesnelia releases.
    • Data Import durations stayed in the same time range in Average for Quesnelia releases but it works stable and without errors.
  • During testing, we noticed that mod-permission did not have any spikes and used 12% CPU for Quesnelia releases. For Poppy releases we had error.

Test

...

This table contains durations for Data Import. 

...

College
(cs00000int_0001)

...

Runs 


Test №

Date

Test Conditions

Results

1.

Mar 4, 2024, 5:42 PM UTC

1 CI or CO per min for small tenants, 2 CICO per min for large tenants

Completed
2.Mar 4, 2024, 8:19 PM UTC

2 CI or CO per min for small tenants, 4 CICO per min for large tenants

Completed
3.Mar 5, 2024, 2:39 PM UTC

4 CICO per min for one large tenant

Completed


Test Results

This table contains durations for Data Import. 

25503463806
ProfileTest #TenantMARC FileDI Duration Quesnelia
(hh:mm:ss)		Results
DI MARC Bib Update (PTF - Updates Success - 2)1

College
(cs00000int_0001)

25K.mrc0:17:43Completed
2College
(cs00000int_0001)		25K.mrc0:26:04Completed
Professional 
(cs00000int_0002)
25K.mrc0:25:50Completed
3SchoolCollege
(cs00000int_00030001)		25K.mrc0:3726:2946Completed

Comparison

...

Professional 
(cs00000int_0002)		25K.mrc0:38:06Completed
School
(cs00000int_0003)		25K.mrc0:37:29Completed

Comparison

This table contains durations comparison between Poppy and Quesnelia releases.

Profile MARC File Test run # DI DurationDI Delta
(hh:mm:ss) 
Poppy Quesnelia
DI MARC Bib Update (PTF - Updates Success - 2)  25K.mrc  10:17:180:17:43+00:00:25
+2.41%
20:56:410:51:54-00:04:47
-8.44%
31:33:48
with errors*		1:42:21+00:08:33
+9.12%



Resource utilization for Test

Set

№1

Expand
titleResource utilization table



Service CPU

...

Utilization

Here we can see that mod-inventory used 125% CPU, mod-di-converter-storage

...

and nginx-okapi used 65%-70% CPU.


Service Memory Utilization

Here we can see that all modules show a stable trend.


DB CPU Utilization

DB CPU was 95%.


DB Connections

Max number of DB connections was 1650.


DB load

                                                                                                                     

Top SQL-queries



#TOP 5 SQL statements
1

2

3

4

5

Resource utilization for Test №2

Expand
titleResource utilization table



Service CPU Utilization

Here we can see that mod-inventory used 125% CPU, mod-di-converter-storage and nginx-okapi used 65%-70% CPU.

Image Removed


Service Memory Utilization

Here we can see that all modules show a stable trend.

Image Removed


DB CPU Utilization

DB CPU was 95%.Image Removed


DB Connections

Max number of DB connections was 1650.

Image Removed


DB load

  Image Removed                                                                                                                       

Top SQL-queries

Image Removed



INSERT INTO cs00000int_0002_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
#TOP 5 SQL statements
1
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
2
INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
3
 INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date
4
WITH deleted_rows AS 
( delete from marc_indexers mi where exists( select 1 from marc_records_tracking mrt 
where mrt.is_dirty = true and mrt.marc_id = mi.marc_id and mrt.version > mi.version )
returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi 
where exists( select 1 from records_lb where records_lb.id = mi.marc_id and records_lb.state = 'OLD' )
returning mi.marc_id) INSERT INTO marc_indexers_deleted_ids SELECT DISTINCT marc_id FROM deleted_rows UNION SELECT marc_id FROM deleted_rows2
5

2

3

4

5

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

...

  • Test 1: Manually tested 25k records files DI started on College tenant(cs00000int_0001) only.
  • Test 2: Manually tested 25k records files DI started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
  • Test 3: Manually tested 25k records files DI started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.

...