Versions Compared

Version Old Version 30 New Version Current
Changes made by

Stanislav Nehrii

Stanislav Nehrii

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  • Data Import tests finished successfully on qcon environment using the PTF - Create 2 profile and file with 10k, 25k and 50k records.
  • Data Import test for PTF - Updates Success - 2 profile with 25k records file on 3 tenants concurrently finished with errors for two records. 
  • Comparing with previous testing results Poppy and Quesnelia releases
    • Data Import processed DI MARC Bib Create jobs including test on 2 and 3 tenants concurrently without errors for Quesnelia releases.
    • Data Import processed DI MARC Bib Update job with 25k file for 3 tenants concurrently with two errors.
    • Data Import durations for Create job has performance improvement around 50% for Quesnelia releases.
    • Data Import durations for Update job stayed in the same time range in Average for Quesnelia releases.
  • When comparing the durations of the 'main' job on the first tenant and observing how the second and third jobs affect and slow down the main job, it becomes evident that processing times are twice as slow for three tenants compared to two tenants when handling a 10k file.
  • Additionally, for a DI MARC Bib Update job with 25k file , the duration increases by only 6%.

...

ProfileTest #TenantMARC File

DI Duration by Tenant Quesnelia
(hh:mm:ss)

Results
DI MARC Bib Create (PTF - Create 2)




1

College
(cs00000int_0001)		10K.mrc0:04:52Completed
Professional 
(cs00000int_0002)		10K.mrc0:05:24Completed
2


College
(cs00000int_0001)		10K.mrc0:06:35Completed
Professional 
(cs00000int_0002)		10K.mrc0:06:18Completed
School
(cs00000int_0003)		10K.mrc0:08:13Completed
3College
(cs00000int_0001)		25K.mrc0:11:04Completed
Professional 
(cs00000int_0002)		25K.mrc0:16:16Completed
4College
(cs00000int_0001)		25K.mrc0:11:47Completed
Professional 
(cs00000int_0002		25K.mrc0:19:12Completed
School
(cs00000int_0003)		25K.mrc0:23:41Completed
5College
(cs00000int_0001)		50K.mrc0:36:02Completed
Professional 
(cs00000int_0002)		50K.mrc0:42:50Completed
School
(cs00000int_0003)		50K.mrc0:47:13Completed
DI MARC Bib Update (PTF - Updates Success - 2)











6College
(cs00000int_0001)		10K.mrc0:11:53Completed
Professional 
(cs00000int_0002)		10K.mrc0:14:49Completed
7College
(cs00000int_0001)		10K.mrc0:20:39Completed
Professional 
(cs00000int_0002)		10K.mrc0:20:07Completed
School
(cs00000int_0003)		10K.mrc0:21:05Completed
8College
(cs00000int_0001)		25K.mrc0:38:41Completed
Professional 
(cs00000int_0002)		25K.mrc0:39:09Completed
9College
(cs00000int_0001)		25K.mrc0:41:32Completed
with errors*
Professional 
(cs00000int_0002)		25K.mrc0:42:34Completed
School
(cs00000int_0003)		25K.mrc0:16:27Completed
10


College
(cs00000int_0001)		50K.mrc1:43:17Completed
Professional 
(cs00000int_0002)		50K.mrc1:50:34Completed
School
(cs00000int_0003)		50K.mrc1:59:29Completed


This table contains durations for Data Import by each tests. 

ProfileTest #TenantsMARC FileDI Duration by Test Quesnelia
(hh:mm:ss)

DI MARC Bib Create (PTF - Create 2)

1

College (cs00000int_0001)
Professional (cs00000int_0002)

10K.mrc0:09:17
2College (cs00000int_0001)
Professional (cs00000int_0002)
School (cs00000int_0003)
10K.mrc0:12:01
3College (cs00000int_0001)
Professional (cs00000int_0002)		25K.mrc0:20:09
4College (cs00000int_0001)
Professional (cs00000int_0002)
School (cs00000int_0003)		25K.mrc0:30:54
5College (cs00000int_0001)
Professional (cs00000int_0002)
School (cs00000int_0003)		50K.mrc1:00:44

DI MARC Bib Update (PTF - Updates Success - 2)

6College (cs00000int_0001)
Professional (cs00000int_0002)		10K.mrc00:20:07
7College (cs00000int_0001)
Professional (cs00000int_0002)
School (cs00000int_0003)		10K.mrc00:27:56
8College (cs00000int_0001)
Professional (cs00000int_0002)		25K.mrc00:47:08
9College (cs00000int_0001)
Professional (cs00000int_0002)
School (cs00000int_0003)		25K.mrc00:26:46
10College (cs00000int_0001)
Professional (cs00000int_0002)
School (cs00000int_0003)		50K.mrc02:22:13

Comparison

...


ProfileTest #TenantMARC File

DI Duration for second and third jobs
сompare to main job
(hh:mm:ss)

DI MARC Bib Create (PTF - Create 2)




1

College
(cs00000int_0001)		10K.mrc0:04:52
Professional 
(cs00000int_0002)		10K.mrc0:05:24
+11%
2


College
(cs00000int_0001)		10K.mrc0:06:35
Professional 
(cs00000int_0002)		10K.mrc0:06:18
-4%
School
(cs00000int_0003)		10K.mrc0:08:13
+25%
3College
(cs00000int_0001)		25K.mrc0:11:04
Professional 
(cs00000int_0002)		25K.mrc0:16:16
+47%
4College
(cs00000int_0001)		25K.mrc0:11:47
Professional 
(cs00000int_0002		25K.mrc0:19:12
+63%
School
(cs00000int_0003)		25K.mrc0:23:41
+101%
5College
(cs00000int_0001)		50K.mrc0:36:02
Professional 
(cs00000int_0002)		50K.mrc0:42:50
+19%
School
(cs00000int_0003)		50K.mrc0:47:13
+31%
DI MARC Bib Update (PTF - Updates Success - 2)











6College
(cs00000int_0001)		10K.mrc0:11:53
Professional 
(cs00000int_0002)		10K.mrc0:14:49
+24%
7College
(cs00000int_0001)		10K.mrc0:20:39
Professional 
(cs00000int_0002)		10K.mrc0:20:07
-3%
School
(cs00000int_0003)		10K.mrc0:21:05
+2%
8College
(cs00000int_0001)		25K.mrc0:38:41
Professional 
(cs00000int_0002)		25K.mrc0:39:09
+1%
9College
(cs00000int_0001)		25K.mrc0:41:32
Professional 
(cs00000int_0002)		25K.mrc0:42:34
+2.5%
School
(cs00000int_0003)		25K.mrc0:16:27
-60%
10


College
(cs00000int_0001)		50K.mrc1:43:17
Professional 
(cs00000int_0002)		50K.mrc1:50:34
+7%
School
(cs00000int_0003)		50K.mrc1:59:29
+16%


This table contains durations comparison between Poppy and Quesnelia releases.

ProfileTest #TenantMARC FileDI Duration
Poppy 
(hh:mm:ss)		DI Duration
Quesnelia
(hh:mm:ss)		DI Delta
(hh:mm:ss)
DI MARC Bib Create




1

College
(cs00000int_0001)		10K.mrc00:10:430:04:52-0:05:51
-54%
Professional 
(cs00000int_0002)		10K.mrc00:10:370:05:24-0:05:13
-49%
2


College
(cs00000int_0001)		10K.mrc00:21:120:06:35-0:14:37
-69%
Professional 
(cs00000int_0002)		10K.mrc00:21:060:06:18-0:14:48
-70%
School
(cs00000int_0003)		10K.mrc00:20:570:08:13-0:12:44
-61%
3College
(cs00000int_0001)		25K.mrc00:23:440:11:04-0:12:40
-53%
Professional 
(cs00000int_0002)		25K.mrc00:23:300:16:16-0:07:14
-31%
4College
(cs00000int_0001)		25K.mrc00:37:110:11:47-0:25:24
-68%
Professional 
(cs00000int_0002		25K.mrc00:37:050:19:12-0:17:53
-48%
School
(cs00000int_0003)		25K.mrc00:36:580:23:41-0:13:17
-36%
5College
(cs00000int_0001)		50K.mrc01:12:540:36:02-0:36:52
-50%
Professional 
(cs00000int_0002)		50K.mrc01:12:440:42:50-0:29:54
-41%
School
(cs00000int_0003)		50K.mrc01:12:350:47:13-0:25:22
-35%
DI MARC Bib Update 











6College
(cs00000int_0001)		10K.mrc00:09:470:11:53+0:02:06
+21%
Professional 
(cs00000int_0002)		10K.mrc00:11:260:14:49+0:03:23
+29%
7College
(cs00000int_0001)		10K.mrc00:19:080:20:39+0:01:31
+8%
Professional 
(cs00000int_0002)		10K.mrc00:19:060:20:07+0:01:01
+5%
School
(cs00000int_0003)		10K.mrc00:18:310:21:05+0:02:34
+14%
8College
(cs00000int_0001)		25K.mrc00:30:490:38:41+0:07:52
+25%
Professional 
(cs00000int_0002)		25K.mrc00:30:520:39:09+0:08:17
+27%
9College
(cs00000int_0001)		25K.mrc00:47:470:41:32-0:06:15
-13%
Professional 
(cs00000int_0002)		25K.mrc00:48:170:42:34-0:05:43
-11%
School
(cs00000int_0003)		25K.mrc00:47:540:16:27-0:31:27
-65%
10


College
(cs00000int_0001)		50K.mrcnot tested1:43:17
Professional 
(cs00000int_0002)		50K.mrcnot tested1:50:34
School
(cs00000int_0003)		50K.mrcnot tested1:59:29



Resource utilization for Test Set №1

...

#TOP 5 SQL statements
1
WITH deleted_rows AS (     delete from marc_indexers mi     where exists(         select 1         from marc_records_tracking mrt         where mrt.is_dirty = true           
and mrt.marc_id = mi.marc_id           and mrt.version > mi.version     )     returning mi.marc_id), deleted_rows2 AS (     delete from marc_indexers mi     
where exists(         select 1         from records_lb         where records_lb.id = mi.marc_id           and records_lb.state = 'OLD'     )     returning mi.marc_id) 
INSERT INTO marc_indexers_deleted_ids SELECT DISTINCT marc_id FROM deleted_rows UNION SELECT marc_id FROM deleted_rows2
2
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)
3
INSERT INTO cs00000int_mod_search.consortium_instance (tenant_id, instance_id, json, created_date, updated_date)
  VALUES ($1, $2, $3::json, $4, $5)
  ON CONFLICT (tenant_id, instance_id)
  DO UPDATE SET json = EXCLUDED.json, updated_date = EXCLUDED.updated_date
4
INSERT INTO cs00000int_0001_mod_source_record_manager.events_processed (handler_id, event_id) VALUES ($1, $2)
5
insert into "marc_records_lb" ("id", "content") values (cast($1 as uuid), cast($2 as jsonb)) on conflict ("id") do update set "content" = cast($3 as jsonb)

Errors

Data Import test for PTF - Updates Success - 2 profile with 25k records file on 3 tenants concurrently finished with errors for two records. 

...

#Data Import Log for Records
1
io.vertx.core.impl.NoStackTraceThrowable: Failed to update MARC record in SRS, instanceId: '830273a3-459b-4a81-945e-139719dd1b71', jobExecutionId: '4b55ca82-9fb1-4d42-b116-c3d3df75b312', status code: 504, 
Record: <html> 
<head><title>504 Gateway Time-out</title></head> 
<body> 
<center><h1>504 Gateway Time-out</h1></center> 
</body> 
</html>
2
io.vertx.core.impl.NoStackTraceThrowable: Failed to update MARC record in SRS, instanceId: '8d74d733-7c8e-40bf-974c-a2e19a01a389', jobExecutionId: '987db8ab-91e4-43ab-9c4f-8456da2466c5', status code: 422, 
Record: {"errors":[{"message":"must not be null","type":"1","code":"jakarta.validation.constraints.NotNull.message","parameters":[{"key":"matchedId","value":"null"}]}]}

Appendix

Infrastructure

...

  • 11 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]

  • 1 instance of db.r6.xlarge database instance: Writer instance

  • Number of records in DB:
    •  cs00000int_0001
      • instances - 6828236
      • items - 7859770
      • holdings - 7328737
    • cs00000int_0002
      • nstances - 1163315
      • items - 1754121
      • holdings - 1332559
    • cs00000int_0003
      • nstances - 1135806
      • items - 1735291
      • holdings - 1309387

  • OpenSearch

    • domain: fse

    • Data nodes Instance type: r6g.xlarge.search

    • Number of nodes: 9

    • Version: OpenSearch_2_7_R20240502

  • MSK - tenat

    • 4 kafka.m5.2xlarge brokers in 2 zones

    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true

    • log.retention.minutes=480

    • default.replication.factor=3

    • Kafka consolidated topics enabled

...

  • Test 1: Manually tested 10k records files DI MARC Bib Create started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
  • Test 2: Manually tested 10k records files DI MARC Bib Create started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.
  • Test 3: Manually tested 25k records files DI MARC Bib Create started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
  • Test 4: Manually tested 25k records files DI MARC Bib Create started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.
  • Test 5: Manually tested 150k records files DI MARC Bib Create started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.

...

  • Test 6: Manually tested 10k records files DI MARC Bib Update started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
  • Test 7: Manually tested 10k records files DI MARC Bib Update started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.
  • Test 8: Manually tested 25k records files DI MARC Bib Update started on College tenant(cs00000int_0001) and Professional tenant(cs00000int_0002) concurrently, step 30%.
  • Test 9: Manually tested 25k records files DI MARC Bib Update started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.
  • Test 10: Manually tested 150k records files DI MARC Bib Update started on College tenant(cs00000int_0001), Professional tenant(cs00000int_0002) and School tenant(cs00000int_0003) concurrently, step 30%.

...