Overview

...

DE jobs perform dramatically better in Quesnelia release if to compare with Poppy. No issues with token. All jobs with file's volume 1000, 100k, 500k records completed successfully.
The improvement varies from file size or job profile - from 4 to 9 times better duration. Additional test was conducted with job profile prepared by script to check consistency of results.
The Average CPU utilization for mod-data-export depends on file size and job profile. Exporting 100k records- 63% in Default and 92% in custom job profile. Exporting 500k - Default instances export job profile- 434%, srs - holdings and items- 296%
Average Memory consumption for mod-data-export was close to 100%. Almost the same as in Poppy release.
Average DB utilization - 17% with 100k and 33% with 500k. DB connections - 1360 instead of 200 in Poppy.

...

Profile	CSV File	DE Duration/Status Poppy 2 set		DE Duration/Status Quesnelia		DE Duration, DELTA Poppy/Quesnelia
		Result	Status	Result	Status	hh:mm:ss
DE MARC Bib (Default instances export job profile)	1kDE.csv	00:00:08	COMPLETED	00:00:02	COMPLETED	00:00:06 - 4 times improvement
	100kDE.csv	00:15:36	COMPLETED	00:02:17	COMPLETED	00:13:19 - 7 times improvement
	500kDE.csv	00:57:25	FAIL	00:05:10	COMPLETED
DE MARC Bib (srs - holdings and items)	1kDE.csv	00:00:29	COMPLETED	00:00:04	COMPLETED	00:00:25 - 7 times improvement
	100kDE.csv	00:47:23	COMPLETED	00:05:13	COMPLETED	00:42:10 - 9 times improvement
	500kDE.csv	04:11:09	FAIL	00:08:58	COMPLETED

...

Service CPU Utilization

Expand

title	CPU utilization

Default instances export job profile with 500k file

srs - holdings and items with 500k file

TOP 20 modules

...

3 files were prepared with query: SELECT id FROM [tenant_id]_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 1000|100000|500000;

All tests were carried out sequentially with each job profile on main tenant fs09000000.

To get status and time range for export jobs the query used:

...

Code Block

select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date


FROM [tenant]_mod_data_export.job_executions


where job_profile_name = 'srs - holdings and items'


ORDER BY completed_date DESC



select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date


FROM [tenant]_mod_data_export.job_executions


where job_profile_name = 'Default instances export job profile'


ORDER BY completed_date DESC



select exported as filesize,completed_date - started_date as duration,job_profile_name,status as status,started_date,completed_date


FROM [tenant]_mod_data_export.job_executions


where job_profile_name = 'Export for Data Import updates(test)1'


ORDER BY completed_date DESC

PTF -environment qcp1

...

Name	Memory GIB	vCPUs	max_connections
db.r6g.xlarge	32 GiB	4 vCPUs	2731