...

• Data Export tests finished successfully on qcon environment using the profiles Default instances export job profile and srs - holdings and items job profile.
• Data Export test were run on College and Central tenants, but results for comparing between environment releases were taken from College tenant.
• Comparing with previous testing results Poppy and Quesnelia releases
  • Data Export processed all files including file with 500k records without errors for Quesnelia releases.
  • Data Export durations improved  - 80% in Average for Quesnelia releases.
• During testing, we noticed spikes in the mod-data-export up to 452% 593% CPU.
• No memory leak is suspected for Data Export modules.

Test Results

This table contains durations for Data Export with 2 job profiles. 

...

Profile

...

•  
• For Test №5 Data Export started on College tenant(cs00000int_0001)

...

• , Central

...

• tenant(cs00000int)

...

• and Professional tenant(cs00000int_0002) concurrently using the Default instances export job profile

...

• , we observed that the CPU usage of the mod-data-export module was initially at 44% before the test began then it spiked to 109% during the test and remained elevated without returning to the initial state. 

Test Results

This table contains durations for Data Export with 2 job profiles. 

...

select mre1_0.id,mre1_0.content,mre1_0.external_id,mre1_0.leader_record_status,mre1_0.record_type,mre1_0.state,mre1_0.suppress_discovery from v_marc_records_lb mre1_0 where mre1_0.external_id in ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$

INSERT INTO job_executions_export_ids (job_execution_id, instance_id) VALUES ($1, $2) ON CONFLICT DO NOTHING

select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)

select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1

select eie1_0.id,eie1_0.instance_id,eie1_0.job_execution_id from job_executions_export_ids eie1_0 where eie1_0.job_execution_id=$1 and eie1_0.instance_id>=$2 and eie1_0.instance_id<=$3 order by eie1_0.instance_id offset $4 rows fetch first $5 rows only

...

only

Service CPU Utilization

Here we can see that mod-data-export used 336% 593% CPU in spike.

Image Added

Service Memory Utilization

Here we can see that all modules show a stable trend.

We observed that the CPU usage of the mod-data-export module was initially at 44% before the test began. It spiked to 109% during the test and remained elevated without returning to the initial state.

Image Added

DB CPU Utilization

DB CPU was 35%50%.

Image Added

DB Connections

DB connections was 13771368.

Image Added

DB load

    Image Added                                                                                                                    

Top SQL-queries

Image Added

Appendix

Infrastructure

PTF - environment Quesnelia (qcon)

...

• Test 1: Manually tested 1k, 100k and 500k records files Data Export started on College tenant(cs00000int_0001) only using Default instances export job profile.
• Test 2: Manually tested 1k, 100k and 500k records files Data Export started on College tenant(cs00000int_0001) only using srs - holdings and items job profile.
• Test 3: Manually tested 1k, 100k and 500k records files Data Export started on Central tenant(cs00000int) only using Default instances export job profile.
• Test 4: Manually tested 1k, 100k and 500k records files Data Export started on Central tenant(cs00000int) only using srs - holdings and items job profile.
• Test 5: Manually tested 500k records file Data Export started on College tenant(cs00000int_0001), Central tenant(cs00000int) and Professional tenant(cs00000int_0002) concurrently using Default instances export job profile. 

To get status and time range for export jobs the query used: 

...