Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device.
Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
/
PTF - Performance testing of Export All Endpoint (Quesnelia) [non-ECS]
This document contains results of testing Data Export All by the endpoint (POST data-export/export-all) with Default instances, authority, export holdings job profiles and with the custom profileExample 1 on the Quesnelia [non-ECS] release on qcp1 environment.
Data Export tests finished successfully on qcp1 environment using the Default instances, authority, holdings export job profileand Custom Mapping Profiles - Example 1.
During the tests data-export-all with Custom Mapping Profiles - Example 1 we observe Errors converting json to marc for instances with 9 and more holdings, task for investigation was created https://folio-org.atlassian.net/browse/MDEXP-780
Two very slow queries when checking deleted records in tables mod_inventory_storage.audit_instance and mod_inventory_storage.audit_holdings_record for Data-Export-All, task for investigation was created https://folio-org.atlassian.net/browse/MODINVSTOR-1234
No memory leaks are observed.
The data export duration for default profiles and the duration for the concurrent test of default profiles plus the export of 10,000 instances with a custom profile are the same. This is because Data-Export-All uses one data-export module task, so if we run concurrent tests, each job will be executed on a separate data-export module task, because data-export module has two tasks.
Here we can see that mod-data-export used 150% CPU.
Service Memory Utilization
Here we can see that data-export module used 95% memory.
Kafkametrics
DB CPU Utilization
DB CPU was 90%.
DB Connections
Max number of DB connections was 750
DB load
Top SQL-queries
#
TOP 5 SQL statements
#
TOP 5 SQL statements
1
SELECT id, content, external_id, record_type, state, leader_record_status, suppress_discovery FROM v_authority_all WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC offset $3 rows fetch next $4 rows only
2
select ie1_0.id,ie1_0.jsonb from v_instance ie1_0 where ie1_0.id in ($1...)
3
SELECT * FROM v_marc_instance_all_non_deleted_non_suppressed WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC offset $3 rows fetch next $4 rows only
4
SELECT * FROM v_folio_instance_all_non_deleted_non_suppressed WHERE id BETWEEN $1 AND $2 ORDER BY id ASC fetch first $3 rows only
5
SELECT * FROM v_marc_instance_all_non_deleted_non_suppressed WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC fetch first $3 rows only
Here we can see that mod-data-export used 194% CPU.
Service Memory Utilization
Here we can see that all modules show a stable trend except mod-source-record-manager.
Kafkametrics
DB CPU Utilization
DB CPU was 92%.
DB Connections
Max number of DB connections was 762.
DB load
Top SQL-queries
#
TOP 5 SQL statements
#
TOP 5 SQL statements
1
SELECT id, content, external_id, record_type, state, leader_record_status, suppress_discovery FROM v_authority_all WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC offset $3 rows fetch next $4 rows only
2
select ie1_0.id,ie1_0.jsonb from v_instance ie1_0 where ie1_0.id in ($1...)
3
SELECT * FROM v_marc_instance_all_non_deleted_non_suppressed WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC offset $3 rows fetch next $4 rows only
4
SELECT * FROM v_folio_instance_all_non_deleted_non_suppressed WHERE id BETWEEN $1 AND $2 ORDER BY id ASC fetch first $3 rows only
5
SELECT * FROM v_marc_instance_all_non_deleted_non_suppressed WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC fetch first $3 rows only
Here we can see that mod-data-export used 145% CPU.
Service Memory Utilization
Here we can see that mod-data-export used 93% memory .
Kafkametrics
DB CPU Utilization
DB CPU was 82%.
DB Connections
Max number of DB connections was 730.
DB load
Top SQL-queries
#
TOP 5 SQL statements
#
TOP 5 SQL statements
1
select ie1_0.id,ie1_0.holdings_record_id,ie1_0.jsonb from v_item ie1_0 where ie1_0.holdings_record_id in ($1)
2
select iwhe1_0.id,iwhe1_0.hrid from v_instance_hrid iwhe1_0 where iwhe1_0.id in ($1,$2...)
3
SELECT * FROM v_marc_instance_all_non_deleted_non_suppressed WHERE external_id BETWEEN $1 AND $2 ORDER BY id ASC offset $3 rows fetch next $4 rows only
4
select hre1_0.id,hre1_0.instance_id,hre1_0.jsonb from v_holdings_record hre1_0 where hre1_0.instance_id=$1
5
SELECT * FROM v_folio_instance_all_non_deleted_non_suppressed WHERE id BETWEEN $1 AND $2 ORDER BY id ASC fetch first $3 rows only
Test 1: Manually tested 10k instances records files with the custom profile Example 1 and concurrently run Export All instances started by JMeter using endpoint(POST data-export/export-all) with profile Default instances export job profile. Data Export started on one tenant (fs09000000) with two tasks for data-export module.
Test 2: Manually tested 10k instances records files with the custom profile Example 1 and concurrently run Export All instances started by JMeter using endpoint(POST data-export/export-all) with profile Default holdings export job profile. Data Export started on one tenant (fs09000000) with two tasks for data-export module.
Test 3: Manually tested 10k instances records files with the custom profile Example 1 and concurrently run Export All instances started by JMeter using endpoint(POST data-export/export-all) with profile Default authority export job profile. Data Export started on one tenant (fs09000000) with two tasks for data-export module.
Test set:
Test 4: Export All instances started by JMeter using endpoint(POST data-export/export-all) with profile Default instances export job profile. Data Export started on one tenant (fs09000000) with one task for data-export module.
Test 5: Export All holdings started by JMeter using endpoint(POST data-export/export-all) with profile Default holdings export job profile. Data Export started on one tenant (fs09000000) with one task for data-export module.
Test 6: Export All authority started by JMeter using endpoint(POST data-export/export-all) with profile Default authority export job profile. Data Export started on one tenant (fs09000000) with one task for data-export module.
Test №7:
Export All instances started by JMeter using endpoint(POST data-export/export-all) with the Custom ProfileExample 1. Data Export started on one tenant (fs09000000) with one task for data-export module.
To get status and time range for import jobs the query used:
SQL Query
select jsonb->'exportedFiles'->0->>'fileName' as fileName, job_profile_name,exported, started_date,completed_date, completed_date - started_date as duration ,status
from fs09000000_mod_data_export.job_executions where started_date > '2024-07-8' order by started_date desc limit 1000;