...
Ticket:
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Summary
Total duration ofThe total duration:
DE_instances T1 for: 500 records is473 seconds; 1000 records is922 seconds; and 2000 is 1724 seconds
DE_instances T2(Max)
...
about 23 hours.
...
DE_holdings T3 for: 500 records is445 seconds; 1000 records is508 seconds; and 2000 is 842 seconds
DE_holdings T4(Max)
...
about 8 hours and 15 minutes.
Data-export duration statistics:
Data export | Instances | Holdings | ||||
500 records, seconds | 1000 records, seconds | 2000 records, seconds | 500 records, seconds | 1000 records, seconds | 2000 records, seconds | |
Average duration | 83 | 426 | 1184 | 10 | 87 | 405 |
Minimal duration | 12 | 32 | 1372 | 5 | 16 | 32 |
Maximal duration | 131 | 571 | 127 | 29 | 127 | 504 |
75t percentile | 114 | 539 | 1350 | 12 | 113 | 490 |
90t percentile | 126 | 553 | 1362 | 19 | 119 | 500 |
DE_instances T1(500-1K-2k). The duration of instance date-export for 500 recordings increased from 12 seconds for the first tenant to about 100 for the 33rd tenant, then the processing time was approximately unity from 100 to 120 for tenants 33 to 62. The situation was similar for export dates of 1000 and 2000 records.
DE_holdings T3(500-1K-2k) For 2000 records Holdings data export duration increased from 32 seconds for the first tenant to about 420 sec for the 22nd tenant, then the processing time was approximately from 420 to 470 sec for tenants 23 - 61.
Recommendations & Jiras
Restart the tests DE_instances T2(Max) and DE_holdings T4(Max) with different POOL_SIZE, and BATCH_SIZE parameters to check Data-export duration improvement. [MDEXP-688] Make configurable params for Data-Export(POOL_SIZE, BATCH_SIZE) - FOLIO Jira (atlassian.net)
Restart the tests DE_authority T5(500-1K-2k) to check how consolidated Kafka topics affects Authority Data-export [MODELINKS-213] Data-import process creating incorrect topics in Kafka - FOLIO Jira (atlassian.net)
Test Results
DE_instances T1(500-1K-2k). This table contains durations for Data-export for jobs with “Default instances export profile” on 61 tenants with 3 sets of data(500 records; 1000 records; 2000 records).
...