Data Import Create MARC Holdings Records [ECS] [Quesnelia]
- Leonid Kolesnykov
Overview
The purpose of the document is getting results of testing Data Import Create MARC holdings records and to detect performance trends in Quesnelia in scope of ticket PERF-855 - Getting issue details... STATUS
Compared with results in previous test report: Data Import Create MARC holdings records [Poppy]
Summary
- Data import create holdings job durations increased significantly in Quesnelia release. 4 times longer with 10k file. Failed to complete with 80k file because it was stopped after 4 hours of test run with only 46 committed jobs (total for the test was 81).
- Top CPU utilization: mod-inventory-b - 16%, nginx-okapi - 5%, mod-source-record-storage-b - 4%, mod-quick-marc-b - 7%. Such low resource utilization from modules side can be explained by DB queries huge average latency during INSERT and UPDATE processes which had lock on the same tuple.
- Top memory consumption: mod-inventory-storage-b - 85%, mod-data-import-b - 52%, mod-source-record-storage-b - 45%, mod-source-record-manager-b - 43%. Growing trend was defined in tests set #1 for mod-inventory-storage-b - 85%
- DI job duration for the same file size grew from test to test if to use the same instance HRID to create holdings
- DI perform faster if to use files with 1 unique instance HRID for every 1000 records. DI duration corresponds to file size with such approach. Memory utilized without growing trend. And module CPU and RDS CPU utilization increased because higher load.
Recommendations & Jiras
- Investigate growing trend for mod-inventory-storage in tests set #1 (using 1 instance HRID to create all Holdings)
- Define high number of Holdings associated with one instance HRID that's still realistic
- Consider limit the request /inventory/items-by-holdings-id with limit. Now limit=0. MODINVSTOR-1229 - Getting issue details... STATUS
Errors
- error status for 32'd split job during 80k file importing- SNAPSHOT_UPDATE_ERROR
Log message:
ERROR taImportKafkaHandler org.folio.inventory.dataimport.exceptions.CacheLoadingException: Error loading jobProfileSnapshot by id: 'aee287c2-0d40-4e8d-9879-4c1c54bcd819', status code: 503
Test Runs
Profile used for testing - Default - Create Holdings and SRS MARC Holdings
| Set of tests № | Scenario | Test Conditions | Status |
|---|---|---|---|
| 1 | DI Holdings Create (previous* approach) | 1K, 5K, 10K, 80K sequentially | 1k, 5k, 10k - Completed 80k - Failed |
| 2 | DI Holdings Create (new** approach) | 1K, 5K, 10K, 80K sequentially | Completed |
*previous approach - Data import Holdings with mrc file where 1 instance HRID is associated to all holdings (1k, 5k, 10k, 80k)
**new approach - Data import Holdings with mrc file where 1 instance HRID is associated to 1000 holdings
Test Results
Set 1 - Files used to test DI create Holdings had 1 instance HRID for all created Holdings
Set 2 - Files used to test DI create Holdings had 1 unique instance HRID for every 1000 created Holdings (new approach)
Test | File | Duration: Orchid (previous results) | Duration: Poppy (previous results) | Duration: Quesnelia [ECS] Set #1 | Status and Errors Quesnelia [ECS] Set #1 | Duration: Quesnelia [ECS] Set #2 | Status and Errors Quesnelia [ECS] Set #2 |
|---|---|---|---|---|---|---|---|
| 1 | 1k | 45s | 32s | 1 min 22 sec | Success | 1 min 3 sec | Success |
| 2 | 5k | 7m 47s | 2m 14s | 8 min | Success | 4 min 16 sec | Success |
| 3 | 10k | 19m 46s | 4m 35s | 22 min 40 sec | Success | 8 min 59 sec | Success |
| 4 | 80k | 20m (error*) | 36m 25s | 4 hours 13 min | Stopped by user after 46 job COMMITTED from 81 - 56% finished 1 job status - ERROR, with error status - SNAPSHOT_UPDATE_ERROR (job number - 32, file_name = '1718290065265-80k_holdings_Create_32.mrc') | 52 min 5 sec | Success |
Previous test report: Data Import Create MARC holdings records [Poppy]
Service CPU Utilization
Set #1: mod-inventory-b - 16%, nginx-okapi - 5%, mod-source-record-storage-b - 4%, mod-quick-marc-b - 7%
Set #1
Set #2
Set #2: mod-inventory-b - 33%, nginx-okapi - 23%, mod-source-record-storage-b - 11%, mod-quick-marc-b - 7%
Memory Utilization
Set #1
Set #2