[Ramsons] [ECS] Bulk Edit of MARC and FOLIO Instance records
Overview
This document contains the results of the testing Bulk Edit of MARC and FOLIO Instance records by submitting list of identifiers for 100, 1000, 10 000 records or triggering bulk edit by submitting the query. This time there're no tests with 100 000 records file and tests with queries (scenario 2, 4, 6 - https://folio-org.atlassian.net/browse/MODFQMMGR-667).
Tests carried out on Ramsons release with ECS Eureka like environment (central tenant) in scope of Eureka testing
The process of Bulk Edit in scenarios 1, 3, 5 includes two steps. First step - uploading file, second one - committing changes.
Summary
Bulk edit operations with instance Ids from csv file completed successfully in all scenarios for 100, 1000 and 10000 records. Scenario 5 lasts 3 times longer in average than scenario 1 (FOLIO) or scenario 3 (MARC).
Memory usage was stable during tests. No memory leaks revealed. Top usage observed from folio-keycloak module - 92% and mod-roles-keycloak - 67% in Bulk Edit with MARC records (scenario 3, 10000 records)
CPU was utilized in accordance with the load.
DB CPU utilized 99% of resources during each test.
DB connections were 1150 in average (scenario 1, scenario 5).
During scenario 3 (MARC records) with 1000 records deadlocks observed which affected 2 records to fail with timeout error message. Issue described in ticket https://folio-org.atlassian.net/browse/MSEARCH-961
Jiras
https://folio-org.atlassian.net/browse/MSEARCH-1004 high latency on DB queries
Test description
Test name | source | Number of records | Action | query-1 | query-2 |
|---|---|---|---|---|---|
Scenario 1 | FOLIO | 100, 1000, 10 000 |
| from .csv file | from .csv file |
Scenario 2 | FOLIO | 2007 |
| (instance.source == "FOLIO") AND (instance.format_names contains any (<format names supported in test environment>)) [computer -- online resource] | English (354) |
Scenario 3 | MARC | 100, 1000, 10 000 |
| from .csv file | from .csv file |
Scenario 4 | MARC | 1148 |
| (instance.source == "MARC") AND (instance.format_names contains any (<format names supported in test environment>)) [unmediated -- object] | Albanian (1055) |
Scenario 5 | FOLIO & MARC | 100, 1000, 10 000 |
| from .csv file | from .csv file |
Scenario 6 | FOLIO & MARC | 562 |
| (instance.format_names contains any (<format names supported in test environment>)) [audio -- other] | Georgian (1294) |
Test Results
Test # | Source | file size | Upload Duration (hh:mm:ss) | Update Duration (hh:mm:ss) | Total time | Results |
|---|---|---|---|---|---|---|
Scenario 1 | FOLIO | 100 | 00:00:12 | 00:00:14 | 00:00:26 | Completed |
FOLIO | 1000 | 00:01:18 | 00:02:28 | 00:03:46 | Completed | |
FOLIO | 10 000 | 00:09:38 | 00:24:13 | 00:33:51 | Completed | |
Scenario 2 | FOLIO | 2166 (query-1) 388 (query-2) |
|
|
| Not executed because of https://folio-org.atlassian.net/browse/MODFQMMGR-667 |
Scenario 3 | MARC | 100 | 00:00:11 | 00:00:28 | 00:00:39 | Completed |
MARC | 1000 | 00:01:37 | 00:01:26 | 00:03:03 | Completed* | |
MARC | 10 000 | 00:11:26 | 00:14:46 | 00:26:12 | Completed | |
Scenario 4 | MARC | 1055 (query-2) |
|
|
| Not executed because of https://folio-org.atlassian.net/browse/MODFQMMGR-667 |
Scenario 5 | FOLIO & MARC | 100 | 00:00:06 | 00:00:58 | 00:01:04 | Completed |
FOLIO & MARC | 1000 | 00:00:57 | 00:08:00 | 00:08:57 | Completed | |
FOLIO & MARC | 10 000 | 00:09:12 | 01:15:00 | 01:24:12 | Completed | |
Scenario 6 | FOLIO & MARC | 1294 (query-2) |
|
|
| Not executed because of https://folio-org.atlassian.net/browse/MODFQMMGR-667 |
* Timeout issue with 2 records observed during the test. Also deadlocks observed during the test (see Other RDS metrics and observations).
Resource utilization
Instance CPU Utilization
Resource utilization was very similar for all of the tests. I took the most representative.
Scenario 1, Scenario 5
