Overview

• This document contains the results of testing Check-in/Check-out and Data Import for MARC Bibliographic records on the Ramsons[ECS] release environment. 

PERF-978 - Getting issue details... STATUS  

Summary

• Data Import (DI) with Check In/Check Out tests finished successfully with PTF - Create 2 and PTF - Updates Success - 6 job profiles with files 5K, 10K, 25K, 50K, 100K records.
• Comparison results:
  
  • DI create jobs duration with CI/CO degraded 25% in average. There's degradation without CI/CO 22% with 10k and 8% with 25k but there's improvement 14% with 50k and 17% with 100k. 
  • DI update jobs durations with CI/CO degraded 20% for 25k, 50k, 100k and 50% for 5k, 10k. There's degradation without CI/CO 20% with 10k, 3% with 50k, 12% with 100k but there's 4% improvement with 25k.
  • DI with CI/CO degraded compared to DI without CI/CO.  DI create jobs - 70%, DI update jobs - 30% in average. 
  • CI/CO response times degraded with DI 20% in average.
• DI create job with 100k file finished successfully but total duration in database was updated in 6 hours after it was finished. Additional 100k finished in 56 minutes (total duration on db side).
• Memory showed some memory growing trend for mod-pubsub and it's possible that a memory leak has been detected on a module side.

Recommendations & Jiras

• The previous results report:
  • PTF - Data Import MARC BIB + CI/CO (Quesnelia) [ECS]

• The mod-pubsub module memory leak investigation ticket: MODPUBSUB-311 - Getting issue details... STATUS
• New behaviour from the mod_orders_storage.po_line table detected during data import create jobs. Consider also [tenant]_mod_orders_storage.internal_lock query. It should be investigated.

Test Runs 

Test Results

This table contains durations for Data Import. 

Check-in/Check-out without DI

Comparison

This table contains DI durations with CICO comparison between Quesnelia and Ramsons releases.

Test №1

Response times

Cluster resource utilization

Service CPU Utilization

Service Memory Utilization

DB resources utilization

RDS CPU Utilizatoin

DB Connections

SELECT jsonb FROM cs00000int_mod_orders_storage.po_line WHERE lower(f_unaccent(po_line.jsonb->>'locations')) LIKE lower(f_unaccent('%69a16007-7c30-44e9-94ed-a0e28e0a6310%')) LIMIT 2147483647 OFFSET 0

SELECT * FROM cs00000int_0001_mod_orders_storage.internal_lock WHERE lock_name = $1 FOR UPDATE

Test №2

Response times

Cluster resource utilization

Service CPU Utilization

Service Memory Utilization

DB resources utilization

RDS CPU Utilization

DB Connections

DB load

Top SQL-queries

MSK resources utilization

CPU (User) usage by broker

Maximum utilization was 60% with 50k file 

Appendix

Methodology/Approach

DI tests scenario (DI MARC Bib Create and Update) were started from UI with delay.

Test runs:

• Test 1: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Create ) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.
• Test 2: Manually tested 5K, 10K, 25K, 50K, 100K consequentially (with 5 min pause) records files, DI (DI MARC Bib Update) started on College tenant(cs00000int_0001) only, and CICO with 8 users on background.

Description

Testing includes data preparation step for Check-in/Check-out with 8 virtual users, 3 hours duration (as a rule it's enough for series of DI create / update jobs). Any convenient duration may be applied.

• Data preparation of Check-in/Check-out for DI tests takes up to 20 minutes and consists of truncating involved in testing tables, populating data and updating statuses of items.
• Test itself depends on duration and virtual users number creating necessary load.

In Ramsons token expiration set to 10 minutes by default so to run any tests use new login implementation from the script. Pay attention to Backend Listener. Replace value of application parameter to make the results visible in Grafana dashboard.

Module configuration recommended setup

Update revision in source-record-storage module to exclude every 30 minutes SQL statements - delete rows in marc_indexers (mi)  WITH deleted_rows

Update mod-serials module. Set number of task with 0 to exclude significant database connection growth.

DB trigger setup in Ramsons

Usual PTF CI/CO data preparation script won’t work in Ramsons. To solve that disable trigger updatecompleteupdateddate_item_insert_update before data preparation for the tenant and enable it before test start.

The sql file was updated to do that step from the script.

Data preparation

• To prepare data establish connection by AWS key
• Run CICO_db_preparation.sh script located in /scripts folder. Before use the file tenats.csv to edit the list of tenants to restore the database.
• Files location: Buckets/fse-ptf/Scripts/CICO/Ramsons/

To start test from AWS instance (load generator) use template for the command. Test locally before start.

To create widgets in AWS dashboard to monitor and collect CI/CO related modules parameters (service CPU and Memory) use these json: