Test status: PASSED

...

• Regression testing of Check-In/Check-Out (CI/CO) fixed load tests on okapi based environment in quesnelia release.
  • The flows triggered by the command from AWS instance load generator. All artefacts needed to run tests tested on local machine first.
  • There're minor changes in script to process CI/CO and handle token expiration.
• Testing includes data preparation step and testing itself
  • Data preparation for each test takes up to 20 minutes and consists of truncating involved in testing tables, populating data and updating statuses of items.
  • Test itself depends on duration and virtual users number creating necessary load.
• The purposes of CI/CO testing:
  • To define response times of transaction controllers such as Check-In Controller and Check-Out Controller
  • To define response times of requests that last more than 100 ms
  • To find any trends for resource utilization and recommend improvements
  • Compare results (current and previous)

• Jiras/ links:

  • The previous results report:
    • Check-in-check-out Test Report (Quesnelia)
  • The current ticket: PERF-970 - [Ramsons] [non-ECS] CI/CO In Review

Summary

• Common results:
  • CI/CO tests showed stable response times across low loads, a 50% increase under high load (75 vUsers), moderate degradation during longevity test, no memory leaks and up to 14% (80 ms) degradation in CO and up to 20% (100 ms) in CI 20 vUsers flow compared to Quesnelia release.

...

• RDS CPU utilization average
  • 8 vUsers - 13%, 20 vUsers - 22%, 30 vUsers - 30%, 75 vUsers - 63% During longevity test CPU grew from 30% to 45%. So it has growing trend during longevity test that can be explained by absent dcb-system-user in mod-dcb module.

• CPU (User) usage by broker
  • Common CPU utilization by broker during all tests was 15% with equal distribution between brokers

Common notes

Recommendations

...

• Error messages: POST_circulation/check-out-by-barcode (Submit_barcode_checkout)_POST_422. 422/Unprocessable Entity. Happen expectedly if instance was checked out already. Error rate - 0.002% which is acceptable.

Response time

The table contains results of Check-in, Check-out tests in Ramsons release. 

...

Resources Utilization

...

45 minute tests

Longevity test

RDS Database Connections

For 45 minute and longevity tests RDS used max 885-920 connections. Without test it was 860 connections.

45 minute tests

Longevity test

CPU (User) usage by broker

Common CPU utilization by broker during all tests was 15% with equal distribution between brokers

45 minute tests

Longevity test

As MSK cluster is linked to all PTF clusters so the time range which can reflect only CI/CO - midnight till 7 a.m. Max consumption rate - 10%. Also we may observe impact of CI/CO - the max consumption rate - 40% for all clusters.

...

45 minutes tests
UPDATE fs09000000_mod_inventory_storage.item SET jsonb=$1 WHERE id=$2 RETURNING jsonb::text
INSERT INTO fs09000000_mod_pubsub.audit_message (id, event_id, event_type, tenant_id, audit_date, state, published_by, correlation_id, created_by, error_message) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
WITH deleted_rows AS ( delete from marc_indexers mi where exists( select ? from marc_records_tracking mrt where mrt.is_dirty = ? and mrt.marc_id = mi.marc_id and mrt.version > mi.version ) returning mi.marc_id), deleted_rows2 AS ( delete from marc_indexers mi where exists( select ? from records_lb where records_lb.id = mi.marc_id and records_lb.state = ? ) returning mi.marc_id) INSERT IN
SELECT fs09000000_mod_inventory_storage.count_estimate('SELECT * FROM fs09000000_mod_inventory_storage.material_type WHERE id=''025ba2c5-5e96-4667-a677-8186463aee69''')
UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
INSERT INTO fs09000000_mod_authtoken.refresh_tokens (id, user_id, is_revoked, expires_at) VALUES ($1, $2, $3, $4)
SELECT upsert('circulation_logs', $1::uuid, $2::jsonb)

Longevity
INSERT INTO fs09000000_mod_pubsub.audit_message (id, event_id, event_type, tenant_id, audit_date, state, published_by, correlation_id, created_by, error_message) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
SELECT fs09000000_mod_patron_blocks.count_estimate('SELECT jsonb FROM fs09000000_mod_patron_blocks.patron_block_limits WHERE (jsonb->>''patronGroupId'') = ''5fc96cbd-a860-42a7-8d2b-72af30206712''')
UPDATE fs09000000_mod_inventory_storage.item SET jsonb=$1 WHERE id=$2 RETURNING jsonb::text
SELECT jsonb FROM fs09000000_mod_patron_blocks.user_summary WHERE (jsonb->>'userId') = '4cd01954-62da-46c5-8558-ebd222bc48eb'
SELECT fs09000000_mod_inventory_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_inventory_storage.service_point WHERE id=''7068e104-aa14-4f30-a8bf-71f71cc15e07''')
UPDATE fs09000000_mod_login.auth_attempts SET jsonb = $1::jsonb WHERE id='9883ca16-ef27-41f7-81d7-6693b79cddad'
INSERT INTO fs09000000_mod_authtoken.refresh_tokens (id, user_id, is_revoked, expires_at) VALUES ($1, $2, $3, $4)
SELECT upsert('circulation_logs', $1::uuid, $2::jsonb)
SELECT COUNT(*) FROM fs09000000_mod_users.users
SELECT fs09000000_mod_circulation_storage.count_estimate('SELECT jsonb,id FROM fs09000000_mod_circulation_storage.loan_policy WHERE id=''2be97fb5-eb89-46b3-a8b4-776cea57a99e''')

45 minute tests

Longevity test

Appendix

Infrastructure

...

Update revision in source-record-storage module to exclude every 30 minutes SQL statements - delete rows in marc_indexers (mi)  WITH deleted_rows

{
"name": "srs.marcIndexers.delete.interval.seconds",
"value": "86400"
},

Update mod-serials module. Set number of task with 0 to exclude significant database connection growth.

Usual PTF CI/CO data preparation script won’t work in Ramsons. To solve that disable trigger updatecompleteupdateddate_item_insert_update before data preparation for the tenant and enable it before test start.

...

• If the command executed from local machine you may encounter with too long query error message. To solve it use PGAdmin to run 2 long queries UPDATE ${TENANT}_mod_inventory_storage.item SET jsonb = jsonb_set(jsonb, '{status, name}', '\"Checked out\"') where id IN.
• Other possible issue - incorrect encoding (on Windows machine). To solve it just add ENCODING 'UTF8'
  • Use pattern: copy ${TENANT}_mod_circulation_storage.loan(id, jsonb) FROM '${LOANS}' DELIMITER E'\t'  ENCODING 'UTF8'

In Ramsons token expiration set to 10 minutes by default so to run any tests use new login implementation from the script. Pay attention to Backend Listener. Replace value of application to make the results visible in Grafana dashboard.

...