Check-Out by users with 2000 loans each (Nolana)
Overview
The test goal is to assess the performance of circulation Check-Out functionality for users with 2000 loans each.
Ticket: 
Summary
Load tests showed that there is performance degradation for 1 - 8 concurrent users tests users with 2000 loans each compared to those without loans.
For 20 concurrent users time of CO is almost the same for both but the degradation of API response time GET_circulation/loans (Submit_barcode_checkout) and GET_circulation/loans (Submit_patron_barcode) is about 5.6%.
Test Runs
Test # | Test Conditions | Duration | Load generator size (recommended) | Load generator Memory(GiB) (recommended) | Notes
|
|---|---|---|---|---|---|
1. | Checkout with 1, 5, 8, and 20 users | 30 min | t3.medium | 3 | ncp3 environment |
Results
Response Times
Baseline (users without loans)
Verification (users with 2000 loans)
At the end of 5 users' test, the response time of CO was 20 s
Checkout response time comparison
Users quantity | Response time 95prc, sec |
Degradation, sec |
Degradation, % | Response time Avg, sec |
Degradation, sec |
Degradation, % | ||
|---|---|---|---|---|---|---|---|---|
Baseline (users without loans) | Verification (users with 2000 loans each) | Baseline (users without loans) | Verification (users with 2000 loans each) | |||||
1 user | 1.146 | 1.269 | 0.123 s | 10.73% | 0.950 | 1.015 | 0.065 s | 6.84% |
5 users | 1.004 | 1.071 | 0.067 s | 6.67% | 0.852 | 0.908 | 0.056 s | 6.57% |
8 users | 0.946 | 0.995 | 0.049 s | 5.18% | 0.837 | 0.884 | 0.047 s | 5.62% |
20 users | 0.932 | 0.928 | -0.004 s | -0.43% | 0.850 | 0.850 | 0 | 0.00% |
API with degraded response time 95prc, all tests degradation
| Degradation, % | |||
API | 1 user | 5 users | 8 users | 20 users |
GET_circulation/loans (Submit_barcode_checkout) | 5.61% | 18.62% | 8.70% | 5.67% |
GET_circulation/loans (Submit_patron_barcode) | 10.16% | 18.00% | 9.35% | 5.67% |
Instance CPU Utilization
Baseline (users without loans)
Verification (users with 2000 loans)
Service CPU Utilization
Baseline (users without loans)
Verification (users with 2000 loans)
Memory Utilization
Baseline (users without loans)
mod-circulation memory utilization increases from 71% to 73%
Verification (users with 2000 loans)
mod-circulation and mod-circulation-storage were restarted before the tests as you can see at the lowest memory utilization percent and then grow reaching baseline numbers.
RDS CPU Utilization
Baseline (users without loans)
Verification (users with 2000 loans)
RDS DB connections
Baseline (users without loans)
Verification (users with 2000 loans)
Database Load
Baseline (users without loans)
Verification (users with 2000 loans)
Errors from modules logs
13:20:03 [970766/circulation;909245/request-storage] [fs09000000] [9eb67301-6f6e-468f-9b1a-6134dc39a684] [mod_circulation_storage] ERROR PostgresClient Unrecognized field "title" (class org.folio.rest.jaxrs.model.Item), not marked as ignorable (one known property: "barcode"])
13:24:41 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR ? The timeout period of 20000ms has been exceeded while executing GET /circulation/rules/request-policy?item_type_id=025ba2c5-5e96-4667-a677-8186463aee69&loan_type_id=33f7f451-0923-4442-9191-0122c4042388&patron_type_id=5fc96cbd-a860-42a7-8d2b-72af30206712&location_id=38baf4b3-4fe7-47c1-826b-5d35e7b41018 for server pvt.lb.ncp3.folio-eis.us-east-1:9130
Appendix
Infrastructure
PTF -environment ncp3
10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3 [ kafka configurations]
4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
Modules memory and CPU parameters:
Modules | Version | Task Definition | Running Tasks | CPU | Memory (Soft/Hard limits) | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|
okapi | 4.14.7 | 1 | 3 | 1024 | 1440/1684 | 512 | 922 |
mod-feesfines | 18.1.1 | 3 | 2 | 128 | 896/1024 | 128 | 768 |
mod-patron-blocks | 1.7.1 | 4 | 2 | 1024 | 896/1024 | 128 | 768 |
mod-pubsub | 2.7.0 | 4 | 2 | 1024 | 1440/1536 | 512 | 922 |
mod-authtoken | 2.12.0 | 3 | 2 | 512 | 1152/1440 | 128 | 922 |
mod-circulation-storage | 15.0.2 | 3 | 2 | 1024 | 1440/1536 | 512 | 896 |
mod-circulation | 23.3.2 | 3 | 2 | 1024 | 896/1024 | 128 | 768 |
mod-configuration | 5.9.0 | 3 | 2 | 128 | 896/1024 | 128 | 768 |
mod-users | 19.0.0 | 4 | 2 | 128 | 896/1024 | 128 | 768 |
mod-remote-storage | 1.7.1 | 3 | 2 | 128 | 1692/1872 | 512 | 1178 |
Methodology/Approach
Conduct necessary commands to return the database to the initial state. Do this before each test run. Wait several minutes before the test start.
Conduct check-out load tests with different numbers of users.
Set a delay of all circulation_ and circulation-storage_ okapi timers to 0 (to avoid background jobs starting). Repeat tests with the same approach but before each test also generates 2000 loans for each user by running the command ./high_circ-data-load.sh psql_[environment name].conf [tenant].
Compare test results.
Grafana dashboard
Verification (users with 2000 loans): http://carrier-io.int.folio.ebsco.com/grafana/d/elIt9zCnz/jmeter-performance-test-copy?orgId=1&from=1675857341711&to=1675878170222&var-percentile=95&var-test_type=baseline&var-test=circulation_CheckOut_nolana&var-env=int&var-grouping=1s&var-low_limit=250&var-high_limit=750&var-db_name=jmeter&var-sampler_type=All
API response comparison: https://docs.google.com/spreadsheets/d/1560SZ7vb9aNip0sBsUesXbaKhn0l5HzDd7_Ts8TxJGQ/edit#gid=1126827422
Please note that dashboards will expire in 6 weeks after test runs.