POST /inventory-hierarchy/items-and-holdings report
In Progress
In Progress
In Progress
In Progress
In Progress
In Progress
Goals:
Define overall performance of POST /inventory-hierarchy/items-and-holdings;
Find bottlenecks if possible;
Measure KPI's for different number of instances included in request;
Measure KPI's for different number of holdings attached per instance ID;
Overview
Mod-inventory-storage has a new API to gather all related instances' data, i.e., items and holdings, in one call in order to eliminate making 3 separate calls to get the data. The API also allows passing in x number of instance IDs to lookup. Since teams are now starting to use this API, it's important to understand the stability of the API with respect to the module's CPU and memory usage, and also the database usage.
Summary
System shows stable work and quick responses in all test
No memory leaks found
There is a link between most used modules (see CPU) and test type. In case if it's extremely fast responses (for example for 5 holdings per instance and one user) - most used module will be mod-authtoken, otherwise (see test 4) that will be mod-inventory-storage. Same for increasing number of instances per request.
DB CPU usage reaches maximum in 8-users test
Fast response for most high load test (test 4). To extract an instance having 300+ holdings it takes 373 ms for 75th percentile. Therefore getting one holding with this endpoint takes 1.2 ms.
No bottlenecks found.
Version:
"okapi-3.1.2";
"mod-inventory-storage-19.4.0-SNAPSHOT.448";
"mod-authtoken-2.5.1".
RDS Data amount:
Precondition, if database free memory is less than 2000 MB, DB instance should be restarted.
Holdings per instance | Number of instances |
|---|---|
>0 | 7183552 |
>1 | 407467 |
>2 | 69890 |
>3 | 22241 |
>4 | 10201 |
>5 | 5635 |
>10 | 1060 |
>15 | 516 |
>20 | 331 |
>25 | 240 |
>40 | 115 |
>100 | 23 |
>300 | 7 |
Test Runs:
Test | Virtual Users | Duration (min) | Instance per request | Holdings per instance |
|---|---|---|---|---|
1 | 1 | 30 | 1 | 5 |
2 | 2 | 30 | 1 | 5 |
3 | 1 | 30 | 1 (adding instance to request each loop) max - 964 | 5 |
4 | 1 | 30 | 1 | 300 |
5 | 1 | 30 | 50 | >1 |
6 | 8 | 30 | 50 | >1 |
7 | 1 | 90 | (adding 5 instance to request each loop) max - 5325 | >1 |
Results
Summary:
During test set we've measured general performance of new endpoint.
Test 1: 1 user, 1 instance
Requests | Total | OK | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct |
|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 36530 | 36530 | 20.294 | 0.033 | 0.040 | 0.043 | 0.057 | 0.101 |
Resource usage
Most used modules
Mod Authtoken
Okapi
Mod-Inventory Storage.
RDS resource usage
RDS memory usage
As you can see, all resources shows stable behavior. During whole test Memory and CPU usage hasn't spikes or anomalies. No slow queries was found.
Test 2: 2 users, 1 instance
Requests | Total | OK | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct |
|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 62338 | 62338 | 34.631 | 0.032 | 0.042 | 0.047 | 0.065 | 0.115 |
Resources
Most used modules:
mod autotoken
okapi
mod inventory storage
RDS memory usage
Comment: As you can see, 75 percentile response time for first and second test are almost the same (43ms VS 47ms) while request total number almost twice higher. That cases almost twice higher CPU usage on mod-autotoken.
Test 3: 1 user with incremental instances per iteration
Requests | Total | OK | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct |
|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 2636 | 2636 | 1.464 | 0.044 | 0.665 | 0.984 | 1.231 | 1.367 |
Resources
Most used module
mod-inventory storage
RDS memory usage
Test 4: 1 user, 1 instance with 300 holdings
Requests | Total | OK | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct |
|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 6461 | 6461 | 3.589 | 0.180 | 0.230 | 0.373 | 0.416 | 0.509 |
RDS CPU usage
RDS memory usage
Services CPU usage
Most used modules
mod-inventory storage
okapi
mod-authtoken
Test 5: 1 user, 50 instances
Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 22372 | 22372 | 0 | 0 | 12.429 | 0.048 | 0.062 | 0.066 | 0.184 | 0.259 | 7.372 | 0.075 | 0.181 |
As you can see, there is a little higher response time at the beginning of the test than at the end of it. During that time (5 minutes) 2000 http requests was executed each request contains 50 instance id's. That mean that using this endpoint we can complete 1000000 instances id's in 5 minutes.
Resources
As you can see none of modules are overloaded or has unexpected spikes.
Same situation for services memory usage
RDS CPU usage
Using 50 instances per request RDS CPU usage reached ~20%.
RDS Memory usage
Memory usage didn't show much usage
Test 6: 8 users, 50 instances per request
Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 38593 | 38593 | 0 | 0 | 21.552 | 0.082 | 0.289 | 0.361 | 0.559 | 1.036 | 254.133 | 0.352 | 0.532 |
Resources
Most used modules
mod-inventory storage
okapi
mod-authtoken
RDS CPU usage
RDS Memory usage
Test 7: 1 user with incremental instances per request, +10 per loop
Note: maximum instances per request reached in this test - 5325 instances in one request.
Requests | Total | OK | KO | % KO | Req/s | Min | 50th pct | 75th pct | 95th pct | 99th pct | Max | Average | Latency |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
[POST] /inventory-hierarchy/items-and-holdings | 1065 | 1065 | 0 | 0 | 0.197 | 0.079 | 5.161 |