IN PROGRESS

PERF-120 - Getting issue details... STATUS

Goals:

Define overall performance of POST /inventory-hierarchy/items-and-holdings;
Find bottlenecks if possible;
Measure KPI's for different number of instances included in request;
Measure KPI's for different number of holdings attached per instance ID;

Overview:

Mod-inventory-storage has a new API to gather all related instances' data, i.e., items and holdings, in one call in order to eliminate making 3 separate calls to get the data. The API also allows passing in x number of instance IDs to lookup. Since teams are now starting to use this API, it's important to understand the stability of the API with respect to the module's CPU and memory usage, and also the database usage.

Summary:

System shows stable work and quick responses in all test;
No memory leaks found;
There is dependency between most used modules (see CPU) and test type. In case if it's extremely fast responses (for example for 5 holdings per instance and one instance) - most used module will be mod-authtoken, otherwise in (see test 4) that will be mod inventory-storage. Same for increasing number of instances per request.
DB CPU usage reach it maximum at 8 users test
Fast response for most high load test (test 4). To extract instance contains > 300 holdings it takes (75 percentile) 373 ms. what means that getting one holding with this endpoint takes 1.2 ms.
No bottlenecks found.

Version:

"okapi-3.1.2";

"mod-inventory-storage-19.4.0-SNAPSHOT.448";

"mod-authtoken-2.5.1".

RDS Data amount:

Precondition, if database free memory is less than 2000 MB, DB instance should be restarted.

Holdings per instance	instance amount
>0	7183552
>1	407467
>2	69890
>3	22241
>4	10201
>5	5635
>10	1060
>15	516
>20	331
>25	240
>40	115
>100	23
>300	7

Test Runs:

Test	Virtual Users	Duration (min)	Instance per request	Holdings per instance
1	1	30	1	5
2	2	30	1	5
3	1	30	1 (adding instance to request each loop)	5
4	1	30	1	300
5	1	30	50	>1
6	8	30	50	>1
7	1	90	(adding 10 instance to request each loop)	>1

Results

Summary:

During test set we've measured general performance of new endpoint.

Test 1:

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct
[POST] /inventory-hierarchy/items-and-holdings	36530	36530	20.294	0.033	0.040	0.043	0.057	0.101

Resource usage:

Most used modules:

Mod Authtoken;
Okapi;
Mod-Inventory Storage.

RDS resource usage:

RDS memory usage

As you can see, all resources shows stable behavior. During whole test Memory and CPU usage hasn't spikes or anomalies. No slow queries was found.

Test 2

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct
[POST] /inventory-hierarchy/items-and-holdings	62338	62338	34.631	0.032	0.042	0.047	0.065	0.115

Resources

most used modules:

mod autotoken
okapi
mod inventory storage

RDS memory usage

Comment: As you can see, 75 percentile response time for first and second test are almost the same (43ms VS 47ms) while request total number almost twice higher. That cases almost twice higher CPU usage on mod-autotoken.

Test 3

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct
[POST] /inventory-hierarchy/items-and-holdings	2636	2636	1.464	0.044	0.665	0.984	1.231	1.367

Resources

Most used module - Mod-inventory storage;

RDS memory usage

Test 4:

Requests	Total	OK	Req/s	Min	50th pct	75th pct	95th pct	99th pct
[POST] /inventory-hierarchy/items-and-holdings	6461	6461	3.589	0.180	0.230	0.373	0.416	0.509

Resources

RDS CPU usage

RDS memory usage

Most used modules:

mod-inventory storage
okapi
mod-authtoken

Test 5

Requests	Total	OK	KO	% KO	Req/s	Min	50th pct	75th pct	95th pct	99th pct	Max	Average	Latency
[POST] /inventory-hierarchy/items-and-holdings	22372	22372	0	0	12.429	0.048	0.062	0.066	0.184	0.259	7.372	0.075	0.181

As you can see, there is a little higher response time at the beginning of the test than at the end of it. During that time (5 minutes) 2000 http requests was executed each request contains 50 instance id's. That mean that using this endpoint we can complete 1000000 instances id's in 5 minutes.