Overview

Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that bulk edits can be performed simultaneously.

How long does it take to export 100, 1000, 2500, 5000 records?
Use it for up to 5 concurrent users.
Look for a memory trnend and CPU usage

Summary.

Test report for Bulk Edits users-app functionality 2023-09-15.

5k records per user, 5 users simultaneously (25k records total) can be uploaded in about 6 min 13 seconds,
Possible job duration degradation compared to NOLANA realese is caused by adding of new mod-bulk-operation module and by fully changing of bulk-edit architecture.
The files with identifiers should be strictly determined.
The memory of all modules during the tests for 5000 records with 5 parallel was stable, the memory gap in Figure 1 was caused by restarting several modules before DI job run.
Instance CPU usage
- maximal value for text 4VU (0.1-1-2.5k-5k) was 26%
- maximal value for text 5VU (0.1-1-2.5k-5k) was 27%
Service CPU usage for test 5VU (0.1-1-2.5k-5k)
- CPU of mod-bulk-operations 126%, for all other modules, did not exceed 22%.
RDS CPU utilization did not exceed 43% for 5jobs 5000 records and 34% for 4jobs 5k.

Recommendations & Jiras

For further testing Users' bulk editing can be performed with 10k records

Test Runs

Total processing time of upload and edit - commit changes. Units =minutes:seconds

Number of virtual user/ Records	1VU	2VU	3VU	4VU	5VU
100 records	00:01:05	00:01:03 00:01:03	00:01:03 00:01:03 00:01:05	00:01:03 00:01:03 00:01:03 00:01:04	00:01:04 00:01:04. 00:01:04 00:01:04 00:01:04
1000 records	00:01:37	00:01:59 00:01:35	00:02:02 00:02:05 00:01:36	00:02:02 00:02:02 00:01:59 00:02:00	00:02:11 00:02:13 00:02:13 00:02:05 00:02:06
2500 records	00:03:28	00:03:30 00:03:31	00:03:42 00:03:42 00:01:10(*1)	00:00:23(*2) 00:03:47 00:03:46 00:03:46	00:03:48 00:04:03 00:03:44 00:04:04 00:04:04
5000 records	00:05:13	00:06:36 00:06:38 *(3)**	00:06:48 00:06:50 00:06:50 *(3)**	00:06:32 00:06:32 00:02:29. 00:02:29 *(3)**	00:06:13 00:06:13 00:06:11 00:06:08 00:06:13 *(3)**

(*1) Duplicate barcodes in CSV input data, 1200 out of 2500 records were processed

(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException) MODEXPW-306 - Getting issue details... STATUS

(*3) Was running as a separate test to avoid duplication of input data

Comparison with previous results

5VU, Records	Nolana	Orchid
2500	2 min 9 sec	4 minutes 4 seconds
5000 records	3 min 47 sec	6minutes 13 seconds

Possible job duration degradation caused by adding of new mod-bulk-operation module and by fully changing of bulk-edit architecture.

Memory usage

The time range marked with # Several modules were restarted and then data import jobs were running on the environment

Memory usage during the testing with 1-5 concurrent and 100-1000-2500-5000 records

Memory usage during the testing with 2-5 concurrent and 5000 records

Memory usage for the same time range as the figure below, but only involved services were selected.

Instance CPU utilization

CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records

The test 5VU(0.1-1-2.5k-5k) was restarted because there were in the input data(Barcodes duplication)

CPU instance utilization for 2 up to 5 Vu with 5000 records

Service CPU utilization

CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records

RDS CPU utilization

For all tests - RDS CPU utilization exceeds 40% (for tests with 5k user records). For the tests with 2500 user records did not exceed 27%.

Database connections

Errors in logs during testing

(*1) Duplicate barcodes in CSV input data, 1200 out of 2500 records were processed

(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException) MODEXPW-306 - Getting issue details... STATUS

Expand log details

2023-09-12T17:01:04.446+03:00
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 546 out of bounds for length 13
ncp5/mod-data-export-worker/c5cd67da96654750b7ddbfa0a4e1467b
054267740449:ncp5-folio-eis
Field Value
@ingestionTime 1694527267769
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-data-export-worker/c5cd67da96654750b7ddbfa0a4e1467b
@message caused by: java.lang.ArrayIndexOutOfBoundsException: Index 546 out of bounds for length 13
@timestamp 1694527264446

Kafka

14:01:05 [] [] [] [] INFO KafkaService Sent Job(id=bedd254a-1224-4624-955e-af6353526c2e, name=null, description=null, source=null, isSystemSource=null, type=null, exportTypeSpecificParameters=null, status=null, files=[https://data-export-folio-eis-us-east-1-int-tenant.s3.us-east-1.amazonaws.com/bedd254a-1224-4624-955e-af6353526c2e/2023-09-12-Matched-Records-2500-user-barcodes1.csv?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJHMEUCIGIcZxlnmOqdOT0g1iJqOKLC61aBoptFcxXNPsFbhpn6AiEAnBxM4fK%2FyJ9jm5B3SXVbKM3xDPSHTCyEMY8Jeg4oJLIq5wMIFxAEGgwwNTQyNjc3NDA0NDkiDAjPFjSJDH%2FOsFxZzyrEAw8vjM5cXfq9yvQ19O5RToH4O4v3ZyuFdWD1wwh91FZEf7fJrYVor3EpV8eRZI5XITzLK7vYFmOYdBqmDVgyqveSEeTr68P3NiE4E8NYeCbwEFD6G46UljHUW3EilB1u24JuTWhq%2FED3JymMI2RZyhqmp2ckxfIyEalwXzSLqjyihbzwd1fe9xCgqiGKN2UmYFV1H09aHkVz82l%2FLEZUHiaqELTcIWQzYp9zgC1Z6QsWm4Ak6IxKrfFqTTo32J24JkkqXVVXJIkUppSokpS8l6m11dRuSO1FvGokp1goP%2FEqW7Jg3rsQsBfayA7jbjrEJn3T%2F9GXTQLorUH626P%2FjxkQBmHJgN93490pasFv8d0tD7W1ntbHBo8M3MrH8j2JLXSRaFFEnTWm1D%2BGYUYoO7f06MfIrT2QOcNxWA%2BJTv84H%2B0SvXEdg7Qq%2FkNpoxEqPfFwu3VrqcKB5%2FjnVAdrN%2BTKS9lPXHXUS31xMX%2FWIudHfjfodh5ubs4qPif9BYhkoAaP9geqlmNqeqi32keDvmaMn%2BedMHlI46XtpRcZzy2Lk3D1mxB8l4KpG2FWUHIUuv7%2B0AsB6J%2BkR8G9EhWSB53PhvkcMMLSgagGOqUBazbshbs3xaSlwmWGD7HBnRVSNoO2m2OhhAWHR1BCER6GIISc2uJMB01EawiQFdpw0inwQ%2BiTBv2GDgJFbBLTx5j6pl4x5fhmsLIC3vLh%2FKYd4QkyxNOnBRXKNCEzeZrs%2Bq%2FAQfbBqZjLQe8ewHuIGNXHEprA4JwJ7yt%2Bl%2BlfYMZjS94Q%2Fb%2Be8QV6gmPZcp0y5QgJiojYxXFCSuslhY9vRVWsunUC&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQZIU4AEQ6NAV5PXF%2F20230912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230912T140105Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=0d179bd89fcc32deb7b57b2c5d3c996a21102e284a3558a4847fcb26a0a2e6c8, , https://data-export-folio-eis-us-east-1-int-tenant.s3.us-east-1.amazonaws.com/bedd254a-1224-4624-955e-af6353526c2e/2023-09-12-Matched-Records-2500-user-barcodes1.json.json?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJHMEUCIGIcZxlnmOqdOT0g1iJqOKLC61aBoptFcxXNPsFbhpn6AiEAnBxM4fK%2FyJ9jm5B3SXVbKM3xDPSHTCyEMY8Jeg4oJLIq5wMIFxAEGgwwNTQyNjc3NDA0NDkiDAjPFjSJDH%2FOsFxZzyrEAw8vjM5cXfq9yvQ19O5RToH4O4v3ZyuFdWD1wwh91FZEf7fJrYVor3EpV8eRZI5XITzLK7vYFmOYdBqmDVgyqveSEeTr68P3NiE4E8NYeCbwEFD6G46UljHUW3EilB1u24JuTWhq%2FED3JymMI2RZyhqmp2ckxfIyEalwXzSLqjyihbzwd1fe9xCgqiGKN2UmYFV1H09aHkVz82l%2FLEZUHiaqELTcIWQzYp9zgC1Z6QsWm4Ak6IxKrfFqTTo32J24JkkqXVVXJIkUppSokpS8l6m11dRuSO1FvGokp1goP%2FEqW7Jg3rsQsBfayA7jbjrEJn3T%2F9GXTQLorUH626P%2FjxkQBmHJgN93490pasFv8d0tD7W1ntbHBo8M3MrH8j2JLXSRaFFEnTWm1D%2BGYUYoO7f06MfIrT2QOcNxWA%2BJTv84H%2B0SvXEdg7Qq%2FkNpoxEqPfFwu3VrqcKB5%2FjnVAdrN%2BTKS9lPXHXUS31xMX%2FWIudHfjfodh5ubs4qPif9BYhkoAaP9geqlmNqeqi32keDvmaMn%2BedMHlI46XtpRcZzy2Lk3D1mxB8l4KpG2FWUHIUuv7%2B0AsB6J%2BkR8G9EhWSB53PhvkcMMLSgagGOqUBazbshbs3xaSlwmWGD7HBnRVSNoO2m2OhhAWHR1BCER6GIISc2uJMB01EawiQFdpw0inwQ%2BiTBv2GDgJFbBLTx5j6pl4x5fhmsLIC3vLh%2FKYd4QkyxNOnBRXKNCEzeZrs%2Bq%2FAQfbBqZjLQe8ewHuIGNXHEprA4JwJ7yt%2Bl%2BlfYMZjS94Q%2Fb%2Be8QV6gmPZcp0y5QgJiojYxXFCSuslhY9vRVWsunUC&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQZIU4AEQ6NAV5PXF%2F20230912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230912T140105Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=e87816c5ab96801d52eb044a3f32dc1527ae474e34d0d0966872372b6635c70c], fileNames=null, startTime=Tue Sep 12 14:00:46 UTC 2023, endTime=Tue Sep 12 14:01:04 UTC 2023, createdDate=Tue Sep 12 14:00:46 UTC 2023, createdByUserId=null, createdByUsername=null, updatedDate=Tue Sep 12 14:00:46 UTC 2023, updatedByUserId=null, updatedByUsername=null, outputFormat=null, errorDetails=Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException), batchStatus=FAILED, exitStatus=exitCode=FAILED;exitDescription=, identifierType=null, entityType=null, progress=class Progress {

Appendix

Infrastructure

PTF -environment ncp5 [ environment name]

8 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 [Number of ECS instances, instance type, location region]
2 instances of db.r6.xlarge database instances: Writer & reader instances
MSK ptf-kakfa-3 [ kafka configurations]
- 4 kafka.m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

Modules memory and CPU parameters:

Module	Task Def. Revision	Module Version	Task Count	Mem Hard Limit	Mem Soft limit	CPU units	Xmx	MetaspaceSize	MaxMetaspaceSize
ncp5-pvt
Wed Sep 13 09:54:48 UTC 2023
mod-agreements	6	mod-agreements:5.5.2	2	1592	1488	128	968	384	512
mod-bulk-operations	17	mod-bulk-operations:1.0.6	2	3072	2600	1024	1536	384	512
mod-users-bl	8	mod-users-bl:7.5.0	2	1440	1152	512	922	88	128
mod-inventory-storage	12	mod-inventory-storage:26.0.0	2	4096	3690	2048	3076	384	512
mod-data-export-worker	10	mod-data-export-worker:3.0.13	1	3072	2800	1024	2048	384	512
mod-inventory	12	mod-inventory:20.0.6	2	2880	2592	1024	1814	384	512
nginx-okapi	7	nginx-okapi:2023.06.14	2	1024	896	128	0	0	0
okapi-b	8	okapi:5.0.1	3	1684	1440	1024	922	384	512
mod-notes	6	mod-notes:5.0.1	2	1024	896	128	952	384	512
mod-data-export-spring	8	mod-data-export-spring:2.0.2	1	2048	1844	256	1536	384	512