...
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that bulk edits can be performed simultaneously.
...
- 5k records per user, 5 users simultaneously (25k records total) can be uploaded in about 6 min 13 seconds,
Possible job duration degradation compared to NOLANA realese is caused by adding of new mod-bulk-operation module and by fully changing of bulk-edit architecture.
- The files with identifiers should be strictly determined.
- The memory of all modules during the tests for 5000 records with 5 parallel was stable, the memory gap in Figure 1 was caused by restarting several modules before DI job run.
- Instance CPU usage
- maximal value for text 4VU (0.1-1-2.5k-5k) was 26%
- maximal value for text 5VU (0.1-1-2.5k-5k) was 27%
- Service CPU usage for test 5VU (0.1-1-2.5k-5k)
- CPU of mod-bulk-operations 126%, for all other modules, did not exceed 22%.
- RDS CPU utilization did not exceed 43% for 5jobs 5000 records and 34% for 4jobs 5k.
...
| Number of virtual user/ Records | 1VU | 2VU | 3VU | 4VU | 5VU |
|---|
| 100 records | 00:01:05 | 00:01:03 00:01:03 | 00:01:03
00:01:03
00:01:05 | 00:01:03
00:01:03
00:01:03
00:01:04 | 00:01:04
00:01:04.
00:01:04
00:01:04
00:01:04 |
| 1000 records | 00:01:37 | 00:01:59
00:01:35 | 00:02:02
00:02:05
00:01:36 | 00:02:02
00:02:02
00:01:59
00:02:00 | 00:02:11
00:02:13
00:02:13
00:02:05
00:02:06 |
| 2500 records | 00:03:28 | 00:03:30
00:03:31 | 00:03:42
00:03:42
00:01:10(*1) | 00:00:23(*2)
00:03:47
00:03:46
00:03:46 | 00:03:48
00:04:03
00:03:44
00:04:04
00:04:04 |
| 5000 records | 00:05:13 | 00:06:36
00:06:38 (*3) | 00:06:48
00:06:50
00:06:50
(*3) | 00:06:3244
00:06:3236
00:0206:2939.
00:0206:29 46
(*3) | 00:06:13
00:06:13
00:06:11
00:06:08
00:06:13
(*3) |
...
(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException) | Jira Legacy |
|---|
| server | System JiraJIRA |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODEXPW-306 |
|---|
|
...
5VU,
Records | Nolana
| Orchid |
|---|
| 2500 | 2 min 9 sec | 4 minutes 4 seconds |
| 5000 records | 3 min 47 sec | 6minutes 13
seconds |
Possible job duration degradation caused by adding of new mod-bulk-operation module and by fully changing of bulk-edit architecture.
Memory usage
The time range marked with # Several modules were restarted and then data import jobs were running on the environment environment (DI jobs were not part of Bulk Edits testing).
Memory usage during the testing with 1-5 concurrent and 100-1000-2500-5000 records
...
CPU instance utilization for 2 up to 5 Vu with 5000 records
Service CPU utilization
Image Removed
Image Added
CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
Image Removed
Image Added
CPU instance utilization for 1 2 up to 5 Vu with 100-1000-2500-VU with 5000 records
RDS CPU utilization
...
(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException)
| Jira Legacy |
|---|
| server | System JiraJIRA |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODEXPW-306 |
|---|
|
| Expand |
|---|
|
2023-09-12T17:01:04.446+03:00
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 546 out of bounds for length 13
ncp5/mod-data-export-worker/c5cd67da96654750b7ddbfa0a4e1467b
054267740449:ncp5-folio-eis
Field Value
@ingestionTime 1694527267769
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-data-export-worker/c5cd67da96654750b7ddbfa0a4e1467b
@message caused by: java.lang.ArrayIndexOutOfBoundsException: Index 546 out of bounds for length 13
@timestamp 1694527264446 Kafka 14:01:05 [] [] [] [] INFO KafkaService Sent Job(id=bedd254a-1224-4624-955e-af6353526c2e, name=null, description=null, source=null, isSystemSource=null, type=null, exportTypeSpecificParameters=null, status=null, files=[https://data-export-folio-eis-us-east-1-int-tenant.s3.us-east-1.amazonaws.com/bedd254a-1224-4624-955e-af6353526c2e/2023-09-12-Matched-Records-2500-user-barcodes1.csv?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJHMEUCIGIcZxlnmOqdOT0g1iJqOKLC61aBoptFcxXNPsFbhpn6AiEAnBxM4fK%2FyJ9jm5B3SXVbKM3xDPSHTCyEMY8Jeg4oJLIq5wMIFxAEGgwwNTQyNjc3NDA0NDkiDAjPFjSJDH%2FOsFxZzyrEAw8vjM5cXfq9yvQ19O5RToH4O4v3ZyuFdWD1wwh91FZEf7fJrYVor3EpV8eRZI5XITzLK7vYFmOYdBqmDVgyqveSEeTr68P3NiE4E8NYeCbwEFD6G46UljHUW3EilB1u24JuTWhq%2FED3JymMI2RZyhqmp2ckxfIyEalwXzSLqjyihbzwd1fe9xCgqiGKN2UmYFV1H09aHkVz82l%2FLEZUHiaqELTcIWQzYp9zgC1Z6QsWm4Ak6IxKrfFqTTo32J24JkkqXVVXJIkUppSokpS8l6m11dRuSO1FvGokp1goP%2FEqW7Jg3rsQsBfayA7jbjrEJn3T%2F9GXTQLorUH626P%2FjxkQBmHJgN93490pasFv8d0tD7W1ntbHBo8M3MrH8j2JLXSRaFFEnTWm1D%2BGYUYoO7f06MfIrT2QOcNxWA%2BJTv84H%2B0SvXEdg7Qq%2FkNpoxEqPfFwu3VrqcKB5%2FjnVAdrN%2BTKS9lPXHXUS31xMX%2FWIudHfjfodh5ubs4qPif9BYhkoAaP9geqlmNqeqi32keDvmaMn%2BedMHlI46XtpRcZzy2Lk3D1mxB8l4KpG2FWUHIUuv7%2B0AsB6J%2BkR8G9EhWSB53PhvkcMMLSgagGOqUBazbshbs3xaSlwmWGD7HBnRVSNoO2m2OhhAWHR1BCER6GIISc2uJMB01EawiQFdpw0inwQ%2BiTBv2GDgJFbBLTx5j6pl4x5fhmsLIC3vLh%2FKYd4QkyxNOnBRXKNCEzeZrs%2Bq%2FAQfbBqZjLQe8ewHuIGNXHEprA4JwJ7yt%2Bl%2BlfYMZjS94Q%2Fb%2Be8QV6gmPZcp0y5QgJiojYxXFCSuslhY9vRVWsunUC&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQZIU4AEQ6NAV5PXF%2F20230912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230912T140105Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=0d179bd89fcc32deb7b57b2c5d3c996a21102e284a3558a4847fcb26a0a2e6c8, , https://data-export-folio-eis-us-east-1-int-tenant.s3.us-east-1.amazonaws.com/bedd254a-1224-4624-955e-af6353526c2e/2023-09-12-Matched-Records-2500-user-barcodes1.json.json?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJHMEUCIGIcZxlnmOqdOT0g1iJqOKLC61aBoptFcxXNPsFbhpn6AiEAnBxM4fK%2FyJ9jm5B3SXVbKM3xDPSHTCyEMY8Jeg4oJLIq5wMIFxAEGgwwNTQyNjc3NDA0NDkiDAjPFjSJDH%2FOsFxZzyrEAw8vjM5cXfq9yvQ19O5RToH4O4v3ZyuFdWD1wwh91FZEf7fJrYVor3EpV8eRZI5XITzLK7vYFmOYdBqmDVgyqveSEeTr68P3NiE4E8NYeCbwEFD6G46UljHUW3EilB1u24JuTWhq%2FED3JymMI2RZyhqmp2ckxfIyEalwXzSLqjyihbzwd1fe9xCgqiGKN2UmYFV1H09aHkVz82l%2FLEZUHiaqELTcIWQzYp9zgC1Z6QsWm4Ak6IxKrfFqTTo32J24JkkqXVVXJIkUppSokpS8l6m11dRuSO1FvGokp1goP%2FEqW7Jg3rsQsBfayA7jbjrEJn3T%2F9GXTQLorUH626P%2FjxkQBmHJgN93490pasFv8d0tD7W1ntbHBo8M3MrH8j2JLXSRaFFEnTWm1D%2BGYUYoO7f06MfIrT2QOcNxWA%2BJTv84H%2B0SvXEdg7Qq%2FkNpoxEqPfFwu3VrqcKB5%2FjnVAdrN%2BTKS9lPXHXUS31xMX%2FWIudHfjfodh5ubs4qPif9BYhkoAaP9geqlmNqeqi32keDvmaMn%2BedMHlI46XtpRcZzy2Lk3D1mxB8l4KpG2FWUHIUuv7%2B0AsB6J%2BkR8G9EhWSB53PhvkcMMLSgagGOqUBazbshbs3xaSlwmWGD7HBnRVSNoO2m2OhhAWHR1BCER6GIISc2uJMB01EawiQFdpw0inwQ%2BiTBv2GDgJFbBLTx5j6pl4x5fhmsLIC3vLh%2FKYd4QkyxNOnBRXKNCEzeZrs%2Bq%2FAQfbBqZjLQe8ewHuIGNXHEprA4JwJ7yt%2Bl%2BlfYMZjS94Q%2Fb%2Be8QV6gmPZcp0y5QgJiojYxXFCSuslhY9vRVWsunUC&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQZIU4AEQ6NAV5PXF%2F20230912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230912T140105Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=e87816c5ab96801d52eb044a3f32dc1527ae474e34d0d0966872372b6635c70c], fileNames=null, startTime=Tue Sep 12 14:00:46 UTC 2023, endTime=Tue Sep 12 14:01:04 UTC 2023, createdDate=Tue Sep 12 14:00:46 UTC 2023, createdByUserId=null, createdByUsername=null, updatedDate=Tue Sep 12 14:00:46 UTC 2023, updatedByUserId=null, updatedByUsername=null, outputFormat=null, errorDetails=Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException), batchStatus=FAILED, exitStatus=exitCode=FAILED;exitDescription=, identifierType=null, entityType=null, progress=class Progress {
|
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
...
Modules memory and CPU parameters:
| Module | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize |
| ncp5-pvt |
Wed Sep 13 09:54:48 UTC 2023 |
| mod-agreements | 6 | | 2 | 1592 | 1488 | 128 | 968 | 384 | 512 |
| mod-bulk-operations | 17 | | 2 | 3072 | 2600 | 1024 | 1536 | 384 | 512 |
| mod-users-bl | 8 | | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 |
| mod-inventory-storage | 12 | | 2 | 4096 | 3690 | 2048 | 3076 | 384 | 512 |
| mod-data-export-worker | 10 | | 1 | 3072 | 2800 | 1024 | 2048 | 384 | 512 |
| mod-inventory | 12 | | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 |
| nginx-okapi | 7 | | 2 | 1024 | 896 | 128 | 0 | 0 | 0 |
| okapi-b | 8 | | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 |
| mod-notes | 6 | | 2 | 1024 | 896 | 128 | 952 | 384 | 512 |
| mod-data-export-spring | 8 | | 1 | 2048 | 1844 | 256 | 1536 | 384 | 512 |