Overview
Bulk Edits - Establish a performance baseline for combined bulk updates PERF-480 in the Orchid release that has architectural changes that were implemented in UXPROD-3842. The goal is to make sure that bulk edits can be performed simultaneously.
- How long does it take to export 100, 1000, 2500, 5000 records?
- Use it for up to 5 concurrent users.
- Look for a memory trend trnend and CPU usage
Summary.
Test report for Bulk Edits users-app functionality 2023-09-15.
- 5k records per user, 5 users simultaneously (25k records total) can be uploaded in about 6 min 13 seconds,
- The files with identifiers should be strictly determined.
- The memory of all modules during the tests for 5000 records with 5 parallel was stable, the memory gap in Figure 1 was caused by restarting several modules before DI job run.
- Instance CPU usage
- maximal value for text 4VU (0.1-1-2.5k-5k) was 26%
- maximal value for text 5VU (0.1-1-2.5k-5k) was 27%
- Service CPU usage for test 5VU (0.1-1-2.5k-5k)
- CPU of mod-bulk-operations 126%, for all other modules, did not exceed 22%.
- RDS CPU utilization did not exceed 43% for 5jobs 5000 records and 34% for 4jobs 5k.
Recommendations & Jiras
For further testing Users' bulk editing can be performed with 10k records
Test Runs
Total processing time of upload and edit - commit changes. Units =minutes:seconds
...
(*3) Was running as a separate test to avoid duplication of input data
Comparison with previous results
5VU,
Records | Nolana
| Orchid |
|---|
| 2500 | 2 min 9 sec | 4 minutes 4 seconds |
| 5000 records | 3 min 47 sec | 6minutes 13
seconds |
Memory usage
The time range marked with # Several modules were restarted and then data import jobs were running on the environment
...
Memory usage for the same time range as the figure below, but only involved services were selected.
Instance CPU utilization
CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
...
CPU instance utilization for 2 up to 5 Vu with 5000 records
Service CPU utilization
CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
...
CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
RDS CPU utilization
For all tests - RDS CPU utilization exceeds 40% (for tests with 5k user records). For the tests with 2500 user records did not exceed 27%.
Database connections
Errors in logs during testing
(*1) Duplicate barcodes in CSV input data, 1200 out of 2500 records were processed
...
| Expand |
|---|
|
2023-09-12T17:01:04.446+03:00
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 546 out of bounds for length 13
ncp5/mod-data-export-worker/c5cd67da96654750b7ddbfa0a4e1467b
054267740449:ncp5-folio-eis
Field Value
@ingestionTime 1694527267769
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-data-export-worker/c5cd67da96654750b7ddbfa0a4e1467b
@message caused by: java.lang.ArrayIndexOutOfBoundsException: Index 546 out of bounds for length 13
@timestamp 1694527264446 Kafka 14:01:05 [] [] [] [] INFO KafkaService Sent Job(id=bedd254a-1224-4624-955e-af6353526c2e, name=null, description=null, source=null, isSystemSource=null, type=null, exportTypeSpecificParameters=null, status=null, files=[https://data-export-folio-eis-us-east-1-int-tenant.s3.us-east-1.amazonaws.com/bedd254a-1224-4624-955e-af6353526c2e/2023-09-12-Matched-Records-2500-user-barcodes1.csv?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJHMEUCIGIcZxlnmOqdOT0g1iJqOKLC61aBoptFcxXNPsFbhpn6AiEAnBxM4fK%2FyJ9jm5B3SXVbKM3xDPSHTCyEMY8Jeg4oJLIq5wMIFxAEGgwwNTQyNjc3NDA0NDkiDAjPFjSJDH%2FOsFxZzyrEAw8vjM5cXfq9yvQ19O5RToH4O4v3ZyuFdWD1wwh91FZEf7fJrYVor3EpV8eRZI5XITzLK7vYFmOYdBqmDVgyqveSEeTr68P3NiE4E8NYeCbwEFD6G46UljHUW3EilB1u24JuTWhq%2FED3JymMI2RZyhqmp2ckxfIyEalwXzSLqjyihbzwd1fe9xCgqiGKN2UmYFV1H09aHkVz82l%2FLEZUHiaqELTcIWQzYp9zgC1Z6QsWm4Ak6IxKrfFqTTo32J24JkkqXVVXJIkUppSokpS8l6m11dRuSO1FvGokp1goP%2FEqW7Jg3rsQsBfayA7jbjrEJn3T%2F9GXTQLorUH626P%2FjxkQBmHJgN93490pasFv8d0tD7W1ntbHBo8M3MrH8j2JLXSRaFFEnTWm1D%2BGYUYoO7f06MfIrT2QOcNxWA%2BJTv84H%2B0SvXEdg7Qq%2FkNpoxEqPfFwu3VrqcKB5%2FjnVAdrN%2BTKS9lPXHXUS31xMX%2FWIudHfjfodh5ubs4qPif9BYhkoAaP9geqlmNqeqi32keDvmaMn%2BedMHlI46XtpRcZzy2Lk3D1mxB8l4KpG2FWUHIUuv7%2B0AsB6J%2BkR8G9EhWSB53PhvkcMMLSgagGOqUBazbshbs3xaSlwmWGD7HBnRVSNoO2m2OhhAWHR1BCER6GIISc2uJMB01EawiQFdpw0inwQ%2BiTBv2GDgJFbBLTx5j6pl4x5fhmsLIC3vLh%2FKYd4QkyxNOnBRXKNCEzeZrs%2Bq%2FAQfbBqZjLQe8ewHuIGNXHEprA4JwJ7yt%2Bl%2BlfYMZjS94Q%2Fb%2Be8QV6gmPZcp0y5QgJiojYxXFCSuslhY9vRVWsunUC&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQZIU4AEQ6NAV5PXF%2F20230912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230912T140105Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=0d179bd89fcc32deb7b57b2c5d3c996a21102e284a3558a4847fcb26a0a2e6c8, , https://data-export-folio-eis-us-east-1-int-tenant.s3.us-east-1.amazonaws.com/bedd254a-1224-4624-955e-af6353526c2e/2023-09-12-Matched-Records-2500-user-barcodes1.json.json?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEC4aCXVzLWVhc3QtMSJHMEUCIGIcZxlnmOqdOT0g1iJqOKLC61aBoptFcxXNPsFbhpn6AiEAnBxM4fK%2FyJ9jm5B3SXVbKM3xDPSHTCyEMY8Jeg4oJLIq5wMIFxAEGgwwNTQyNjc3NDA0NDkiDAjPFjSJDH%2FOsFxZzyrEAw8vjM5cXfq9yvQ19O5RToH4O4v3ZyuFdWD1wwh91FZEf7fJrYVor3EpV8eRZI5XITzLK7vYFmOYdBqmDVgyqveSEeTr68P3NiE4E8NYeCbwEFD6G46UljHUW3EilB1u24JuTWhq%2FED3JymMI2RZyhqmp2ckxfIyEalwXzSLqjyihbzwd1fe9xCgqiGKN2UmYFV1H09aHkVz82l%2FLEZUHiaqELTcIWQzYp9zgC1Z6QsWm4Ak6IxKrfFqTTo32J24JkkqXVVXJIkUppSokpS8l6m11dRuSO1FvGokp1goP%2FEqW7Jg3rsQsBfayA7jbjrEJn3T%2F9GXTQLorUH626P%2FjxkQBmHJgN93490pasFv8d0tD7W1ntbHBo8M3MrH8j2JLXSRaFFEnTWm1D%2BGYUYoO7f06MfIrT2QOcNxWA%2BJTv84H%2B0SvXEdg7Qq%2FkNpoxEqPfFwu3VrqcKB5%2FjnVAdrN%2BTKS9lPXHXUS31xMX%2FWIudHfjfodh5ubs4qPif9BYhkoAaP9geqlmNqeqi32keDvmaMn%2BedMHlI46XtpRcZzy2Lk3D1mxB8l4KpG2FWUHIUuv7%2B0AsB6J%2BkR8G9EhWSB53PhvkcMMLSgagGOqUBazbshbs3xaSlwmWGD7HBnRVSNoO2m2OhhAWHR1BCER6GIISc2uJMB01EawiQFdpw0inwQ%2BiTBv2GDgJFbBLTx5j6pl4x5fhmsLIC3vLh%2FKYd4QkyxNOnBRXKNCEzeZrs%2Bq%2FAQfbBqZjLQe8ewHuIGNXHEprA4JwJ7yt%2Bl%2BlfYMZjS94Q%2Fb%2Be8QV6gmPZcp0y5QgJiojYxXFCSuslhY9vRVWsunUC&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQZIU4AEQ6NAV5PXF%2F20230912%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230912T140105Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=e87816c5ab96801d52eb044a3f32dc1527ae474e34d0d0966872372b6635c70c], fileNames=null, startTime=Tue Sep 12 14:00:46 UTC 2023, endTime=Tue Sep 12 14:01:04 UTC 2023, createdDate=Tue Sep 12 14:00:46 UTC 2023, createdByUserId=null, createdByUsername=null, updatedDate=Tue Sep 12 14:00:46 UTC 2023, updatedByUserId=null, updatedByUsername=null, outputFormat=null, errorDetails=Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException), batchStatus=FAILED, exitStatus=exitCode=FAILED;exitDescription=, identifierType=null, entityType=null, progress=class Progress {
|
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
...