| Table of Contents |
|---|
...
- How long does it take to export 100, 1000, 2500, 5000 records?
- Use it for up to 5 concurrent users.
- Look for a memory trend and CPU usage
Summary
Test report for Bulk Edits users-app functionality 2022-10-26.
- 5k records per user, 5 users simultaneously (25k records total) can be uploaded in about 6 min 13 seconds,
- The files with identifiers should be strictly determined.
- The memory of all modules during the tests for 5000 records with 5 parallel was stable, the memory gap in figure 1 was cause by the import data process.
- Instance CPU usage
- maximal value for text 4VU (0.1-1-2.5k-5k) was 26%
- maximal value for text 5VU (0.1-1-2.5k-5k) was 27%
- Service CPU usage
- CPU - 2500 records per user for 5 parallel jobs -CPU of mod-users was 199%, for all other modules did not exceed 15%. 5000 or 10k records per user for 5 parallel jobs -CPU of mod-users was 174%, and for all other modules did not exceed 23%.
- RDS CPU utilization did not exceed 36% for 5jobs 2500 records and 49% for 5jobs 5k or 10k records.
Recommendations & Jiras
For further testing Users' bulk editing can be performed with 10k records
...
Infrastructure
PTF -environment
- 9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
- 2 instances of db.r6.xlarge database instances, one reader, and one writer
- MSK ptf-kakfka-1
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=truec
- log.retention.minutes=480
- default.replication.factor=3
...
| Number of virtual user/ Records | 1VU | 2VU | 3VU | 4VU | 5VU |
|---|---|---|---|---|---|
| 100 records | 00:01:05 | 00:01:03 00:01:03 | 00:01:03 00:01:03 00:01:05 | 00:01:03 00:01:03 00:01:03 00:01:04 | 00:01:04 00:01:04. 00:01:04 00:01:04 00:01:04 |
| 1000 records | 00:01:37 | 00:01:59 00:01:35 | 00:02:02 00:02:05 00:01:36 | 00:02:02 00:02:02 00:01:59 00:02:00 | 00:02:11 00:02:13 00:02:13 00:02:05 00:02:06 |
| 2500 records | 00:03:28 | 00:03:30 00:03:31 | 00:03:42 00:03:42 00:01:10(*1) | 00:00:23(*2) 00:03:47 00:03:46 00:03:46 | 00:03:48 00:04:03 00:03:44 00:04:04 00:04:04 |
| 5000 records | 00:05:13 | 00:06:36 (*3) | 00:06:48 00:06:50 00:06:50 (*3) | 00:06:32 00:06:32 00:02:29. 00:02:29 (*3) | 00:06:13 00:06:13 00:06:11 00:06:08 00:06:13 (*3) |
(*1) Duplicates Duplicate barcodes in csv CSV input data, 1200 out of 2500 records were processed
(*2) Index 546 out of bounds for length 13 (ArrayIndexOutOfBoundsException) Jira Legacy server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODEXPW-306
(*3) Was running as a separate test to avoid duplication of input data
Comparison with previous results
| 5VU, Records | Nolana | Orchid |
|---|---|---|
| 2500 | 2 min 9 sec | 4 minutes 4 seconds |
| 5000 records | 3 min 47 sec | 6minutes 13 seconds |
Memory usage
Figure 1 shows memory usage during the testing with 1-5 concurrent and 100-1000-2500-5000 records
...
Figure 3 shows memory usage for the same time range as Figure 2, but only used involved services were selected.
The time range marked with # Several modules were restarted and then data import jobs were running on the environment
...
Figure 3
Instance CPU utilization
Figure 4. CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
Figure 4 The test 5VU(0.1-5VU (100-1000-2500-5000)1-2.5k-5k) was restarted because there were errors in the input data
Figure 5. 2-5VU (5000)CPU instance utilization for 2 up to5 Vu with 5000 records
Service CPU utilization
Figure 6. CPU for all modules did not exceed 79% for all of the tests except the first test for bulk editing with 10k users: mod-users CPU utilization was 157%(at the same time memory grows by 2%) but for all further tests, it was below 80%.
instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
Figure 7. CPU instance utilization for 1 up to 5 Vu with 100-1000-2500-5000 records
RDS CPU utilization
For all tests - RDS CPU utilization did not exceed 59% exceeds 40% (for tests with 10k 5k users records). For the tests with 2500 user records did not exceed 27%.
Database connections
Errors in logs during testing
...
| Code Block |
|---|
2023-03-22T19:47:19.985Z
19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
Field Value
@ingestionTime 1679514442775
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-bulk-operations/bfcbe6d984e1443bb3e2e49dbd14601e
@message 19:47:19 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR oldingsDataProcessor Holdings source was not found by id=null
@timestamp 1679514439985 |
| Code Block |
12023-03-23T17:21:26.756Z
17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
Field Value
@ingestionTime 1679592091556
@log 054267740449:ncp5-folio-eis
@logStream ncp5/mod-data-export-worker/4d440d31a9474250822321c461926b2f
@message 17:21:26 [${FolioLoggingContext:requestid}] [${FolioLoggingContext:tenantid}] [${FolioLoggingContext:userid}] [${FolioLoggingContext:moduleid}] ERROR BulkEditController A job execution for this job is already running: JobExecution: id=8318, version=1, startTime=2023-03-23T17:21:26.700667, endTime=null, lastUpdated=2023-03-23T17:21:26.701196, status=STARTED, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=8318, version=0, Job=[BULK_EDIT_IDENTIFIERS-USER]], jobParameters=[{'jobId':'{value=11cd26e7-93f6-4acf-a85d-a76f71b822a4, type=class java.lang.String, identifying=true}','fileName':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/100-user-barcodes1.csv, type=class java.lang.String, identifying=true}','exportType':'{value=BULK_EDIT_IDENTIFIERS, type=class java.lang.String, identifying=true}','entityType':'{value=USER, type=class java.lang.String, identifying=true}','tempOutputFilePath':'{value=mod-data-export-worker/bulk_edit/11cd26e7-93f6-4acf-a85d-a76f71b822a4/2023-03-23-Matched-Records-100-user-barcodes1, type=class java.lang.String, identifying=true}','identifierType':'{value=BARCODE, type=class java.lang.String, identifying=true}','totalCsvLines':'{value=100, type=class java.lang.Long, identifying=true}'}]
@timestamp 1679592086756 |
Appendix
Infrastructure
PTF -environment ncp5 [ environment name]
...