| Table of Contents | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
Test # | Data-import test | Duration Poppy | Duration Quesnelia | Difference, sec | Results | |
|---|---|---|---|---|---|---|
1. | 1k MARC BIB Create | PTF - Create 2 | 39 sec | 54 sec | ↓ 15 sec | Completed |
2. | 5k MARC BIB Create | PTF - Create 2 | 2 min 22 sec | 3 min 20 sec | ↓ 58 sec | Completed |
3. | 10k MARC BIB Create | PTF - Create 2 | 4 min 29 sec | 6 minutes | ↓ 1 min 31 sec | Completed |
4. | 25k MARC BIB Create | PTF - Create 2 | 10 min 38 sec | 13 min 41 sec | ↓ 3 min 3 sec | Completed |
5. | 50k MARC BIB Create | PTF - Create 2 | 20 min 26 sec | 21 min 59 sec | ↓ 1 min 33 sec | Completed |
6. | 100k MARC BIB Create | PTF - Create 2 | 2 hours 46 min Cancelled | 40 min 16 sec | Not applicable | Completed |
7. | 500k MARC BIB Create | PTF - Create 2 | Not tested | 3 hours 27 min | Not applicable | Completed |
8. | 1k MARC BIB Update | PTF - Updates Success - 6 | 34 sec (PTF - Updates Success - 1) | 1 min 59 sec | Not applicable | Completed |
9 | 2k MARC BIB Update | PTF - Updates Success - 6 | 1 min 09 sec (PTF - Updates Success - 1) | 2 min 43 sec | Not applicable | Completed |
10 | 5k MARC BIB Update | PTF - Updates Success - 6 | 2 min 31 sec (PTF - Updates Success - 1) | 7 min 10 sec | Not applicable | Completed |
11 | 10k MARC BIB Update | PTF - Updates Success - 6 | 5 min 13 sec (PTF - Updates Success - 1) | 10 min 27 sec | Not applicable | Completed |
12 | 25k MARC BIB Update | PTF - Updates Success - 6 | 12 min 27 sec (PTF - Updates Success - 1) | 23 min 16 sec | Not applicable | Completed |
13 | 50k MARC BIB Update | PTF - Updates Success - 6 | Not tested | 40 min 52 sec | Not applicable | Completed |
14 | 100k MARC BIB Update | PTF - Updates Success - 6 | Not tested | 1 hrs 2 min | Not applicable | Completed |
15 | 500k MARC BIB Update | PTF - Updates Success - 6 | Not tested | 5 hrs 31 min | Not applicable | Completed |
Test Runs
MARC BIB CREATE
...
| Code Block |
|---|
fields @timestamp, @message, @logStream, @log ############ DISABLED MODULES | filter @logStream not like "mod-remote-storage" and @logStream not like "edge-caiasoft" and @logStream not like "mod-graphql" and @logStream not like "edge-ncip" and @logStream not like "mod-rtac" | filter @logStream not like "mod-finance" and @logStream not like "mod-inn-reach" ############ DISABLED MODULES | filter @message like "Error" or @message like "error" or @message like "ERROR" #| filter @message like "Exception" or @message like "exception" or @message like "WARN" ############ mod-authtoken/ | filter @message not like "Invalid token" and @message not like "Token validation failure: Access token has expired" and @message not like "Access token has expired" ############ okapi-b , nginx-okapi | filter @message not like "statusAny=ERROR" and @message not like "No suitable module found for path and tenant" and @message not like "Removing connection to endpoint" and @message not like "ErrorTypeException: 503:" | filter @message not like "error_page" ############ mod-search | filter @message not like "invokeBatchErrorHandler" and @message not like "ErrorHandlingUtils" and @message not like "FallbackBatchErrorHandler" and @message not like "DefaultErrorHandler" and @message not like "errorFromXContent" ############ mod-data-export-spring | filter @message not like "java.lang.NoClassDefFoundError: io/vertx/core/Vertx" and @message not like "ERROR StatusConsoleListener Resolver failed to lookup FolioLoggingContext" ############ mod-inventory ---- Instances contains error in title | filter @message not like "INFO cceedingTitlesHelper createPrecedingSucceedingTitles" ########### mod-pub-sub | filter @message not like "TopicExistsException" and @message not like "Some of the topics" ########### OTHER ERRORS | filter @message not like "Failed to index resource event" | filter @message not like "FeignException.errorStatus" and @message not like "FeignException.clientErrorStatus" and @message not like "/var/log/nginx/error" | filter @message not like "ErrorDecoder$Default.decode" and @message not like "InvocationContext.decodeError" and @message not like "Records indexed to elasticsearch" | filter @message not like "HeapDumpOnOutOfMemoryError" and @message not like "while connecting to upstream" and @message not like "TopicExistsException" and @message not like "main ERROR Unrecognized" | sort @timestamp desc | limit 5000 |
Appendix
Infrastructure
PTF -environment qcp1
10 m6i11 m6i.2xlarge EC2 instances located in in US East East (N. Virginia)us-east-1
2 database instances, writer/reader
NameMemory GIBvCPUsmax_connections
db.r6g.xlarge
32 GiB4 vCPUs2731
MSK tenant
4 [Number of ECS instances, instance type, location region]1 instance of db.r6.xlarge database instance: Writer instance
MSK - tenat
4 kafka.m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
auto.create.topics.enable=true
log.retention.minutes=480
default.replication.factor=3
...