Modules:
Data Import Module (mod-data-import:2.8.0-SNAPSHOT.258)
Data Import Converter Storage (mod-di-converter-storage:2.1.0-SNAPSHOT.8)
Source Record Manager Module (mod-source-record-manager:3.7.0-SNAPSHOT.759)
Source Record Storage Module (mod-source-record-storage:5.7.0-SNAPSHOT.494)
Inventory Module (mod-inventory:20.1.0-SNAPSHOT.604)
Inventory Storage Module (mod-inventory-storage:26.1.0-SNAPSHOT.826)
Configuration:
Kafka: 2 partitions for each topic and the replication factor is 2
2 items for each module.
mod-data-import: CPU:256/128, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=512m -Xmx1292m
mod-di-converter-storage: CPU:128/128, mem:1024Mi/896Mi, -XX:MaxMetaspaceSize=128m -Xmx768m
mod-source-record-manager: CPU:1024/512, mem:2048Mi/1844Mi, db_maxpoolsize: 15, -XX:MaxMetaspaceSize=800m -Xmx1024m
mod-source-record-storage: CPU:1024/512, mem:1536Mi/1440Mi, db_maxpoolsize: 15, -XX:MaxMetaspaceSize=512m -Xmx1024m
mod-inventory: CPU:1024/512, mem:2880Mi/2592Mi, -XX:MaxMetaspaceSize=512m -Xmx1814m
mod-inventory-storage: CPU:1024/512, mem:2208Mi/1952Mi, -XX:MaxMetaspaceSize=512m -Xmx1440m
| Create Profile | Update Profile |
|---|---|
|
|
5k rec | 10k rec | 50k rec | 100k rec | |
|---|---|---|---|---|
Profile: Create | "startedDate" : "2023-05-01T14:18:08.828+00:00", "completedDate" : "2023-05-01T14:32:59.472+00:00" 14 min | "startedDate" : "2023-05-01T15:33:55.004+00:00", "completedDate" : "2023-05-01T16:04:15.136+00:00", 31 min | "startedDate" : "2023-05-01T17:26:36.552+00:00", "completedDate" : "2023-05-01T18:59:33.557+00:00", 1h 33min | |
Profile: Update | "startedDate" : "2023-05-01T14:58:59.770+00:00", "completedDate" : "2023-05-01T15:31:51.023+00:00" 33 min | "startedDate" : "2023-05-01T16:25:12.856+00:00", "completedDate" : "2023-05-01T17:26:05.342+00:00", 1h 1min | Timeout |
Create Index:
Size: marc_records_lb = 9`874`818
ERRORS:
2023-05-02 09:24:12.668 [vert.x-worker-thread-2] [] [] [] [] ERROR KafkaConsumerWrapper businessHandlerCompletionHandler:: Consumer - id: 34 subscriptionPattern: SubscriptionDefinition(eventType=DI_COMPLETED, subscriptionPattern=folio-perf-folijet\.Default\.\w{1,}\.DI_COMPLETED) Error while commit offset: 123274
org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets {folio-perf-folijet.Default.fs09000000.DI_COMPLETED-1=OffsetAndMetadata{offset=123274, leaderEpoch=null, metadata=''}}