Modules:
Data Import Module (mod-data-import:2.8.0-SNAPSHOT.258)
Data Import Converter Storage (mod-di-converter-storage:2.1.0-SNAPSHOT.8)
Source Record Manager Module (mod-source-record-manager:3.7.0-SNAPSHOT.759)
Source Record Storage Module (mod-source-record-storage:5.7.0-SNAPSHOT.494)
Inventory Module (mod-inventory:20.1.0-SNAPSHOT.604)
Inventory Storage Module (mod-inventory-storage:26.1.0-SNAPSHOT.826)
Orders Module (mod-orders:12.7.0-SNAPSHOT.654)
Orders Storage Module (mod-orders-storage:13.6.0-SNAPSHOT.302)
Configuration:
Kafka: 2 partition for each topic and replication factor is 2
2 items for each module.
mod-data-import: CPU:256/128, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=512m -Xmx1292m
mod-di-converter-storage: CPU:128/128, mem:1024Mi/896Mi, -XX:MaxMetaspaceSize=128m -Xmx768m
mod-source-record-manager: CPU:1024/512, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=800m -Xmx1024m
mod-source-record-storage: CPU:1024/512, mem:1536Mi/1440Mi, -XX:MaxMetaspaceSize=512m -Xmx1024m
mod-inventory: CPU:1024/512, mem:2880Mi/2592Mi, -XX:MaxMetaspaceSize=512m -Xmx1814m
mod-inventory-storage: CPU:1024/512, mem:2208Mi/1952Mi, -XX:MaxMetaspaceSize=512m -Xmx1440m
mod-orders: CPU:1024/512, mem:2048Mi/1440Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders.hprof
mod-orders-storage: CPU:512/256, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders-storage.hprof
mod-finance: CPU:256/128, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance.hprof
mod-finance-storage: CPU:128/128, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance-storage.hprof
mod-configuration: CPU:128/128, mem:1024Mi/896Mi, -XX:MaxMetaspaceSize=128m -Xmx768m
| 3 poLines per order (1 item) | 5k rec | 10k rec | 30k rec |
|---|---|---|---|
Profile: Default create instance & SRS MARC BIB | "startedDate" : "2023-03-15T17:22:44.416+00:00", "completedDate" : "2023-03-15T17:27:22.393+00:00", 3 min | "startedDate" : "2023-03-15T17:28:15.385+00:00", "completedDate" : "2023-03-15T17:39:07.001+00:00", 11 min | "startedDate" : "2023-03-15T17:40:05.862+00:00", "completedDate" : "2023-03-15T18:12:30.559+00:00", 32 min |
(*) mark: the 500 records import was finished with 35 errors and 10k records import was imported with 700 errors. All errors look like this:
HttpException: {"message":"ISBN value is invalid","code":"invalidISBN","parameters":[{"key":"isbn","value":"9719429267 (pbk.)"}]}
HttpException: {"message":"ISBN value is invalid","code":"invalidISBN","parameters":[{"key":"isbn","value":"9781405734240 (v.1)"}]}
HttpException: {"message":"ISBN value is invalid","code":"invalidISBN","parameters":[{"key":"isbn","value":"0585098646 (electronic bk.)"}]}