Skip to end of banner
Go to start of banner

Folijet - Performance of importing MARC BIB on Orchid Snapshot (Draft)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Modules:

Data Import Module (mod-data-import:2.8.0-SNAPSHOT.258)

Data Import Converter Storage (mod-di-converter-storage:2.1.0-SNAPSHOT.8)

Source Record Manager Module (mod-source-record-manager:3.7.0-SNAPSHOT.759)

Source Record Storage Module (mod-source-record-storage:5.7.0-SNAPSHOT.494)

Inventory Module (mod-inventory:20.1.0-SNAPSHOT.604)

Inventory Storage Module (mod-inventory-storage:26.1.0-SNAPSHOT.826)

Orders Module (mod-orders:12.7.0-SNAPSHOT.654)

Orders Storage Module (mod-orders-storage:13.6.0-SNAPSHOT.302)

Configuration:

Kafka: 2 partition for each topic and replication factor is 2

2 items for each module.


mod-data-import: CPU:256/128, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=512m -Xmx1292m

mod-di-converter-storage: CPU:128/128, mem:1024Mi/896Mi,  -XX:MaxMetaspaceSize=128m -Xmx768m

mod-source-record-manager: CPU:1024/512, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=800m -Xmx1024m

mod-source-record-storage: CPU:1024/512, mem:1536Mi/1440Mi, -XX:MaxMetaspaceSize=512m -Xmx1024m

mod-inventory: CPU:1024/512, mem:2880Mi/2592Mi, -XX:MaxMetaspaceSize=512m -Xmx1814m

mod-inventory-storage: CPU:1024/512, mem:2208Mi/1952Mi, -XX:MaxMetaspaceSize=512m -Xmx1440m

mod-orders: CPU:1024/512, mem:2048Mi/1440Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders.hprof

mod-orders-storage: CPU:512/256, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders-storage.hprof

mod-finance: CPU:256/128, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance.hprof

mod-finance-storage: CPU:128/128, mem:1024Mi/896Mi, db_maxpoolsize: 5,  -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance-storage.hprof    

mod-configuration: CPU:128/128, mem:1024Mi/896Mi, -XX:MaxMetaspaceSize=128m -Xmx768m



5k rec

10k rec30k rec

Profile:

Default create instance & SRS MARC BIB

"startedDate" : "2023-03-15T17:22:44.416+00:00",

"completedDate" : "2023-03-15T17:27:22.393+00:00",

3 min

"startedDate" : "2023-03-15T17:28:15.385+00:00",

"completedDate" : "2023-03-15T17:39:07.001+00:00",

11 min

"startedDate" : "2023-03-15T17:40:05.862+00:00",

"completedDate" : "2023-03-15T18:12:30.559+00:00",

32 min

Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-16T15:33:10.726+00:00",

"completedDate" : "2023-03-16T15:38:55.351+00:00",

5 min

"startedDate" : "2023-03-16T15:41:39.053+00:00",

"completedDate" : "2023-03-16T15:55:22.105+00:00",

14 min

"startedDate" : "2023-03-16T18:38:24.782+00:00",

"completedDate" : "2023-03-16T19:24:57.711+00:00",

46 min(*)

Julian changes in inventory-storage


Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-16T20:05:48.739+00:00",

"completedDate" : "2023-03-16T20:13:20.760+00:00",

8 min

"startedDate" : "2023-03-16T20:13:52.860+00:00",

"completedDate" : "2023-03-16T20:30:27.189+00:00",

17 min



(*) mark:  1 error in Item creation: io.netty.channel.StacklessClosedChannelException











  • No labels