Skip to end of banner
Go to start of banner

Folijet - Performance of importing MARC BIB on Orchid Snapshot (Draft)

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 16 Next »

Modules:

Data Import Module (mod-data-import:2.8.0-SNAPSHOT.258)

Data Import Converter Storage (mod-di-converter-storage:2.1.0-SNAPSHOT.8)

Source Record Manager Module (mod-source-record-manager:3.7.0-SNAPSHOT.759)

Source Record Storage Module (mod-source-record-storage:5.7.0-SNAPSHOT.494)

Inventory Module (mod-inventory:20.1.0-SNAPSHOT.604)

Inventory Storage Module (mod-inventory-storage:26.1.0-SNAPSHOT.826)

Orders Module (mod-orders:12.7.0-SNAPSHOT.654)

Orders Storage Module (mod-orders-storage:13.6.0-SNAPSHOT.302)

Configuration:

Kafka: 2 partitions for each topic and the replication factor is 2

2 items for each module.


mod-data-import: CPU:256/128, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=512m -Xmx1292m

mod-di-converter-storage: CPU:128/128, mem:1024Mi/896Mi,  -XX:MaxMetaspaceSize=128m -Xmx768m

mod-source-record-manager: CPU:1024/512, mem:2048Mi/1844Mi, db_maxpoolsize: 15, -XX:MaxMetaspaceSize=800m -Xmx1024m

mod-source-record-storage: CPU:1024/512, mem:1536Mi/1440Mi, db_maxpoolsize: 15, -XX:MaxMetaspaceSize=512m -Xmx1024m

mod-inventory: CPU:1024/512, mem:2880Mi/2592Mi, -XX:MaxMetaspaceSize=512m -Xmx1814m

mod-inventory-storage: CPU:1024/512, mem:2208Mi/1952Mi, -XX:MaxMetaspaceSize=512m -Xmx1440m

mod-orders: CPU:1024/512, mem:2048Mi/1440Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders.hprof

mod-orders-storage: CPU:512/256, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders-storage.hprof

mod-finance: CPU:256/128, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance.hprof

mod-finance-storage: CPU:128/128, mem:1024Mi/896Mi, db_maxpoolsize: 5,  -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance-storage.hprof    

mod-configuration: CPU:128/128, mem:1024Mi/896Mi, -XX:MaxMetaspaceSize=128m -Xmx768m


Increase connection pool:

mod-source-record-manager: CPU:2048/1024, mem:5600Mi/5000Mi, db_maxpoolsize: 100, -XX:MaxMetaspaceSize=800m -Xmx5000m

mod-source-record-storage: CPU:2048/1024, mem:5600Mi/5000Mi, db_maxpoolsize: 100, -XX:MaxMetaspaceSize=512m -Xmx5000m



5k rec

10k rec30k rec50k rec

Profile:

Default create instance & SRS MARC BIB

"startedDate" : "2023-03-15T17:22:44.416+00:00",

"completedDate" : "2023-03-15T17:27:22.393+00:00",

3 min

"startedDate" : "2023-03-15T17:28:15.385+00:00",

"completedDate" : "2023-03-15T17:39:07.001+00:00",

11 min

"startedDate" : "2023-03-15T17:40:05.862+00:00",

"completedDate" : "2023-03-15T18:12:30.559+00:00",

32 min


Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-16T15:33:10.726+00:00",

"completedDate" : "2023-03-16T15:38:55.351+00:00",

5 min/(MG: 7min)

"startedDate" : "2023-03-16T15:41:39.053+00:00",

"completedDate" : "2023-03-16T15:55:22.105+00:00",

14 min/(MG: 16min)

"startedDate" : "2023-03-16T18:38:24.782+00:00",

"completedDate" : "2023-03-16T19:24:57.711+00:00",

46 min(*)


Julian changes in inventory-storage



Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-16T20:05:48.739+00:00",

"completedDate" : "2023-03-16T20:13:20.760+00:00",

8 min/(MG: 7min)

"startedDate" : "2023-03-16T20:13:52.860+00:00",

"completedDate" : "2023-03-16T20:30:27.189+00:00",

17 min/(MG: 16min)

a lot of StacklessClosedChannelException and Timeouts exceptions
Increase DB connection pool (with inventory-storage changes)



Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-23T10:10:09.297+00:00",

"completedDate" : "2023-03-23T10:23:39.168+00:00",

13 min/(MG: 7min)

"startedDate" : "2023-03-23T10:24:31.073+00:00",

"completedDate" : "2023-03-23T10:43:41.657+00:00",

19 min/(MG: 16min)

"startedDate" : "2023-03-23T10:57:11.525+00:00",

"completedDate" : "2023-03-23T11:39:07.009+00:00",

42 min

"startedDate" : "2023-03-23T11:58:47.557+00:00",

"completedDate" : "2023-03-23T12:52:03.823+00:00",

54 min/(MG: 59min)

Increase DB connection pool (without inventory-storage changes)



Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-23T15:45:54.863+00:00",

"completedDate" : "2023-03-23T15:51:36.077+00:00",

6 min/(MG: 7min)

"startedDate" : "2023-03-23T16:50:59.795+00:00",

"completedDate" : "2023-03-23T17:04:49.625+00:00",

14 min/(MG: 16min)


"startedDate" : "2023-03-23T17:07:32.020+00:00",

"completedDate" : "2023-03-23T18:01:08.354+00:00",

54 min/(MG: 59min)

Increase DB connection pool (with record-storage changes)





Profile:

BF Default: MARC file import with creating of new instance, holding and item

"startedDate" : "2023-03-23T19:11:30.875+00:00",

"completedDate" : "2023-03-23T19:19:50.519+00:00",

8 min/(MG: 7min)

"startedDate" : "2023-03-24T09:29:52.948+00:00",

"completedDate" : "2023-03-24T09:47:49.251+00:00",

18 min/(MG: 16min)


"startedDate" : "2023-03-24T09:50:29.560+00:00",

"completedDate" : "2023-03-24T11:09:18.798+00:00",

1h 19min/(MG: 59min)


(*) mark:  1 error in Item creation: io.netty.channel.StacklessClosedChannelException











  • No labels