Folijet - Performance of importing orders on Orchid Snapshot

Owned by Aliaksandr Fedasiuk
Last updated: Mar 16, 2023
2 min read

Modules:

Data Import Module (mod-data-import:2.8.0-SNAPSHOT.258)

Data Import Converter Storage (mod-di-converter-storage:2.1.0-SNAPSHOT.8)

Source Record Manager Module (mod-source-record-manager:3.7.0-SNAPSHOT.759)

Source Record Storage Module (mod-source-record-storage:5.7.0-SNAPSHOT.494)

Inventory Module (mod-inventory:20.1.0-SNAPSHOT.604)

Inventory Storage Module (mod-inventory-storage:26.1.0-SNAPSHOT.826)

Orders Module (mod-orders:12.7.0-SNAPSHOT.654)

Orders Storage Module (mod-orders-storage:13.6.0-SNAPSHOT.302)

Configuration:

Kafka: shared cluster with 2 nodes (replication factor is 2)


mod-data-import: CPU:256/128, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=512m -Xmx1292m

mod-di-converter-storage: CPU:128/128, mem:1024Mi/896Mi,  -XX:MaxMetaspaceSize=128m -Xmx768m

mod-source-record-manager: CPU:1024/512, mem:2048Mi/1844Mi, -XX:MaxMetaspaceSize=800m -Xmx1024m

mod-source-record-storage: CPU:1024/512, mem:1536Mi/1440Mi, -XX:MaxMetaspaceSize=512m -Xmx1024m

mod-inventory: CPU:1024/512, mem:2880Mi/2592Mi, -XX:MaxMetaspaceSize=512m -Xmx1814m

mod-inventory-storage: CPU:1024/512, mem:2208Mi/1952Mi, -XX:MaxMetaspaceSize=512m -Xmx1440m

mod-orders: CPU:1024/512, mem:2048Mi/1440Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders.hprof

mod-orders-storage: CPU:512/256, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-orders-storage.hprof

mod-finance: CPU:256/128, mem:1024Mi/896Mi, db_maxpoolsize: 5, -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance.hprof

mod-finance-storage: CPU:128/128, mem:1024Mi/896Mi, db_maxpoolsize: 5,  -XX:MaxMetaspaceSize=128m -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump/mod-finance-storage.hprof    

mod-configuration: CPU:128/128, mem:1024Mi/896Mi, -XX:MaxMetaspaceSize=128m -Xmx768m


3 poLines per order (1 item)

500 rec

5k rec10k rec

Profile:

pending orders only

"startedDate" : "2023-03-07T07:28:26.132+00:00",

"completedDate" : "2023-03-07T07:33:02.215+00:00",

5min

"startedDate" : "2023-03-13T13:26:11.088+00:00",

"completedDate" : "2023-03-13T14:01:28.452+00:00",

35min

"startedDate" : "2023-03-13T15:05:33.637+00:00",

 "completedDate" : "2023-03-13T16:13:50.101+00:00",

1h 11min

Profile:

open orders &

create instances

"startedDate" : "2023-03-07T17:50:50.265+00:00",

"completedDate" : "2023-03-07T17:56:21.468+00:00",

6min

"startedDate" : "2023-03-07T19:05:22.933+00:00",

"completedDate" : "2023-03-07T19:46:37.662+00:00",

41min (5 errors: NoSuchElementException)

"startedDate" : "2023-03-07T19:56:50.779+00:00",

"completedDate" : "2023-03-07T21:18:07.797+00:00",

1h 49min (12 errors: NoSuchElementException)

Profile:

open orders &

create instances, holdings, items

 "startedDate" : "2023-03-10T10:42:41.518+00:00",

"completedDate" : "2023-03-10T10:47:38.666+00:00",

6min*

"startedDate" : "2023-03-13T17:01:53.549+00:00",

"completedDate" : "2023-03-13T17:45:38.502+00:00",

44min

"startedDate" : "2023-03-10T11:54:02.095+00:00",

"completedDate" : "2023-03-10T13:17:55.613+00:00"

2h*

3 poLines per order (2 items)


Profile:

pending orders only

"startedDate" : "2023-03-15T18:49:46.897+00:00",

"completedDate" : "2023-03-15T18:52:34.898+00:00",

3 min

"startedDate" : "2023-03-15T18:53:40.506+00:00",

"completedDate" : "2023-03-15T19:13:41.364+00:00",

20 min

"startedDate" : "2023-03-16T04:13:29.645+00:00",

"completedDate" : "2023-03-16T04:52:42.887+00:00",

39 min

Profile:

open orders &

create instances

"startedDate" : "2023-03-15T19:15:11.179+00:00",

"completedDate" : "2023-03-15T19:18:07.061+00:00",

3 min

"startedDate" : "2023-03-15T19:24:15.628+00:00",

 "completedDate" : "2023-03-15T19:47:35.427+00:00",

23 min


Profile:

open orders &

create instances, holdings, items

"startedDate" : "2023-03-16T03:37:33.745+00:00",

"completedDate" : "2023-03-16T03:40:47.818+00:00",

3 min*

"startedDate" : "2023-03-16T03:41:57.577+00:00",

"completedDate" : "2023-03-16T04:09:48.357+00:00",

28 min



(*) mark: the 500 records import was finished with 35 errors and 10k records import was imported with 700 errors. All errors look like this:

HttpException: {"message":"ISBN value is invalid","code":"invalidISBN","parameters":[{"key":"isbn","value":"9719429267 (pbk.)"}]}

HttpException: {"message":"ISBN value is invalid","code":"invalidISBN","parameters":[{"key":"isbn","value":"9781405734240 (v.1)"}]}

HttpException: {"message":"ISBN value is invalid","code":"invalidISBN","parameters":[{"key":"isbn","value":"0585098646 (electronic bk.)"}]}



  MODORDERS-864 - Getting issue details... STATUS Task for fix NoSuchElementException


mod-orders 12h:mod-orders-storage 12h: