Data Import test report (Nolana)

Owned by Roman_Fedynyshyn
Last updated: Aug 17, 2023 by Martin Tran
7 min read

It's been found after testing that the actual durations of the imports performed were about 2 (two) times longer than what was reported. This is due to the PTF environment missing a DB trigger that, when restored, doubled the imports' durations.



Overview

This document contains the results of testing Data Import for MARC Bibliographic records in the Nolana release to detect performance trends. PERF-341 - Getting issue details... STATUS

The figures achieved in PTF performance testing have not been achieved in Nolana Bugfest. Developers are reviewing the results to determinethe causes for the differences, in MODDATAIMP-752 - Getting issue details... STATUS

Infrastructure

  • 10 m6i.2xlarge EC2 instances  
  • 2 instances of db.r6.xlarge database instances, one reader and one writer
  • MSK
    • 4 m5.2xlarge brokers in 2 zones 
    • auto.create-topics.enable = true
    • log.retention.minutes=480
    • 2 partitions per DI topics
    • default.replication.factor=3
  • mod-inventory memory
    • 1024 CPU units, 2592MB mem
    • inventory.kafka.DataImportConsumerVerticle.instancesNumber=10
    • inventory.kafka.MarcBibInstanceHridSetConsumerVerticle.instancesNumber=10
    • kafka.consumer.max.poll.records=10
  • mod-inventory-storage
    • 1024 CPU units, 1962MB mem
  • mod-source-record-storage
    • 1024 CPU units, 1440MB mem
  • mod-source-record-manager
    • 1024 CPU units, 3688MB mem
  • mod-data-import
    • 256 CPU units, 1844MB mem
  • mod-data-import-cs 
    • 128 CPU units, 896MB mem

Software versions

  • mod-data-import v2.6.1
  • mod-data-import-converter-storage v1.15.1
  • mod-source-record-manager v3.5.0
  • mod-source-record-storage v5.5.2
  • mod-inventory v19.0.1
  • mod-inventory-storage v25.0.1

Summary

  • Data Import in Nolana has more or less the same DI durations as Morning Glory. For instance it's +20 seconds for 10K creation, however it's - 40 s for updates, for 50K records it's +2 minutes on creation and -2 minutes for updates.
  • One issue was detected it's MODSOURMAN-908 .  This is deadlocks in database which make DI work slow (when issue happens on 50 K duration increases up to 6 hours).
    • After MODSOURMAN-908 was fixed - we were not able to reproduce this deadlock issue.
  • R/W Split Enabled:
    • For most of tests we can see an improvement of DI duration. For example 10K create with R/W split - 3m 43s, without R/W split it's 4m 55 s.
    • We can see that RDS CPU usage on writer node is even higher than it was without read/write split enable.
    • With R/W split for data import creates/updates - reader node took on 15-17% of DB load.
  • MARC BIB Update and Create take less time for Nolana with new version of DI modules*.  PERF-388 - Getting issue details... STATUS reproduced for 10k MARC BIB Update without Check-In/Check-Out. For job started after 5k MARC BIB Update with less than a minute timeout between jobs.

Results


Profile

Duration

Nolana with new version of modules*

Duration

Nolana

Duration

Morning Glory

Duration Lotus

1K MARC CreatePTF - Create 240 s46 s50 s1 min 9 s

1K MARC Update

PTF - Updates Success - 135 s50 s39s1 min 30 s
2K MARC CreatePTF - Create 256 s1 min 13 s1 min 2s1 min 34 s
2K MARC UpdatePTF - Updates Success - 158 s1 min 7 s1 min 11 s1 min 54 s

5K MARC Create

PTF - Create 22m 8 s2 min 51 s2 min 20s

3 min 54 s

5K MARC UpdatePTF - Updates Success - 12 min 10 s2 min 27s3 min 4 s4 min 12 s

10K MARC Create 

PTF - Create 24 min 20 s4 min 55 s4 min 33 s

6 min 45 s

10K MARC Update PTF - Updates Success - 14 min 8 s4 min 50 s5 min 29 s8 min 4 s
25K MARC CreatePTF - Create 210 min 41 s11 min 56 s10 min 55 s16 min 8s
25K MARC UpdatePTF - Updates Success - 110 min 40 s12 min 20 s13 min 37 s19 min 50s
50K MARC CreatePTF - Create 221 min 11 s23 min 43 s21 min 37 s32 min 28 s

50K MARC Update

PTF - Updates Success - 120 min 57 s24 min 5 s26 min 10 s39 min 5 s
100K MARC CreatePTF - Create 242 min 35 s49 min 40 s44 min 4 s1 hr 11 min

100K MARC Update

PTF - Updates Success - 141 min 56 s51 min 15 s55 min 33 s1 hr 19 min
500K MARC CreatePTF - Create 2DNR

4 hr 2 min

Completed with errors*

3 hr 55 min

Completed with errors*

7 hr 4 min (Completed with errors)*
  • 500K MARC Create import failed due to 500K records file corruption. 
  • So far we can only compare results of PTF-Create-2 job profile, while Update-success-2 is not available on our Morning-Glory env. 

* - Modules and versions comparison table

Nolananew versionold version
mod-data-import2.6.22.6.1
mod-data-import-converter-storage1.15.21.15.1
mod-source-record-manager 3.5.63.5.0
mod-source-record-storage5.5.25.5.2
mod-inventory19.0.219.0.1
mod-inventory-storage25.0.325.0.1


Resources usage MARC Create



Note: CPU usage for mod-date-import-cs in Nolana release is less that it was for Morning Glory.

For MG release it was ±320%, and for Nolana it's less then 100% 


Note: We can observe increasing of memory usage for most related modules and it wasn't released even days later. Most disturbing part here is mod-data-import. It increases memory level from ±20% to 80%. 

We'll investigate potential memory leak in future. PERF-358

Update: After additional investigation in scope of PERF-358 we can say that there is no memory leaks in any of the modules. For most of them memory will grow up to some point and then stop growing. See details in ticket. 




Note: RDS CPU usage stays the same as was in previous releases ±80%.


Resources usage MARC Update


Service CPU usage


Service memory usage 



Instance level CPU usage

Note: In comparison to Morning Glory release instance CPU level is higher by 20% on one of the instances. However it can be explained with another distribution of containers on instances. 



100 K Update

Service CPU usage


Service memory usage

Note: Here is the same growing of memory usage as was in previous tests, ticket to investigate: PERF-358

Update: After additional investigation in scope of PERF-358 we can say that there is no memory leaks in any of the modules. For most of them memory will grow up to some point and then stop growing. See details in ticket. 


Instance level CPU usage


DB CPU usage


Read Write Split enabled 

Note:R/W was enabled for: 

  • mod-inventory-storage
  • mod-source-record-manager
  • source-record-storage
  • mod-data-import
  • mod-data-import-cs



Profile

Duration With RW split

Duration

Nolana

1K MARC CreatePTF - Create 248 s46 s

1K MARC

Update

PTF - Updates Success - 1

1 min 1 s

50 s
2K MARC CreatePTF - Create 21 min 9 s1 min 13 s
2K MARC UpdatePTF - Updates Success - 158 s1 min 7 s

5K MARC Create

PTF - Create 22 min 4 s2 min 51 s
5K MARC UpdatePTF - Updates Success - 12 min 7 s2 min 27s

10K MARC Create 

PTF - Create 23 min 43 s4 min 55 s
10K MARC Update PTF - Updates Success - 14 min 8 s 4 min 50 s
25K MARC CreatePTF - Create 28 min 58 s11 min 56 s
25K MARC UpdatePTF - Updates Success - 110 min 46 s12 min 20 s
50K MARC CreatePTF - Create 218 min 2 s23 min 43 s

50K MARC

Update

PTF - Updates Success - 122 min 22 s24 min 5 s
100K MARC CreatePTF - Create 235 min 4 s49 min 40 s

100K MARC

Update

PTF - Updates Success - 150 min 19 s51 min 15 s


Resources usage "creates"

Service CPU usage

CPU usage create RW


Service Memory Utilization 

Note: All tests were performed on "fresh" modules this can explain growing memory usage on modules. They will stop growing memory according to PERF-358


Note: During "create" imports with R/W split - we can see that there was 15% load on reader DB node.

Note: Each spike on this chart corresponding to each DI test performed.



100K create



Resources usage "updates"




Note: With R/W split for data import updates - reader node took on 15-17% of DB load.

Note: Each spike on this chart corresponding to each DI test performed.