Table of Contents

...

Ticket: 

Summary

• There is significant performance improvement for data import in Poppy with file splitting feature compared with Orchid (40% for DI Create, 25% for DI Update) but small degradation (up to 5%) compared to Poppy without file splitting feature running with CICO. CO response times are almost the same as Poppy without file splitting feature. CI response time is 20% slower with and without Data Import.
• Average CPU utilization did not exceed 150% for all the modules. The highest consumption was observed from mod-inventory. It was growing from 110% up to 250% at the end of the test (So as memory grows too, we can suspect the issue

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODINV-944

  . It is fixed in version 20.1.9 but this test was run on version 20.1.7 of mod-inventory). Spikes of mod-data-import were observed in Data Import jobs with 50k files up to 130%. for jobs and a 250% spike for 100k. For Data Import jobs CPU utilization didn't exceed 110% for all other modules
• Memory utilization increase is a result of previous modules restarting (everyday cluster shutdown process). Memory consumption before tests for mod-search was 45% and for mod-inventory - 55%. During the test with 100k files mod-search memory utilization increases to 90% and mod-inventory up to 100%.
• Average DB CPU usage during data import is about 95% The same results compared to the same tests in Orchid.
• Average connection count during data import is about 600 connections for Create jobs which is two times higher than without file splitting feature. For Update jobs - 560 connections

Test Runs 

...

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 2 database  instances, writer/reader

  
  

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...