Table of Contents

Overview

...

Ticket: 

Summary

• There is significant performance improvement for in data import performance in Poppy with Poppy using the file splitting feature compared with to Orchid (40% for DI Create, 25% for DI Update) but . However, there is a small degradation (up to 5%) compared to Poppy without the file splitting feature when running with CICO. CO response times are almost the same as identical to Poppy without the file splitting feature. The CI response time is 20% slower with and without Data Import.
• Average CPU utilization did not exceed 150% for all the modules. The highest consumption was observed from mod-inventory. It was growing from 110% up to 250% at the end of the test (So as memory grows too, we can suspect the issue

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODINV-944

  . It is fixed in version 20.1.9 but this test was run on version 20.1.7 of mod-inventory). Spikes of mod-data-import were observed in Data Import jobs with 50k files up to 130%. for jobs and a 250% spike for 100k. For Data Import jobs CPU utilization didn't exceed 110% for all other modules
• Memory utilization increase is a result of previous modules restarting (everyday cluster shutdown process). Memory consumption before tests for mod-search was 45% and for mod-inventory - 55%. During the test with 100k files mod-search memory utilization increases to 90% and mod-inventory up to 100%.
• Average DB CPU usage during data import is about 95% The same results compared to 95% which is consistent with the performance observed during the same tests in Orchid.
• Average The average connection count during data import is about approximately 600 connections for Create jobs, which is two times higher than without twice as high as when the file splitting feature is disabled. For Update jobs - 560 connections, the connection count is 560.

Test Runs 

...

Infrastructure

PTF -environment pcp1

• 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1

• 1 database  instance, writer

  
  

  Name	Memory GIB	vCPUs	max_connections
  db.r6g.xlarge	32 GiB	4 vCPUs	2731

  
  

• MSK cluster - tenant
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3

...