Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
outlinetrue

...

  1. These tests are run to investigate the performance difference of mod-data-export with logLevel=info vs logLevel=warn for Juniper release. This testing was part of
    Jira Legacy
    serverSystem Jira
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId01505d01-b853-3c2e-90f1-ee9b165564fc
    keyMDEXP-394
    where we observed that mod-data-export is continuously writing a lot of data to log. For 100K DE job, mod-data-export is writing 42Million records. This could also result in a crash if there is not enough CPU, memory allocated to mod-data-export.
  2. In mod-data-export log4j2.properties file was modified with rootLogger.level = warn, logger.netty.level = warn, status = warn


Backend:

  • mod-data-export-4.1.1 (snapshot version)
  • mod-source-record-storage-5.1.4
  • mod-source-record-manager-3.1.3
  • okapi-4.8.2
  • mod-authtoken-2.8.0

...

  • 8 million inventory records
  • 74 FOLIO back-end modules deployed in 144 ECS services
  • 3 okapi ECS services
  • 12 m5.large  EC2 instances
  • writer db.r6g.xlarge 1 reader db.r6g.xlarge AWS RDS instance
  • INFO logging level / WARN looging level

...

  1. With WARN level logging, seeing 9% improvement in memory utilization
  2. With WARN level logging, No bumps in the memory were observed. Memory utilization stays stable for multiple Data Export job runs.A new feature added for data-export is it will kill failed job every 6 hours
  3. Improvement by 42Million records for WARN. 42 Million fewer records were written.
  4. How many data-export jobs can run in parallel? Multiple jobs can run in parallel and 2.75 Million instance records can be exported at any given point in time. If we try to export more than that for example 3 Million instance records, mod-data-export crashes with OOM(Out Of Memory)

...