Bursar (Nolana)


Overview

Bursar workflow was tested with Nolana snapshot. The purpose of this testing is to define baseline of bursar workflow, find possible memory leaks, bottlenecks, etc.

According to numbers mentioned here Feature - Team Responsibility Matrix - we did test bursar workflow from 0 records to 2000 records subsequently.


Infrastructure

PTF -environment (ncp1)

  • m6i.2xlarge EC2 instances located in us-west-2. 
  • 2 instances of db.r6.xlarge database instances, one reader and one writer
  • MSK (ptf-kafka-1 cluster)
    • 4 m5.2xlarge brokers in 2 zones
    • auto.create-topics.enable = true
    • log.retention.minutes=480
    • default.replication.factor=3
    • Apache Kafka v2.8.0
    • EBS storage volume per broker = 300GB
    • Kafka topics 

Memory parameters for relevant modules:

Module

Version

Max Metaspace Size (MB)

XmX (MB)

Soft Limit (MB)

Hard Limit (MB)

CPU

Number of ECS Tasks

mod-data-export-spring1.5.0-SNAPSHOT.585121536184420482561
mod-data-export-worker2.0.25122048 2600307210242
mod-feesfines:18.2.0-SNAPSHOT.13212876889610241282
mod-users
12876889610241282
okapi4.14.45129221360151210243
nginx-okapi2022.03.02--89610241282
pub-okapi2022.03.02--89610241282

High Level Summary

All jobs from 100 records to 2000 records execution times lays between 9-12 seconds. We did perform 69 jobs (all of them ended successfully)

job #Recordstime
110011 s

2

20011s
330010s
44009s
55009s
6100010s
712009s
815009s
917009s
1020009s


Resource Usage

Note: Each spike on CPU graph corresponds to one test. As you can see with higher number of records it requires more CPU usage.

Note: we can observe growing of memory usage on mod-feesfines. It looks like there is a memory leak (actual memory growing from 31% to 43% during set of tests).  Ticket to investigate potential memory leak: PERF-357

Same screenshot with another scale:

Note: Max CPU usage on RDS database is 10%