Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Jira Legacy
serverSystem JiraJIRA
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyPERF-144

...

  • 61 back-end modules deployed in 110 ECS services
  • 3 okapi ECS services
  • 8 m5.large  EC2 instances
  • 2 db.r5.xlarge AWS RDS instance (1 reader, 1 writer)

Software version

mod-oai-pmh 3.4.2

Summary 

We're able to harvest all data set we have, which is 7.2 M records with different "Max records per response" parameter value.

Possibly there's a memory leak on oai-pmh side as we've seen continuously growing memory and CPU usage (see screenshots below).


Tests and results


testMax records per responseTime to completeResultIssues
11006 hours 26 minutesall data harvestedGrowing CPU/RAM usage
23002 hours 31 min5.5 M records harvestedconnection lost with load generator
not an oai-pmh issue
35002 hour 27 minall data harvestedGrowing CPU/RAM usage


Service CPU usage

Image Added


Service Memory usage

Image Added


Source-record-storage memory usage

Image Added


Source-record-storage CPU usage


Image Added



Source-record-manager memory usage

Image Added


Source-record-manager CPU usage

Image Added


mod-inventory-storage CPU usage

Image Added



mod-inventory-storage memory usage


Image Added


Heap Analysis

There's 2 issues as a leak suspects common for each heap dumps taken (after each test):

io.vertx.core.http.impl.HttpClientImpl:

There is growing instances number 7 347 → 13 248 → 20 664;

Image Added

Image Added

Image Added




io.vertx.core.http.impl.ConnectionManager:

There is growing instances number 14 694 →26 514 → 41 328;

Image Added

Image Added

Image Added