...

PERF-526 - NLA stress test

• It is recommended to increase database size to 2x large: General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

• Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                       "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

  {                     "name": "JAVA_OPTS",                     "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m"                 },

  
  

• If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid PERF-582 - Response 500 for NLA normal load test with Data Import job (2 SP) BLOCKED :

...

a lot of (this queries taking more than 1-2 seconds to execute)

SELECT COUNT(*) FROM fs00001069_mod_inventory_storage Open question - which workflow is calling that taking 

• SELECT COUNT(*) FROM fs00001069_mod_inventory_storage.instance (AVG latency up to 2359.19 ms 0.03 calls/sec)
• SELECT COUNT(*) FROM fs00001069_mod_inventory_storage.holdings_record (AVG latency up to 1465.36 ms 0.03 calls/sec)
• SELECT COUNT(*) FROM fs00001069_mod_inventory_storage.item (AVG latency up to 1580.26 ms 0.03 calls/sec)

SELECT fs00001069_mod_organizations_storage.count_estimate(?) (AVG latency up to 261.67 ms 0.20 calls/sec)

long one

2023-07-06 23:19:28 UTC:10.23.10.189(47150):fs00001069_mod_finance_storage@folio:[16764]:LOG: duration: 1808067.854 ms statement: DO
$$
begin
PERFORM fs00001069_mod_finance_storage.budget_encumbrances_rollover('
{ "id": "f2803540-7d7c-4b2a-8068-40dcd0725d16", "ledgerId": "cb8ba034-a78c-46f6-b41c-c0faec101c39", "rolloverType": "Preview", "fromFiscalYearId": "2284fa2a-926c-49bb-b949-0a791fd8353b", "toFiscalYearId": "bad9ec7d-186c-48c4-a64d-a23676044eab", "restrictEncumbrance": true, "restrictExpenditures": true, "needCloseBudgets": true, "currencyFactor": 2, "budgetsRollover": [ { "rolloverAllocation": true, "rolloverBudgetValue": "Available", "setAllowances": false, "adjustAllocation": 0, "addAvailableTo": "Available" } ], "encumbrancesRollover": [ { "orderType": "Ongoing", "basedOn": "InitialAmount", "increaseBy": 0 }, { "orderType": "Ongoing-Subscription", "basedOn": "InitialAmount", "increaseBy": 0 }, { "orderType": "One-time", "basedOn": "InitialAmount", "increaseBy": 0 } ], "metadata": { "createdDate": "2023-07-06T22:49:20.492+00:00", "createdByUserId": "5f31c651-e0cd-4b1a-9acc-b5d1a0d160ce", "updatedDate": "2023-07-06T22:49:20.492+00:00", "updatedByUserId": "5f31c651-e0cd-4b1a-9acc-b5d1a0d160ce" } }');
end;
$$ LANGUAGE plpgsql;

Appendix

Infrastructure

PTF -environment nptf 

• 8 m6g.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
• 1 instance of db.r6.xlarge,db.r6.2xlarge,db.r6.4xlarge database (depending on a test)
• MSK ptf-kakfa-3
  
  • 4 m5.2xlarge brokers in 2 zones

  • Apache Kafka version 2.8.0

  • EBS storage volume per broker 300 GiB

  • auto.create.topics.enable=true
  • log.retention.minutes=480
  • default.replication.factor=3
• mod-source-record-storage DB_CONNECTION_TIMEOUT =40;
• mod-source-record-manager DB_RECONNECTATTEMPTS = 3, DB_RECONNECTINTERVAL=1000

...

Different configurations testing

Summary

• General response time for all workflows is about 24% improved for 2xlarge database size compared to xlarge (for stress test) and improved about 23% for a normal load test. Tested with default services configuration and without Data Import.

• Changes in configurations for services do not improve response time for workflows but for mod-circulation, it should be revised due to configuration issues found in the task definition: excessive parameters
                       "-XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m"

  Code Block
  { "name": "JAVA_OPTS", "value": "-Dport=8082 -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/ms/mod-circulation.hprof -XX:OnOutOfMemoryError=/usr/ms/heapdump.sh -XX:MetaspaceSize=88m -XX:MaxMetaspaceSize=128m -Xmx768m -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx1814m" },

  
  

• If release versions for mod-source-record-storage will be 5.6.5 and for mod-source-record-manager - 3.6.2 then changes of parameters DB_MAXPOOLSIZE for both modules, and DB_CONNECTION_TIMEOUT for mod-source-record-storage needed to avoid

  Jira Legacy
  server System Jira serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key PERF-582

  :

...