[Quesnelia][Eureka] Fiscal year rollover testing

Test status: COMPLETED WITH ERRORS

Overview

  • Regression testing of Fiscal Year Rollover (FYR) on Eureka based environment in quesnelia release. 
    • The testing triggered from UI for the ledger with generated amount of defined data set (Data quantity**: 50000 orders with Open status and 50000 orders with Pending status).
  • Testing include two different phases of the fiscal year transition process:
    • #1 phase: Test Fiscal Year Rollover - is a risk free simulation pre-step to #2 phase which includes only Finance part of the data (to preview how the rollover settings will affect funds, budgets, and encumbrances).
    • #2 phase: Fiscal Year Rollover - perform a permanent changes. It is the actual process which move financial data, budgets, and encumbrances into the new fiscal year. It includes not only Finance but Orders parts of the data too.
  • The purpose of testing is to define duration time of those 2 phases, to do comparison between current and previous results of each phase, to find any trends for resource utilization. To recommend improvements how to make it work better if any confirmed.
    • Previous successful testing for #2 phase took 29 hours
  • Jiras/ links:

Summary

  • Running FYR on eureka completed with errors. Tests failed multiple times on overall order rollover with 504 Gateway Time-out issue.
  • The problem confirmed on okapi based environment. In scope of investigation of the problem the Time-out was solved changing parameter max_locks_per_transaction in DB configuration from 64 to 1024.
  • Eureka based environment (after changes were applied): 
    • #1 Phase Test FYR took 3 hours 35 minutes, completed successfully.
    • #2 Phase actual FYR took 8 hours and completed with errors* (Order rollover Error).
  • Okapi based environment (after changes were applied, testing is carried out in scope of investigation to check functionality of FYR):
    • #1 Phase Test FYR took 3 hours 20 minutes. Completed successfully.
    • #2 Phase actual FYR took 9 hours 7 minutes. Completed successfully.
  • Service CPU utilization for both phases was under 5% on eureka environment except a spike at the beginning of test rollover (mod-finance-storage - 16%, mod-invoice - 10%). Module mod-inventory - 11% during whole test. 
  • DB CPU usage was on level of 9% during #1 Phase and there was growing trend from 9% to 15% during the rollover itself on Eureka environment with 4xlarge database. On xlarge database - 39% without growth.
  • No memory leaks were found.

Recommendations & Jiras

  • Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid Time-out issue during actual rollover (#2 Phase).
  • Investigation of issues on Eureka based environment MODFISTO-495 - Getting issue details... STATUS - Time out issue.

Errors

  • 127.0.0.1:43264 POST /finance/ledger-rollovers null HTTP_1_1 500 265 400027 tid=[tenant] Internal Server Error. This error happens when we try FYR with more than 10k open orders. The last attempt to run FYR with 50k open orders after changed recommended parameter max_locks_per_transaction failed. A new observation here is that budgets became unavailable with open orders >= 42000 after max_locks_per_transaction parameter was changed from 64 to 1024. Probably it may cause fail of FYR. 
  • Error message: io.vertx.core.impl.NoStackTraceThrowable: Response status code 401 is not between 200 and 299,qelc2/mod-finance-storage/9044d47f773b48a99cc6c8420864b724 which indicate possible RTR issue.
  • WARN [org.fol.sid.ser.rou.RequestMatchingService] (vert.x-eventloop-thread-4) Egress routing entry was not found for the request's path. Forwarding request to the Gateway:moduleId = sc-mod-finance-storage, path = /orders/rollover, destination = https://kong-qelc2.int.aws.folio.org, x-okapi-module-id = null",qelc2/sidecar-mod-finance-storage/9044d47f773b48a99cc6c8420864b724,054267740449:qelc2-folio-eis
  • ERROR edgerRolloverService handleOrderRolloverError:: Orders rollover failed for ledger f38a29f5-feaa-4363-98a1-1eba79557c9c

Test Runs and Results

Data size: 100K orders (50K orders in open status, 50K - pending), 200K order lines, 400K transactions. 

Test #

Scenario

Load level

Fiscal year rollover duration Orchid*

Fiscal year rollover duration

Poppy*

FYR duration

Quesnelia (Eureka)

FYR duration

Quesnelia (non-Eureka)

Comments
1Test Fiscal year rollover1 rollover at a time

-

3.5 hours

3.5 hours3 hours 20 minutesTest rollover finished successfully.
2Fiscal year rollover1 rollover at a time

6 days (failed with 504 error)

29 hours

8 hours, failed due to 500(server error)

9 hours 7 minutes

On eureka Rollover competed with error.

On non-eureka Rollover completed successfully.

*Results are taken from the previous test report. Details can be found at the link: [Orchid] Fiscal year rollover testing

*Results are taken from the previous test report. Details can be found at the link: [Poppy] Fiscal year rollover testing

**Data structure can be found at the link: Fiscal year rollover testing

Service CPU Utilization

Note: Instance level CPU screenshots is not included in this report. Taking into account that Fiscal Year Rollover process is mostly happening on DB side - it's barely visible on service CPU, so it's invisible on instance CPU. 

Test FYR (Eureka)

Low CPU usage during FYR reflects that it can handle large fiscal year transitions efficiently. As the process runs mostly on database level. 

Maximum CPU utilization by modules:

mod-finance storage - 16% (spike at the beginning)

mod-inventory - 10%

mod-orders, mod-orders-storage - 1,5%, 3%

mod-finance - under 1%

FYR (Eureka)

Maximum CPU utilization by modules on eureka:

mod-inventory - 11%

mod-orders - 1.5%

mod-orders-storage - 3%

mod-finance - 0.15%

mod-finance storage - 1.2%

Memory Utilization

Test FYR (Eureka)

This graph shows memory usage during #1 phase of FYR. No signs of memory leaks trends.

Memory usage by modules:

mod-finance-storage - 85%

mod-finance - 41%

mod-orders - 57%

mod-orders-storage - 46%

FYR (Eureka)

This graph shows memory usage during #2 phase of FYR. No signs of memory leaks trends too. Consumption is on the same level as #1 phase.

Memory usage by modules:

mod-finance-storage - 85%

mod-finance - 42%

mod-orders - 58%

mod-orders-storage - 46%

DB CPU Utilization

Test FYR (Eureka)

DB CPU usage range: 9% - 20%. Utilization was stable during #1 phase except of spikes to 20%. It competed successfully so it didn't affect the test.

Test FYR (non-Eureka)

The #1 phase of FYR was carried out on xlarge DB instance. That's the reason why we see 34% of DB CPU utilization. The utilization was stable - 34%. This phase completed successfully.

FYR (Eureka)

Maximum DB CPU usage - 99%

FYR (non-Eureka)

Spikes are connected to fqm manager queries (as example - REFRESH MATERIALIZED VIEW CONCURRENTLY [tenant]_0001_mod_fqm_manager.drv_langu...). Do not affect duration of FYR.

The #2 phase completed successfully in non-eureka. It utilized 35% which is almost the same as in #1 phase.

DB Connections

DB connections stay on the same level during all phases of FYR. Connections in Eureka - 690 and non-Eureka 1380.. The difference can be explained by enabled mod-serials-management module on qcon environment. 

Test FYR (Eureka)

Test FYR (non-Eureka)

FYR (Eureka)

FYR (non-Eureka)

On qcon module mod-serials spiking affect DB connections growth from 700 to 1400.

DB Load

Test FY rollover (non-Eureka)

On this graphs sql statement responsible for budget encumbrances rollover consume most of CPU.

FY rollover


Top SQL

Test FY rollover (non-Eureka)

On this graphs sql statement responsible for budget encumbrances rollover consume most of CPU.

FY rollover

Appendix

Infrastructure

PTF -environment qelc2

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instance, writer.

    NameMemory GIBvCPUsmax_connections

    db.r6g.4xlarge

    128 GiB16 vCPUs4000
    db.r6g.xlarge32 GB4vCPUs2000

    database cluster and instance parameters max_locks_per_transaction - 64, changed to 1024

  • MSK fse-tenant
    • 4 kafka.m7g.xlarge brokers in 2 zones (2 brokers per zone)
    • Apache Kafka version 3.7.x, metadata mode - KRaft

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=2

PTF -environment qcon

  • 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
  • 1 database  instance, writer

    NameMemory GIBvCPUsmax_connections

    db.r6g.xlarge

    32 GiB4 vCPUs2000

    database cluster and instance parameters max_locks_per_transaction - 64, changed to 1024

 Module QELC2

Cluster Resources - qelc2-pvt (Wed Nov 06 11:34:51 UTC 2024)

ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage22mod-remote-storage:3.2.004920447210243960512512
mod-remote-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-finance-storage24mod-finance-storage:8.6.101024896102470088128
mod-finance-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-ncip23mod-ncip:1.14.501024896076888128
mod-ncip - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-agreements22mod-agreements:7.0.60159214880000
mod-agreements - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-ebsconet21mod-ebsconet:2.2.0012481024128700128256
mod-ebsconet - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-sip26edge-sip2:3.2.501024896076888128
mod-organizations23mod-organizations:1.9.201024896076888128
mod-organizations - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-settings21mod-settings:1.0.30102489620076888128
mod-settings - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-serials-management22mod-serials-management:1.0.302480231201792384512
mod-serials-management - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-search21mod-search:3.2.6025922480204814405121024
mod-search - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-import21mod-data-import:3.1.10204818442561292384512
mod-data-import - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-dematic9edge-dematic:2.2.301024896076888128
mod-consortia-keycloak20mod-consortia-keycloak:1.4.205136477610244416384512
mod-consortia-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-tags21mod-tags:2.2.001024896076888128
mod-tags - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-courses9edge-courses:1.4.201024896076888128
mod-inventory-update22mod-inventory-update:3.3.101024896076888128
mod-inventory-update - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-notify21mod-notify:3.2.001024896076888128
mod-notify - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-configuration23mod-configuration:5.10.001024896076888128
mod-configuration - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-orders-storage36mod-orders-storage:13.7.40102489651270088128
mod-orders-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
folio-keycloak16folio-keycloak:25.0.1.1500168415360204800
edge-caiasoft9edge-caiasoft:2.2.401024896076888128
mod-password-validator21mod-password-validator:3.2.10144012980768384512
mod-password-validator - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-gobi23mod-gobi:2.8.101024896076888128
mod-gobi - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-licenses21mod-licenses:6.0.30248023125121792384512
mod-licenses - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-bulk-operations17mod-bulk-operations:2.0.203072260001536384512
mod-bulk-operations - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-fqm-manager24mod-fqm-manager:2.0.501024896076888128
mod-fqm-manager - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-graphql21mod-graphql:1.12.101024896076888128
mod-graphql - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-finance26mod-finance:4.9.001024896076888128
mod-finance - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-roles-keycloak29mod-roles-keycloak:1.4.401024896051288256
mod-roles-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-batch-print18mod-batch-print:1.1.001024896076888128
mod-batch-print - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-copycat21mod-copycat:1.6.001024896076888128
mod-copycat - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-lists23mod-lists:2.0.601024896076888128
mod-lists - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-permissions38mod-permissions:6.5.00168415445121024384512
mod-permissions - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-entities-links21mod-entities-links:3.0.2025922480400144001024
mod-entities-links - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
pub-edge2pub-edge:2023.06.1401024896076800
mod-orders26mod-orders:12.8.802048174010241024384512
mod-orders - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
kong14kong:3.7.1.83061446144204876888128
edge-patron9edge-patron:5.1.20102489625676888128
mod-users-keycloak32mod-users-keycloak:1.5.301024896051288256
mod-users-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-ncip9edge-ncip:1.10.101024896076888128
mgr-applications10mgr-applications:1.3.30102489651251288256
mod-users-bl18mod-users-bl:7.8.001440115251292288128
mod-users-bl - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mgr-tenants10mgr-tenants:1.3.30102489651251288256
mod-invoice29mod-invoice:5.8.201440115251292288128
mod-invoice - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-inventory-storage24mod-inventory-storage:27.1.404096369020483076384512
mod-inventory-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-user-import21mod-user-import:3.8.001024896076888128
mod-user-import - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-sender22mod-sender:1.12.001024896076888128
mod-sender - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-oai-pmh9edge-oai-pmh:2.9.201512136010241440384512
mgr-tenant-entitlements10mgr-tenant-entitlements:1.2.40102489651251288256
mod-login-keycloak31mod-login-keycloak:1.4.301024896051288256
mod-login-keycloak - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-export-worker21mod-data-export-worker:3.2.403072280010242048384512
mod-data-export-worker - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-rtac21mod-rtac:3.6.001024896076888128
mod-rtac - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-circulation-storage31mod-circulation-storage:17.2.302880259215361814384512
mod-circulation-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-scheduler22mod-scheduler:1.2.301024896051288256
mod-scheduler - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-calendar21mod-calendar:3.1.001024896076888128
mod-calendar - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-source-record-storage26mod-source-record-storage:5.8.605600500020483500384512
mod-source-record-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-event-config21mod-event-config:2.7.101024896076888128
mod-event-config - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-courses20mod-courses:1.4.1001024896076888128
mod-courses - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-inventory24mod-inventory:20.2.702880259210241814384512
mod-inventory - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-email21mod-email:1.17.00280025505121800384512
mod-email - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-di-converter-storage23mod-di-converter-storage:2.2.201024896076888128
mod-di-converter-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-circulation28mod-circulation:24.2.502880259215361814384512
mod-circulation - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-pubsub23mod-pubsub:2.14.00153614401024922384512
mod-pubsub - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-orders11edge-orders:3.0.301024896076888128
edge-rtac9edge-rtac:2.7.301024896076888128
mod-users23mod-users:19.3.101024896076888128
mod-users - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-template-engine21mod-template-engine:1.20.001024896076888128
mod-template-engine - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-patron-blocks18mod-patron-blocks:1.10.001024896102476888128
mod-patron-blocks - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-audit22mod-audit:2.9.001024896076888128
mod-audit - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-fqm9edge-fqm:2.0.201024896076888128
mod-source-record-manager22mod-source-record-manager:3.8.605600500020483500384512
mod-source-record-manager - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
nginx-edge2nginx-edge:2023.06.14010248960000
mod-quick-marc21mod-quick-marc:5.1.102288217601664384512
mod-quick-marc - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-feesfines21mod-feesfines:19.1.001024896076888128
mod-feesfines - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-invoice-storage25mod-invoice-storage:5.8.101872153610241024384512
mod-invoice-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-service-interaction22mod-service-interaction:4.0.20204818442561290384512
mod-service-interaction - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-export20mod-data-export:5.0.402592248010241440881024
mod-data-export - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-patron25mod-patron:6.1.001024896076888128
mod-patron - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-oai-pmh22mod-oai-pmh:3.13.204096369020483076384512
mod-oai-pmh - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-connexion6edge-connexion:1.3.101024896076888128
mod-kb-ebsco-java21mod-kb-ebsco-java:4.0.001024896076888128
mod-kb-ebsco-java - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-notes22mod-notes:5.2.0010248960952384512
mod-notes - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-data-export-spring20mod-data-export-spring:3.3.00204818442561536384512
mod-data-export-spring - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
mod-organizations-storage23mod-organizations-storage:4.7.001024896076888128
mod-organizations-storage - Sidecar 1N/Afolio-module-sidecar:1.0.0.312N/A665512012800
edge-erm6edge-erm:1.2.101024896076888128
 Module QCON

Cluster Resources - qcon-pvt (Wed Nov 06 11:28:29 UTC 2024)

ModuleTask Definition RevisionModule VersionTask CountMem Hard LimitMem Soft LimitCPU UnitsXmxMetaspace SizeMax Metaspace Size
mod-remote-storage1mod-remote-storage:3.2.024920447210243960512512
mod-finance-storage1mod-finance-storage:8.6.021024896102470088128
mod-ncip1mod-ncip:1.14.42102489612876888128
mod-agreements1mod-agreements:7.0.0215921488128000
mod-ebsconet1mod-ebsconet:2.2.0212481024128700128256
mod-consortia1mod-consortia:1.1.023072204812820485121024
mod-organizations1mod-organizations:1.9.02102489612870088128
mod-serials-management1mod-serials-management:1.0.02248023121281792384512
mod-settings2mod-settings:1.0.32102489620076888128
mod-search9mod-search:3.3.0-SNAPSHOT.224225922480204814405121024
edge-dematic1edge-dematic:2.2.01102489612876888128
mod-data-import1mod-data-import:3.1.01204818442561292384512
mod-tags1mod-tags:2.2.02102489612876888128
mod-authtoken6mod-authtoken:2.15.121440115251292288128
edge-courses1edge-courses:1.4.02102489612876888128
mod-inventory-update1mod-inventory-update:3.3.02102489612876888128
mod-notify1mod-notify:3.2.02102489612876888128
mod-configuration1mod-configuration:5.10.02102489612876888128
mod-orders-storage3mod-orders-storage:13.7.42102489651270088128
edge-caiasoft1edge-caiasoft:2.2.02102489612876888128
mod-login-saml1mod-login-saml:2.8.02102489612876888128
mod-erm-usage-harvester1mod-erm-usage-harvester:4.5.02102489612876888128
mod-password-validator1mod-password-validator:3.2.0214401298128768384512
mod-licenses1mod-licenses:6.0.02248023121281792384512
mod-gobi1mod-gobi:2.8.02102489612870088128
mod-bulk-operations1mod-bulk-operations:2.0.023072260010241536384512
mod-fqm-manager1mod-fqm-manager:2.0.12300026001282048384512
edge-dcb1edge-dcb:1.1.02102489612876888128
mod-graphql1mod-graphql:1.12.12102489612876888128
mod-finance1mod-finance:4.9.02102489612870088128
mod-erm-usage1mod-erm-usage:4.7.02102489612876888128
mod-batch-print1mod-batch-print:1.1.02102489612876888128
mod-copycat1mod-copycat:1.6.02102451212876888128
mod-lists1mod-lists:2.0.02300026001282048384512
mod-entities-links1mod-entities-links:3.0.0225922480400144001024
mod-permissions4mod-permissions:6.6.0-SNAPSHOT.3792168415445121024384512
pub-edge1pub-edge:2023.06.142102489612876800
mod-orders3mod-orders:12.8.822048174010241024384512
edge-patron1edge-patron:5.1.02102489625676888128
edge-ncip1edge-ncip:1.9.22102489612876888128
mod-users-bl1mod-users-bl:7.7.021440115251292288128
mod-invoice5mod-invoice:5.8.22288023042204888128
mod-inventory-storage2mod-inventory-storage:27.2.0-SNAPSHOT.73824096369020483076384512
mod-user-import1mod-user-import:3.8.02102489612876888128
mod-sender1mod-sender:1.12.02102489612876888128
edge-oai-pmh1edge-oai-pmh:2.9.021512136010241440384512
mod-data-export-worker1mod-data-export-worker:3.2.123072204810242048384512
mod-rtac1mod-rtac:3.6.02102489612876888128
mod-circulation-storage1mod-circulation-storage:17.2.022880259215361814384512
mod-source-record-storage1mod-source-record-storage:5.8.025600500020483500384512
mod-calendar1mod-calendar:3.1.02102489612876888128
mod-event-config1mod-event-config:2.7.02102489612876888128
mod-courses1mod-courses:1.4.102102489612876888128
mod-circulation-item1mod-circulation-item:1.0.021024896128000
mod-inventory3mod-inventory:20.2.022880259210241814384512
mod-email1mod-email:1.17.02102489612876888128
mod-circulation2mod-circulation:24.2.122880259215361814384512
mod-pubsub1mod-pubsub:2.13.02153614401024922384512
mod-di-converter-storage2mod-di-converter-storage:2.2.22102489612876888128
edge-orders1edge-orders:3.0.02102489612876888128
edge-rtac1edge-rtac:2.7.12102489612876888128
mod-users3mod-users:19.3.1220481792128102488256
mod-template-engine1mod-template-engine:1.20.02102489612876888128
mod-patron-blocks1mod-patron-blocks:1.10.021024896102476888128
mod-audit1mod-audit:2.9.02102489612876888128
edge-fqm1edge-fqm:2.0.02102489612876888128
mod-source-record-manager4mod-source-record-manager:3.8.025600500020483500384512
nginx-edge1nginx-edge:2023.06.1421024896128000
mod-quick-marc2mod-quick-marc:5.1.01228821761281664384512
nginx-okapi1nginx-okapi:2023.06.1421024896128000
okapi-b1okapi:5.3.03168414401024922384512
mod-feesfines1mod-feesfines:19.1.02102489612876888128
mod-invoice-storage3mod-invoice-storage:5.8.121872153621024384512
mod-reading-room7mod-reading-room:1.0.0-SNAPSHOT.112102489612876888128
mod-service-interaction1mod-service-interaction:4.0.12204818442561290384512
mod-dcb1mod-dcb:1.1.02102489612876888128
mod-patron1mod-patron:6.1.02102489612876888128
mod-data-export1mod-data-export:5.0.01204815241024000
mod-oai-pmh1mod-oai-pmh:3.13.024096369020483076384512
edge-connexion1edge-connexion:1.2.02102489612876888128
mod-notes1mod-notes:5.2.021024896128952384512
mod-kb-ebsco-java1mod-kb-ebsco-java:4.0.02102489612876888128
mod-login2mod-login:7.11.02204817921024102488256
mod-organizations-storage1mod-organizations-storage:4.7.02102489612870088128
mod-data-export-spring1mod-data-export-spring:3.2.01204818442561536384512
pub-okapi1pub-okapi:2023.06.142102489612876800
mod-eusage-reports1mod-eusage-reports:2.1.12102489612876888128

Methodology/Approach

In order to run FYR:

truncate tables with query

Truncate
truncate table [tenant]_mod_organizations_storage.organizations cascade;
truncate table [tenant]_mod_orders_storage.purchase_order cascade;
truncate table [tenant]_mod_invoice_storage.invoices cascade;

Create organization (Vendor) to reuse its id as access provider in the next query which generates needed data set.

Replace FYR id, access provider, fund id, fund name in accordance with FYR ledger info. Also check all instance related fields which should be replaced with actual data.  The script prepared to generate needed amount of organizations, orders, po lines, invoices.

 Queries to generate data
Generate
CREATE OR REPLACE FUNCTION public.generate_0062_data_for_edifact_export(
scenario_prefix TEXT,
organizations_amount INTEGER,
orders_per_vendor INTEGER,
polines_per_order INTEGER,
pieces_per_order INTEGER
) RETURNS VOID AS
$$
DECLARE
orgName TEXT DEFAULT 'perf_test_vendor';
orgCodePrefix TEXT DEFAULT 'PERF_TEST_ORG';
vendor_id TEXT;
BEGIN
FOR org_counter IN 1..organizations_amount
LOOP
INSERT INTO [tenant]_mod_organizations_storage.organizations (id, jsonb)
VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'code', concat(orgCodePrefix, '_', scenario_prefix, '_', org_counter),
'erpCode', '12345',
'isVendor', true,
'name', concat(orgCodePrefix, '_', scenario_prefix, '_', org_counter),
'status', 'Active',
'metadata', jsonb_build_object(
'createdDate', '2023-02-08T00:00:00.000+0000',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedDate', '2023-02-08T00:00:00.000+0000',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
)
)
)
RETURNING id INTO vendor_id;

PERFORM public.generate_0062_orders(orders_per_vendor, polines_per_order, pieces_per_order, vendor_id,1,2);
END LOOP;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION public.generate_0062_orders(
orders_per_vendor INTEGER,
polines_per_order INTEGER,
pieces_per_order INTEGER,
vendor_id TEXT,
invoices_per_organization INTEGER,
invoicelines_per_invoice INTEGER
) RETURNS VOID AS
$$
DECLARE
order_id TEXT;
poline_id TEXT;
newPoNumber INTEGER;
BEGIN
FOR order_counter IN 1..orders_per_vendor
LOOP
SELECT nextval('[tenant]_mod_orders_storage.po_number') INTO newPoNumber;

INSERT INTO [tenant]_mod_orders_storage.purchase_order (id, jsonb)
VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'id', public.uuid_generate_v4(),
'notes', jsonb_build_array(),
'acqUnitIds', jsonb_build_array(),
'approved', true,
'reEncumber', true,
'nextPolNumber', 2,
'workflowStatus', 'Pending',
'poNumber', newPoNumber,
'vendor', vendor_id,
'orderType', 'One-Time',
'metadata', jsonb_build_object(
'createdDate', '2024-10-06T00:00:00.000+0000',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedDate', '2024-10-06T00:00:00.000+0000',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
)
)
)
RETURNING id INTO order_id;

PERFORM public.generate_0062_polines(order_id, polines_per_order, pieces_per_order, newPoNumber, vendor_id);
PERFORM public.generate_0062_invoices(invoices_per_organization, invoicelines_per_invoice, vendor_id, poline_id);
END LOOP;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION public.generate_0062_polines(
order_id TEXT,
polines_per_order INTEGER,
pieces_per_order INTEGER,
ponumber INTEGER,
vendor_id TEXT
) RETURNS VOID AS
$$
DECLARE
polineNumber TEXT;
holding_id UUID;
currPoNumber INTEGER;
iteration_counter INT := 1;

BEGIN
SELECT id INTO holding_id
FROM [tenant]_mod_inventory_storage.holdings_record 
WHERE permanentlocationid = '9ffb23a9-0e83-4d8e-a70d-ddf44ab67c84' 
AND instanceid = 'b94d2138-bdfc-5dea-bab9-ac99e21b7970'
LIMIT 1;

FOR line_counter IN 1..polines_per_order
LOOP
SELECT currval('[tenant]_mod_orders_storage.po_number') INTO currPoNumber;

INSERT INTO [tenant]_mod_orders_storage.po_line (id, jsonb)
VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'id', public.uuid_generate_v4(),
'cost', jsonb_build_object(
'currency', 'USD',
'discountType', 'percentage',
'quantityElectronic', 2,
'poLineEstimatedPrice', 4,
'listUnitPriceElectronic', 2
),
'rush', false,
'alerts', jsonb_build_array(),
'claims', jsonb_build_array(),
'source', 'User',
'details', jsonb_build_object(
'productIds', jsonb_build_array(
jsonb_build_object(
'productId', '73121721',
'productIdType', 'c858e4f2-2b6b-4385-842b-60732ee14abb'
)
),
'isAcknowledged', false
),
'edition', '',
'metadata', jsonb_build_object(
'createdDate', '2024-10-06T11:09:59.181Z',
'updatedDate', '2024-10-06T11:10:48.652Z',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
),
'eresource', jsonb_build_object(
'trial', false,
'activated', false,
'accessProvider', 'c88742c3-dc2f-4589-a7cf-a90694c2a296',
'createInventory', 'Instance, Holding'
),
'isPackage', false,
'locations', jsonb_build_array(
jsonb_build_object(
'quantity', 2,
'locationId', '9ffb23a9-0e83-4d8e-a70d-ddf44ab67c84',
'quantityElectronic', 2
)
),
'publisher', 'Praeger Publishers',
'collection', false,
'instanceId', 'b94d2138-bdfc-5dea-bab9-ac99e21b7970',
'orderFormat', 'Electronic Resource',
'checkinItems', false,
'contributors', jsonb_build_array(
jsonb_build_object(
'contributor', 'Yurchenco, Henrietta',
'contributorNameTypeId', '2b94c631-fca9-4892-a730-03ee529ffe2a'
)
),
'poLineNumber', currPoNumber || '-' || iteration_counter,
'vendorDetail', jsonb_build_object(
'instructions', '',
'vendorAccount', '',
'referenceNumbers', jsonb_build_array()
),
'paymentStatus', 'Pending',
'receiptStatus', 'Pending',
'claimingActive', false,
'reportingCodes', jsonb_build_array(),
'titleOrPackage', '!Hablamos! Puerto Ricans speak / Photos. by Julia Singer.',
'automaticExport', false,
'publicationDate', '[1971]',
'purchaseOrderId', order_id,
'fundDistribution', jsonb_build_array(
jsonb_build_object(
'code', 'FYND',
'value', 100,
'fundId', '8003d427-34a3-4ad4-97cd-c1aae19f2bfc',
'distributionType', 'percentage'
)
),
'acquisitionMethod', '796596c4-62b5-4b64-a2ce-524c747afaa2',
'searchLocationIds', jsonb_build_array(
'9ffb23a9-0e83-4d8e-a70d-ddf44ab67c84'
),
'donorOrganizationIds', jsonb_build_array(),
'checkinItems', true
)
)
RETURNING id INTO polineNumber;
iteration_counter := iteration_counter + 1;

-- Call the function to generate data for titles table
PERFORM public.generate_0062_titles_data(polineNumber, concat(ponumber, '-', line_counter), pieces_per_order);
END LOOP;
END;
$$ LANGUAGE plpgsql;



CREATE OR REPLACE FUNCTION public.generate_0062_titles_data(
po_line_id TEXT,
po_line_number TEXT,
pieces_per_order INTEGER
) RETURNS VOID AS
$$
DECLARE
title_id UUID;
BEGIN
-- Insert data into titles table
INSERT INTO [tenant]_mod_orders_storage.titles (id, jsonb, creation_date, created_by, polineid)
VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'id', '4945c26f-6cf7-4050-93c1-0a8a25a2f4d7',
'title', '!Hablamos! Puerto Ricans speak / Photos. by Julia Singer.',
'edition', '',
'metadata', jsonb_build_object(
'createdDate', '2024-10-06T11:09:59.247Z',
'updatedDate', '2024-10-06T11:09:59.247Z',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
),
'poLineId', po_line_id,
'publisher', 'Praeger Publishers',
'acqUnitIds', jsonb_build_array(),
'instanceId', 'b94d2138-bdfc-5dea-bab9-ac99e21b7970',
'productIds', jsonb_build_array(
jsonb_build_object(
'productId', '73121721',
'productIdType', 'c858e4f2-2b6b-4385-842b-60732ee14abb'
)
),
'contributors', jsonb_build_array(
jsonb_build_object(
'contributor', 'Yurchenco, Henrietta',
'contributorNameTypeId', '2b94c631-fca9-4892-a730-03ee529ffe2a'
)
),
'poLineNumber', po_line_number,
'publishedDate', '[1971]',
'claimingActive', false,
'isAcknowledged', false
),
current_timestamp, -- Assuming creation date is the current timestamp
'created_by_user_id', -- Replace with the actual user ID
po_line_id::uuid
)
RETURNING id INTO title_id;

-- Call the function to generate data for pieces table
PERFORM public.generate_0062_pieces_data(title_id, po_line_id, pieces_per_order);
END
$$ LANGUAGE plpgsql;


CREATE OR REPLACE FUNCTION public.generate_0062_pieces_data(
title_id UUID,
polineNumber TEXT,
pieces_per_order INTEGER
) RETURNS VOID AS
$$
DECLARE
piece_counter INTEGER;
holding_id UUID;
BEGIN
SELECT id INTO holding_id
FROM [tenant]_mod_inventory_storage.holdings_record 
WHERE permanentlocationid = '9ffb23a9-0e83-4d8e-a70d-ddf44ab67c84' 
AND instanceid = 'b94d2138-bdfc-5dea-bab9-ac99e21b7970'
LIMIT 1;

FOR piece_counter IN 1..pieces_per_order
LOOP
-- Replace the following line with your actual poLineId
INSERT INTO [tenant]_mod_orders_storage.pieces VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'id', public.uuid_generate_v4(),
'format', 'Electronic',
'titleId', title_id,
'metadata', jsonb_build_object(
'createdDate', '2024-10-06T11:43:04.384Z',
'updatedDate', '2024-10-06T11:43:04.384Z',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
),
'poLineId', polineNumber,
'holdingId', 'aefd322b-9a32-4cd9-80e0-a5a3f882f2b1',
'copyNumber', piece_counter,
'supplement', true,
'displaySummary', piece_counter,
'receivingStatus', 'Expected',
'displayOnHolding', false,
'discoverySuppress', false,
'statusUpdatedDate', '2024-06-05T11:43:04.385+00:00'
),
NULL,
NULL,
'eb79cb72-0fb5-4773-a580-b1428fa724e9'::UUID, -- Replace with the actual poLineId
title_id
);
END LOOP;
END
$$ LANGUAGE plpgsql;






CREATE OR REPLACE FUNCTION public.generate_0062_invoices(
invoices_per_organization INTEGER,
invoicelines_per_invoice INTEGER,
vendor_id TEXT,
order_id TEXT
) RETURNS VOID AS
$$
DECLARE
invoice_id TEXT;
newInvoiceNumber INTEGER;
BEGIN
FOR invoice_counter IN 1..invoices_per_organization
LOOP
SELECT nextval('[tenant]_mod_invoice_storage.invoice_number') INTO newInvoiceNumber;

INSERT INTO [tenant]_mod_invoice_storage.invoices (id, jsonb)
VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'id', public.uuid_generate_v4(),
'total', 0.0,
'source', 'User',
'status', 'Approved',
'currency', 'USD',
'metadata', jsonb_build_object(
'createdDate', '2024-10-06T09:07:45.366Z',
'updatedDate', '2024-10-06T09:08:47.954Z',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
),
'subTotal', 0.0,
'vendorId', vendor_id,
'poNumbers', ARRAY[newInvoiceNumber]::TEXT[],
'acqUnitIds', ARRAY[]::UUID[],
'adjustments', ARRAY[]::JSONB[],
'invoiceDate', '2024-02-27T00:00:00.000+00:00',
'batchGroupId', '2a2cb998-1437-41d1-88ad-01930aaeadd5',
'fiscalYearId', '562bf586-11e9-4c36-98d7-3341042c856c',
'paymentMethod', 'Cash',
'accountingCode', '12345',
'folioInvoiceNo', newInvoiceNumber::TEXT,
'enclosureNeeded', false,
'vendorInvoiceNo', invoice_counter::TEXT,
'adjustmentsTotal', 0.0,
'exportToAccounting', false,
'nextInvoiceLineNumber', 2,
'chkSubscriptionOverlap', true
)
) RETURNING id INTO invoice_id;

PERFORM public.generate_0062_invoicelines(
invoicelines_per_invoice,
invoice_id,
newInvoiceNumber,
vendor_id,
order_id
);
END LOOP;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION public.generate_0062_invoicelines(
invoicelines_per_invoice INTEGER,
invoice_id TEXT,
newInvoiceNumber INTEGER,
vendor_id TEXT,
order_id TEXT
) RETURNS VOID AS
$$
DECLARE
invoiceline_id TEXT;
BEGIN
FOR invoiceline_counter IN 1..invoicelines_per_invoice
LOOP
INSERT INTO [tenant]_mod_invoice_storage.invoice_lines (id, jsonb)
VALUES (
public.uuid_generate_v4(),
jsonb_build_object(
'id', public.uuid_generate_v4(),
'total', 0.0,
'metadata', jsonb_build_object(
'createdDate', '2024-10-06T09:08:47.393Z',
'updatedDate', '2024-10-06T09:08:47.393Z',
'createdByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42',
'updatedByUserId', '9f9d1c46-52e1-4bb7-9c6c-56e6bb945c42'
),
'poLineId', order_id,
'quantity', 2,
'subTotal', 0.0,
'invoiceId', invoice_id,
'adjustments', ARRAY[]::JSONB[],
'description', 'Its awesome if its working',
'accountNumber', 'libraryorders@library.tam',
'accountingCode', '12345',
'adjustmentsTotal', 0.0,
'referenceNumbers', ARRAY[]::JSONB[],
'fundDistributions', jsonb_build_array(
jsonb_build_object(
'code', 'FYND',
'value', 100.0,
'fundId', '8003d427-34a3-4ad4-97cd-c1aae19f2bfc',
'distributionType', 'percentage'
)
),
'invoiceLineNumber', invoiceline_counter,
'invoiceLineStatus', 'Approved',
'releaseEncumbrance', false
)
) RETURNING id INTO invoiceline_id;

-- You can perform additional actions if needed
END LOOP;
END
$$ LANGUAGE plpgsql;


-- CREATE sample data
-- 1 - amount of organizations to be created
-- 2 - amount of orders per organization
-- 3 - amount of polines per order
-- 4 - amount of pieces per order

SELECT public.generate_0062_data_for_edifact_export('FYNG',1,1,1,1);



-- CLEANUP


To create encumbrances linked to the current budget orders should be with open status. To change the status from pending to open select needed data for the jmeter script. Save to use in the PUT request orders/composite-orders/${po_id}. Run needed number of requests to prepare open orders.

Data preparation to Open Orders
SELECT id as po_id, jsonb->>'vendor' as vendor_id, jsonb->>'poNumber' as po_number
select count(id)
FROM cs00000int_mod_orders_storage.purchase_order
where jsonb->>'workflowStatus'='Pending'
BODY DATA
{"id":"${po_id}","approved":true,"notes":[],"poNumber":"${po_number}","orderType":"One-Time","reEncumber":true,"vendor":"${vendor_id}","workflowStatus":"Open","acqUnitIds":[],"needReEncumber":false}

Legacy approach about data preparation instructions can be found at the link: Steps for testing process#Fiscalyearrollover Use this link to get exact number of organizations, orders, POLined, invoices

  • Set max_locks_per_transaction parameter in DB cluster and instance configurations from default 64 to 1024 to avoid problems with ordersRollover.
  • Additional info for testing (scripts, queries) saved in S3 bucket - Buckets/fse-ptf/FYR/
    • envs_data_gen_scripts - txt files with sql which populate DB with data
    • qelc2 and qcon jmeter scripts to change status from pending to open