EBSCONet Orders Renewal (Nolana)

Overview

This is initial report for EBSCONET orders renewal workflow testing PERF-285 - Getting issue details... STATUS . The purpose of this document is to highlight KPIs of EBSCONET workflow, find possible bottlenecks/ issues, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.

Summary

During testing we've find several issues: 

  • NullPointerException
  • Internal server error

No memory leaks were found. 

High level results:

Test #VUsersDurationresponse time (get) ms.

response time (put) ms

Error rate
avg75%95%avg75%95%
127 min 10 s249 2572871781841972.6% (52 calls)
227 min2472542881781841973.3% (67 calls)
3104 min 30 s2983194042362533391.14% (57 calls)
4104 min 10 s3083324172422634651.56% (78 calls)


Recommendations & Jiras

  • MODEBSNET-56 - Getting issue details... STATUS  Ticket to handle NullPointerException Which is reproducible even with 2 users.
  • MODEBSNET-57 - Getting issue details... STATUS Ticket to handle internal server error .
  • We have noticed that mod-orders-storage has allocated only 128 CPU units. It's better to increase CPU units value of mod-orders-storage based on CPU utilization (at least 3x times)

Test Runs 

Test #

VUsers

Data setLoad generator size Load generator Memory(GiB) 
  1. normal conditions
2 users
  • Number of order lines being renewed 1000
  • Number of orders being renewed 804
  • 4 of these orders have 50 POLs each - 800 orders with 1 POL each


t3.medium3
2. normal conditions2 users
  • Number of order lines being renewed 1000
  • Number of orders being renewed 804
  • 4 of these orders have 50 POLs each - 800 orders with 1 POL each


t3.medium3

3. extreme conditions

10 users
  • Number of order lines being renewed 5000
  • Number of orders being renewed 4804
  • 4 of these orders have 50 POLs each - 4800 orders with 1 POL each
t3.medium3
4 extreme conditions10 users
  • Number of order lines being renewed 5000
  • Number of orders being renewed 4804
  • 4 of these orders have 50 POLs each - 4800 orders with 1 POL each
t3.medium3

Results


Test #VUsersDurationresponse time (get) ms.

response time (put) ms

Error rate
avg75%95%avg75%95%
127 min 10 s249 2572871781841972.6% (52 calls)
227 min2472542881781841973.3% (67 calls)
3104 min 30 s2983194042362533391.14% (57 calls)
4104 min 10 s3083324172422634651.56% (78 calls)


Memory Utilization



Nolana Avg%

mod-organizations

25%

mod-organizations-storage

24%

mod-finance

29%

mod-finance-storage

28%

nginx-edge

2%

mod-ebsconet

37%

nginx-okapi

3%

mod-orders

44%

mod-orders-storage

34%

okapi

36%

mod-notes

43%

mod-configuration 

25%

edge-orders

20%

CPU Utilization 

Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage



RDS CPU Utilization 

Appendix

Infrastructure

PTF -environment ncp3-pvt

  • m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1 
  • 2 instances of db.r6.xlarge database instances, one reader, and one writer
  • MSK ptf-kakfa-3 
    • 4 m5.2xlarge brokers in 2 zones
    • Apache Kafka version 2.8.0

    • EBS storage volume per broker 300 GiB

    • auto.create.topics.enable=true
    • log.retention.minutes=480
    • default.replication.factor=3


Modules memory and CPU parameter

Modules

Version

Task Definition

Running Tasks 

CPU

Memory

MemoryReservation

MaxMetaspaceSize

Xmx

mod-organizations

1.6.0221281024896128700

mod-organizations-storage

4.4.022128

1024

896

128700

mod-finance

4.6.2221281024896128700

mod-finance-storage

8.3.1221281024896128700

nginx-edge

nginx-edge:2022.03.02121281024896N/AN/A

mod-ebsconet

1.4.0221281024896256700

nginx-okapi

nginx-okapi:2022.03.02121281024896N/AN/A

mod-orders

12.5.422102420481440256896
mod-orders-storage13.4.0221281024896128700
okapi4.14.713102416841440922922

mod-mod-notes

4.0.0221281024896128322
mod-configuration5.9.0321281024896128768

edge-orders

2.7.0221281024896128700

Methodology/Approach

According to EBSCONET order renewal integration testing we have designed test containing two calls [get] /orders/order-lines/${polineNumber}?type=EBSCONET&apiKey=${API_KEY}

and [put] /orders/order-lines/${polineNumber}?type=EBSCONET&apiKey=${API_KEY} with payload :

{
    "currency": "USD",
    "fundCode": "NEW2023",
    "poLineNumber": "${polineNumber}",
    "quantity": 1,
    "unitPrice": 1.0,
    "vendor": "ZHONEWAX$%",
    "vendorAccountNumber": "libraryorders@library.tam",
    "vendorReferenceNumbers": [],
    "workflowStatus": "Open"
}

Test data creation

In order to create test data (orders with PO lines) we used SQL script. This script will create orders for particular organization ()

CREATE OR REPLACE FUNCTION public.generate_data_for_edifact_export(organizations_amount integer,
                                                                   orders_per_vendor integer,
                                                                   polines_per_order integer) RETURNS VOID as
$$

DECLARE
  -- !!! SET DEFAULT TENANT NAME !!!
  orgName   text DEFAULT 'perf_test_vendor';
  orgCode   TEXT default 'PERF_TEST_ORG';
  vendor_id TEXT;
BEGIN
  for org_counter in 1..organizations_amount
    loop
      /*            INSERT INTO fs09000000_mod_organizations_storage.organizations (id, jsonb)
                  VALUES (public.uuid_generate_v4(),
                          jsonb_build_object('code', concat(orgCode, org_counter),
                                             'erpCode', '12345',
                                             'isVendor', true,
                                             'name', concat(orgName, org_counter),
                                             'status', 'Active',
                                             'metadata', jsonb_build_object(
                                                     'createdDate', '2023-02-08T00:00:00.000+0000',
                                                     'createdByUserId', '28d1057c-d137-11e8-a8d5-f2801f1b9fd1',
                                                     'updatedDate', '2023-02-08T00:00:00.000+0000',
                                                     'updatedByUserId', '28d1057c-d137-11e8-a8d5-f2801f1b9fd1'
                                                 )
                              ))
                  RETURNING id INTO vendor_id;*/

      PERFORM public.generate_orders(orders_per_vendor, polines_per_order, '2e6d8468-0620-475b-a092-045e659a0aaa');-------------------------------
    end loop;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION public.generate_orders(orders_per_vendor integer, polines_per_order integer, vendor_id text) RETURNS VOID as
$$
DECLARE
  order_id    text;
  newPoNumber integer;
BEGIN
  for order_counter in 1..orders_per_vendor
    loop
      SELECT nextval('fs09000000_mod_orders_storage.po_number') INTO newPoNumber;
      --
      INSERT INTO fs09000000_mod_orders_storage.purchase_order (id, jsonb)
      VALUES (public.uuid_generate_v4(),
              jsonb_build_object('id', public.uuid_generate_v4(),
                                 'reEncumber', true,
                                 'workflowStatus', 'Pending',
                                 'poNumber', newPoNumber,
                                 'vendor', vendor_id,
                                 'orderType', 'One-Time',
                                 'metadata', jsonb_build_object(
                                   'createdDate', '2023-02-08T00:00:00.000+0000',
                                   'createdByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684',
                                   'updatedDate', '2023-02-08T00:00:00.000+0000',
                                   'updatedByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684'
                                   )
                ))


      RETURNING id INTO order_id;
      PERFORM public.generate_polines(order_id, polines_per_order, newPoNumber);
    end loop;
END
$$ LANGUAGE plpgsql;


CREATE OR REPLACE FUNCTION public.generate_polines(order_id text, polines_per_order integer, ponumber integer) RETURNS VOID as
$$
DECLARE
  polineNumber text;
BEGIN
  for line_counter in 1..polines_per_order
    loop
      INSERT INTO fs09000000_mod_orders_storage.po_line (id, jsonb)
      VALUES (public.uuid_generate_v4(),
               -- add other fields to increase processing complexity
              jsonb_build_object('id', public.uuid_generate_v4(),
                                 'acquisitionMethod', 'df26d81b-9d63-4ff8-bf41-49bf75cfa70e',
                                 'rush', false,
                                 'cost', json_build_object(
                                   'currency', 'USD',
                                   'discountType', 'percentage',
                                   'listUnitPrice', 1,
                                   'quantityPhysical', 1,
                                   'poLineEstimatedPrice', 1
                                   ),
                                  'fundDistribution', json_build_array(
                                   jsonb_build_object(
                                         'code', 'NEW2023',
                                         'fundId', '9dde6d9d-a567-43f6-9024-eb00ac1fc076',
                                        'distributionType', 'percentage',
                                        'value', 100
                                      )
                                   ),
								   'locations', json_build_array(
                                      jsonb_build_object(  
									   'locationId', 'f4619e23-d081-4447-a589-e278037e7f5e',
									   'quantity', 2,
									   'quantityElectronic', 0,
									   'quantityPhysical', 2
                                       )
                                    ),
                                 'alerts', json_build_array(),
                                 'source', 'User',
                                 'physical', jsonb_build_object('createInventory', 'None'),
                                 'details', jsonb_build_object(),
                                 'isPackage', false,
                                 'orderFormat', 'Physical Resource',
                                 'vendorDetail', jsonb_build_object('vendorAccount', 'libraryorders@library.tam'),
                                 'titleOrPackage', 'ABA Journal',
                                 'automaticExport', true,
                                 'publicationDate', '1915-1983',
                                 'purchaseOrderId', order_id,
                                 'poLineNumber', concat(ponumber, '-', line_counter),
                                 'claims', json_build_array(),
                                 'metadata', jsonb_build_object(
                                   'createdDate', '2023-02-08T00:00:00.000+0000',
                                   'createdByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684',
                                   'updatedDate', '2023-02-08T00:00:00.000+0000',
                                   'updatedByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684'
                                   )
                ));
    end loop;
END
$$ LANGUAGE plpgsql;


-- CREATE sample data
-- 1 - amount of organizations to be created
-- 2 - amount of orders per organization
-- 3 - amount of polines per order
select public.generate_data_for_edifact_export(1, 1, 1);

-- CLEANUP