Table of Contents

Overview

This is initial report for EBSCONET orders renewal workflow testing

Jira Legacy

server	System JIRA
serverId	01505d01-b853-3c2e-90f1-ee9b165564fc
key	PERF-285

. The purpose of this document is to highlight KPI KPIs of EBSCONET workflow, find possible bottlenecks/ issues. Define baseline

Summary

[ A bulleted-list of the most important and relevant observations from the test results. What are the most important things the readers need to know about this testing effort? Some suggestions

Comparison to previous test or release of response times or API durations
Any notable changes
Particular response time or durations
Service memory and/or CPU utilization
RDS memory and/or CPU utilization
Other interesting observations

]

used modules:

mod-organizations
mod-organizations-storage
mod-finance
mod-finance-storage
nginx-edge
mod-ebsconet
nginx-okapi
mod-orders
mod-orders-storage
okapi
mod-mod-notes
mod-configuration
edge-orders

Recommendations & Jiras (Optional)

[ If there are recommendations for the developers or operations team, or anything worth calling out, list them here. Also include any Jiras created for follow-up work]

, and define the baseline performance for this workflow since it hasn't been tested before. EBSCONet is a client application that calls edge-orders to renew orders. The tests simulate EBSCONet calling edge-orders by issuing calls to retrieve a Purchase Order Line (POL) and updating it with relevant information for renewal.

Summary

During testing we've find several issues:

NullPointerException
Internal server error

No memory leaks were found.

High level results:

Test #	VUsers	Duration	avg	75%	95%	avg	75%	95%	Error rate
Test #	VUsers	Duration	response time (get) ms.			response time (put) ms			Error rate
1	2	7 min 10 s	249	257	287	178	184	197	2.6% (52 calls)
2	2	7 min	247	254	288	178	184	197	3.3% (67 calls)
3	10	4 min 30 s	298	319	404	236	253	339	1.14% (57 calls)
4	10	4 min 10 s	308	332	417	242	263	465	1.56% (78 calls)

Recommendations & Jiras

Jira Legacy
server System JIRA
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODEBSNET-56
Ticket to handle NullPointerException Which is reproducible even with 2 users.
Jira Legacy
server System JIRA
serverId 01505d01-b853-3c2e-90f1-ee9b165564fc
key MODEBSNET-57
Ticket to handle internal server error .
We have noticed that mod-orders-storage has allocated only 128 CPU units. It's better to increase CPU units value of mod-orders-storage based on CPU utilization (at least 3x times)

Test Runs

Test #	VUsers	Data set	Load generator size	Load generator Memory(GiB)
normal conditions	2 users	Number of order lines being renewed 1000 Number of orders being renewed 804 4 of these orders have 50 POLs each - 800 orders with 1 POL each	t3.medium	3
2. normal conditions	2 users	Number of order lines being renewed 1000 Number of orders being renewed 804 4 of these orders have 50 POLs each - 800 orders with 1 POL each	t3.medium	3
3. extreme conditions	10 users	Number of order lines being renewed 5000 Number of orders being renewed 4804 4 of these orders have 50 POLs each - 4800 orders with 1 POL each	t3.medium	3
4 extreme conditions	10 users	Number of order lines being renewed 5000 Number of orders being renewed 4804 4 of these orders have 50 POLs each - 4800 orders with 1 POL each	t3.medium	3

Results

Test #	VUsers	Duration	avg	75%	95%	avg	75%	95%	Error rate
Test #	VUsers	Duration	response time (get) ms.			response time (put) ms			Error rate
1	2	7 min 10 s	249	257	287	178	184	197	2.6% (52 calls)
2	2	7 min	247	254	288	178	184	197	3.3% (67 calls)
3	10	4 min 30 s	298	319	404	236	253	339	1.14% (57 calls)
4	10	4 min 10 s	308	332	417	242	263	465	1.56% (78 calls)

Memory Utilization

	Nolana Avg%
mod-organizations	25%
mod-organizations-storage	24%
mod-finance	29%
mod-finance-storage	28%
nginx-edge	2%
mod-ebsconet	37%
nginx-okapi	3%
mod-orders	44%
mod-orders-storage	34%
okapi	36%
mod-notes	43%
mod-configuration	25%
edge-orders	20%

CPU Utilization

[Description of notable observations of modules and instances CPU utilization with screenshots (of all modules and involved modules) and tables]Most used module - mod-orders-storage. During tests with 10 users it reaches 320% of CPU usage

RDS CPU Utilization

Additional information from module and database logs (Optional)

[ Although it is optional to look at logs it is always recommended to look at the logs to see if there were any errors, exceptions, or warnings. If there were any, create Jiras fo rthe module that generated the warnings/errors/exceptions]

Appendix

Infrastructure

PTF -environment ncp3-pvt

9 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 instances of db.r6.xlarge database instances, one reader, and one writer
MSK ptf-kakfa-3 [ kafka configurations]3
- 4 m5.2xlarge brokers in 2 zones
- Apache Kafka version 2.8.0
- EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3

...

Modules	Version	Task Definition	Running Tasks	CPU	Memory	MemoryReservation	MaxMetaspaceSize	Xmx
mod-organizations	1.6.0	2	2	128	1024	896	128	700
mod-organizations-storage	4.4.0	2	2	128	1024	896	128	700
mod-finance	4.6.2	2	2	128	1024	896	128	700
mod-finance-storage	8.3.1	2	2	128	1024	896	128	700
nginx-edge	nginx-edge:2022.03.02	1	2	128	1024	896	N/A	N/A
mod-ebsconet	1.4.0	2	2	128	1024	896	256	700
nginx-okapi	nginx-okapi:2022.03.02	1	2	128	1024	896	N/A	N/A
mod-orders	12.5.4	2	2	1024	2048	1440	256	896
mod-orders-storage	13.4.0	2	2	128	1024	896	128	700
okapi	4.14.7	1	3	1024	1684	1440	922	922
mod-mod-notes	4.0.0	2	2	128	1024	896	128	322
mod-configuration	5.9.0	3	2	128	1024	896	128	768
edge-orders	2.7.0	2	2	128	1024	896	128	700

Methodology/Approach

[ List the high-level methodology that was used to carry out the tests. This is important for complex tests that involve multiple workflows. It's important to inform readers of how the tests were performed so that they can comment on any flaw in the test approach or that they can try to reproduce the test results themselves. For example:

Start CICO test first
Run a Data Import job after waiting for 10 minutes
Run an eHoldings job after another 10 minutes
On another tenant run another DI job after 30 minutes in

The steps don't need to be very specific because the details are usually contained in the participating workflow's README files (on GitHub). However, anything worth calling out that was not mentioned elsewhere should be mentioned here.]

Additional Screenshots of graphs or charts (Optional)

[ Optionally include additional screenshots of graphs on the Cloudwatch and Grafana dashboards for completeness sake. These don't have to be solely included in here but can be added in any section if they complement other graphs and fit the narrative. ]According to EBSCONET order renewal integration testing we have designed test containing two calls [get] /orders/order-lines/${polineNumber}?type=EBSCONET&apiKey=${API_KEY}

and [put] /orders/order-lines/${polineNumber}?type=EBSCONET&apiKey=${API_KEY} with payload :

{
"currency": "USD",
"fundCode": "NEW2023",
"poLineNumber": "${polineNumber}",
"quantity": 1,
"unitPrice": 1.0,
"vendor": "ZHONEWAX$%",
"vendorAccountNumber": "libraryorders@library.tam",
"vendorReferenceNumbers": [],
"workflowStatus": "Open"
}

Test data creation

In order to create test data (orders with PO lines) we used SQL script. This script will create orders for particular organization ()

Code Block

language	sql

CREATE OR REPLACE FUNCTION public.generate_data_for_edifact_export(organizations_amount integer,
                                                                   orders_per_vendor integer,
                                                                   polines_per_order integer) RETURNS VOID as
$$

DECLARE
  -- !!! SET DEFAULT TENANT NAME !!!
  orgName   text DEFAULT 'perf_test_vendor';
  orgCode   TEXT default 'PERF_TEST_ORG';
  vendor_id TEXT;
BEGIN
  for org_counter in 1..organizations_amount
    loop
      /*            INSERT INTO fs09000000_mod_organizations_storage.organizations (id, jsonb)
                  VALUES (public.uuid_generate_v4(),
                          jsonb_build_object('code', concat(orgCode, org_counter),
                                             'erpCode', '12345',
                                             'isVendor', true,
                                             'name', concat(orgName, org_counter),
                                             'status', 'Active',
                                             'metadata', jsonb_build_object(
                                                     'createdDate', '2023-02-08T00:00:00.000+0000',
                                                     'createdByUserId', '28d1057c-d137-11e8-a8d5-f2801f1b9fd1',
                                                     'updatedDate', '2023-02-08T00:00:00.000+0000',
                                                     'updatedByUserId', '28d1057c-d137-11e8-a8d5-f2801f1b9fd1'
                                                 )
                              ))
                  RETURNING id INTO vendor_id;*/

      PERFORM public.generate_orders(orders_per_vendor, polines_per_order, '2e6d8468-0620-475b-a092-045e659a0aaa');-------------------------------
    end loop;
END
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION public.generate_orders(orders_per_vendor integer, polines_per_order integer, vendor_id text) RETURNS VOID as
$$
DECLARE
  order_id    text;
  newPoNumber integer;
BEGIN
  for order_counter in 1..orders_per_vendor
    loop
      SELECT nextval('fs09000000_mod_orders_storage.po_number') INTO newPoNumber;
      --
      INSERT INTO fs09000000_mod_orders_storage.purchase_order (id, jsonb)
      VALUES (public.uuid_generate_v4(),
              jsonb_build_object('id', public.uuid_generate_v4(),
                                 'reEncumber', true,
                                 'workflowStatus', 'Pending',
                                 'poNumber', newPoNumber,
                                 'vendor', vendor_id,
                                 'orderType', 'One-Time',
                                 'metadata', jsonb_build_object(
                                   'createdDate', '2023-02-08T00:00:00.000+0000',
                                   'createdByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684',
                                   'updatedDate', '2023-02-08T00:00:00.000+0000',
                                   'updatedByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684'
                                   )
                ))


      RETURNING id INTO order_id;
      PERFORM public.generate_polines(order_id, polines_per_order, newPoNumber);
    end loop;
END
$$ LANGUAGE plpgsql;


CREATE OR REPLACE FUNCTION public.generate_polines(order_id text, polines_per_order integer, ponumber integer) RETURNS VOID as
$$
DECLARE
  polineNumber text;
BEGIN
  for line_counter in 1..polines_per_order
    loop
      INSERT INTO fs09000000_mod_orders_storage.po_line (id, jsonb)
      VALUES (public.uuid_generate_v4(),
               -- add other fields to increase processing complexity
              jsonb_build_object('id', public.uuid_generate_v4(),
                                 'acquisitionMethod', 'df26d81b-9d63-4ff8-bf41-49bf75cfa70e',
                                 'rush', false,
                                 'cost', json_build_object(
                                   'currency', 'USD',
                                   'discountType', 'percentage',
                                   'listUnitPrice', 1,
                                   'quantityPhysical', 1,
                                   'poLineEstimatedPrice', 1
                                   ),
                                  'fundDistribution', json_build_array(
                                   jsonb_build_object(
                                         'code', 'NEW2023',
                                         'fundId', '9dde6d9d-a567-43f6-9024-eb00ac1fc076',
                                        'distributionType', 'percentage',
                                        'value', 100
                                      )
                                   ),
								   'locations', json_build_array(
                                      jsonb_build_object(  
									   'locationId', 'f4619e23-d081-4447-a589-e278037e7f5e',
									   'quantity', 2,
									   'quantityElectronic', 0,
									   'quantityPhysical', 2
                                       )
                                    ),
                                 'alerts', json_build_array(),
                                 'source', 'User',
                                 'physical', jsonb_build_object('createInventory', 'None'),
                                 'details', jsonb_build_object(),
                                 'isPackage', false,
                                 'orderFormat', 'Physical Resource',
                                 'vendorDetail', jsonb_build_object('vendorAccount', 'libraryorders@library.tam'),
                                 'titleOrPackage', 'ABA Journal',
                                 'automaticExport', true,
                                 'publicationDate', '1915-1983',
                                 'purchaseOrderId', order_id,
                                 'poLineNumber', concat(ponumber, '-', line_counter),
                                 'claims', json_build_array(),
                                 'metadata', jsonb_build_object(
                                   'createdDate', '2023-02-08T00:00:00.000+0000',
                                   'createdByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684',
                                   'updatedDate', '2023-02-08T00:00:00.000+0000',
                                   'updatedByUserId', '9eb67301-6f6e-468f-9b1a-6134dc39a684'
                                   )
                ));
    end loop;
END
$$ LANGUAGE plpgsql;


-- CREATE sample data
-- 1 - amount of organizations to be created
-- 2 - amount of orders per organization
-- 3 - amount of polines per order
select public.generate_data_for_edifact_export(1, 1, 1);

-- CLEANUP

Versions Compared

Old Version 4

New Version Current

Key

Overview

Summary

Recommendations & Jiras (Optional)

Summary

Recommendations & Jiras

Test Runs

Results

Memory Utilization

CPU Utilization

RDS CPU Utilization

Additional information from module and database logs (Optional)

Appendix

Infrastructure

Methodology/Approach

Additional Screenshots of graphs or charts (Optional)

Test data creation

Page Comparison

Versions Compared

Old Version 4

New Version Current

Key

Overview

Summary

Recommendations & Jiras (Optional)

Summary

Recommendations & Jiras

Test Runs

Results

Memory Utilization

CPU Utilization

RDS CPU Utilization

Additional information from module and database logs (Optional)

Appendix

Infrastructure

Methodology/Approach

Additional Screenshots of graphs or charts (Optional)

Test data creation