SIP2-288 - Book-sorter performance issues

SIP2-288 - Book-sorter performance issues

Submitted

Jul 14, 2024

Approved

 

Status

in progress

Impact

MEDIUM

Arch Ticket

https://folio-org.atlassian.net/browse/SIP2-288

Prod ticket

https://folio-org.atlassian.net/browse/PERF-1168

Problem Statement

LOC’s book sorter machine has an issue with data lag/sync between FOLIO and the machine.

Context

  • Book sorter machine integrates with Folio via SIP2

  • The process involves 2 calls to Folio

    • 1. The item barcode is scanned, and a call is made to identify the item.

    • 2. Using the returned response, the item’s location is established, and the second call is made to check in the item to the appropriate tenant.

  • The book sorter machine will then deliver the item to the appropriate bin for delivery to be shelved.

  • Check-in rate corresponds to a worker manually placing each book on the book sorter machine (every few seconds at peak)

Issue

  • The book sorter will time out during the operation, fail to sort the book, and deliver the book to the error bin instead.

    • The book sorter machine has a hard-coded timeout of 1 second for the entire process

    • FOLIO will respond to each call within approximately 2 seconds, with an overall response time of 4 seconds or more.

SIP2 Request Flows

  • Each executed command performs a login request with the provided pair of credentials.

  • Credentials are provided during the login command (username | password | servicePointId).

  • Each new pair of credentials could affect the authorization cache in folio-module-sidecar (uses sessionId for authorization cache-key)

1. Login Request Diagram

2. Item Information Request Diagram

3. Item Check-In Request Diagram

Related Information

Solution

1. Minimize the number of login requests per session

The total number of requests per the item sorting series of requests is 10:

  • 2 for the Login command

  • 5 for the Item Information command

  • 3 for the Item Checkin command

Although each login operation is relatively quick, performing it 10 times within a single session significantly impacts performance. To optimize this, an access token should be issued once at the beginning of the session and refreshed as needed using the HTTP POST /authn/refresh Token API request. If the refresh operation fails for any reason, a new access token should be obtained and stored in the cache as a fallback.

Optimization Results

edge-sip2 container being used on Volaris Rancher environment with increased memory limits (with default ~120Mi container is unstable):

resources: limits: memory: 600Mi requests: memory: 500Mi

Before optimization (Single Login - 5 min)

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received
KB/sec

Sent
KB/sec

Avg. Bytes

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received
KB/sec

Sent
KB/sec

Avg. Bytes

Init connection

1

197

0

197

0

0.00%

5.07614

0.15

0

31

Login

1

401

0

401

0

0.00%

2.49377

0.03

0

12

Item Information Request

202

744

0

1472

64.75

0.00%

0.67395

0.06

0

98.1

Item Checkin Request

202

741

0

2684

142.24

0.00%

0.6745

0.08

0

123.8

Close connection

1

4

0

4

0

0.00%

250

0

0

0

TOTAL

409

735

0

2684

130.44

0.00%

1.35827

0.15

0

109.8

Before optimization (Login per check-in - 5 min)

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received KB/sec

Sent KB/sec

Avg. Bytes

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received KB/sec

Sent KB/sec

Avg. Bytes

Init connection

149

190

0

234

7.21

0.00%

0.49827

0.02

0

31

Login

149

352

0

412

12.47

0.00%

0.49804

0.01

0

12

Item Information Request

149

735

0

995

47.95

0.00%

0.49741

0.05

0

98

Item Checkin Request

148

738

0

912

42.01

0.00%

0.49782

0.06

0

123.7

Close connection

148

0

0

9

1.04

0.00%

0.49923

0

0

0

TOTAL

745

402

0

995

295.76

0.00%

2.48125

0.13

0

52.9

After optimization (Single Login - 5 min)

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received KB/sec

Sent KB/sec

Avg. Bytes

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received KB/sec

Sent KB/sec

Avg. Bytes

Create SIP2 Session class

1

8

0

8

0

0.00%

125

4.03

0

33

Init connection

1

197

0

197

0

0.00%

5.07614

0.15

0

31

Login

1

313

0

313

0

0.00%

3.19489

0.04

0

12

Item Information Request

373

329

0

635

29.5

0.00%

1.24331

0.12

0

98.2

Item Checkin Request

372

475

0

666

22.7

0.00%

1.24353

0.15

0

124

Close connection

1

6

0

6

0

100.00%

166.6667

0

0

0

TOTAL

749

400

0

666

80.57

0.13%

2.49158

0.27

0

110.6

Before optimization (Login per check-in - 5 min)

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received KB/sec

Sent KB/sec

Avg. Bytes

Label

# Samples

Average

Min

Max

Std. Dev.

Error %

Throughput

Received KB/sec

Sent KB/sec

Avg. Bytes

Init connection

213

188

0

201

3.98

0.00%

0.71073

0.02