Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

POC: Measure performance of mapping and saving authority records to a file

Description

Purpose/Overview:
In the scope of the design described on the page: https://folio-org.atlassian.net/wiki/display/FOLIJET/Long+term+solution+for+migrating+authority+records it's required to test the performance of reading MARC authority records from a database, mapping them to Authority records and saving the result of mapping in JSON format to the file.

Requirements/Scope:

  1. Prepare a database with at least 10000 MARC authority records (in SRS)

  2. Implement logic of reading records from the database (from the prepared view), mapping them to Authority records, and saving the records to a file. Use JDBC to interact with the database.

  3. Test on perf env and measure time of the process

Environment

None

Potential Workaround

None

CSP Request Details

None

CSP Rejection Details

None

Checklist

hide

TestRail: Results

Activity

Show:

Pavlo SmahinMay 12, 2023 at 12:45 PM

The results of POC are provided in the table. The processing involves retrieving records from the database, mapping them to the Authority record, and saving them to files in chunks of 10,000 records each. The time required for this process is linear and proportional to the number of records being processed.

Threads / ConPool

Memory / CPU

Num records

Time1

Time2

Time3

Avg

5 / 10

640Mi / 512Mi

96179

1:32

1:28

1:30

1:30

5 / 10

640Mi / 512Mi

278657

2:42

3:23

2:53

2:59

10 / 20

640Mi / 512Mi

278657

3:00

3:20

3:15

3:11

10 / 20

1280Mi / 1Gi

278657

2:49

2:33

2:34

2:38

The POC codebase could be found in the branch: https://github.com/folio-org/mod-entities-links/tree/MODELINKS-84-long-term-poc

Done

Details

Assignee

Reporter

Labels

Priority

Story Points

Sprint

Development Team

Spitfire

Release

Poppy (R2 2023)

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created May 4, 2023 at 6:32 PM
Updated July 5, 2024 at 12:57 PM
Resolved May 19, 2023 at 2:16 PM
TestRail: Cases
TestRail: Runs