Skip to end of banner
Go to start of banner

Data import - Reliability & Performance testing

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 29 Next »

About this document: Several teams are working on improving data import reliability. From institutions, we need 

  • Job profiles (including match/action/field mapping profiles) 
  • Environment that a hosting provider can run tests (if applicable) 
  • Files to use for testing  
  • Expected time to complete job 
  • Who submitted this information and from which institution

Goals

  • Measure reliability and performance during low and high peak activity periods
  • Measure against competitors benchmarks 
  • Measure goal of handling a large record file of 100,000 
  • Define clear and concise FOLIO benchmarks 


Job profile nameExpected outcome of importEnvironment

Precondition(s)

  • does another profile need to be run beforehand?
  • do we need to edit the data in some way?

File to use for testing

  • please specify number of records in file. 
Can this same file be processed over and over? (Y/N)Reporting institution/reporter

Default - Create instance and SRS MARC Bib

 

A MARC bib and connected FOLIO instance is created 

Available in real-time for search via Inventory app

Orchid BugfestN/A

4900 records (.mrc)

100,000 records_1 (.mrc)

100,000 records_2 (.mrc)

238,000 records (.mrc)

Yes

Default - Create instance and SRS MARC Bib

 

A MARC bib and connected FOLIO instance is created 

Available in real-time for search via Inventory app

Orchid BugfestN/A~400 records (marcxml)Yes

Default - Create instance and SRS MARC Bib

 

A MARC bib and connected FOLIO instance is created 

Available in real-time for search via Inventory app

Orchid BugfestN/A

vanity_fair (json)

NOTE - I cannot get this record to even load. 



Default - Create SRS MARC Authority

  • A MARC authority is created 
  • Available in real-time for search via MARC authority app
Orchid BugfestN/A

1000 records file (.mrc)

5000 records file (.mrc)

10,000 records file (.mrc)

25,000 records file (.mrc)

50,000 records file (.mrc)

~198,000 records file (.mrc)

265,000 records file (.mrc) 

Yes
Default - Create SRS MARC Authority
  • A MARC authority is created 
  • Available in real-time for search via MARC authority app
Orchid Bugfest

2_meetingname records (marcxml)

2_geographic records (marcxml)




Default - Create SRS MARC Authority
  • A MARC authority is created 
  • Available in real-time for search via MARC authority app
Orchid Bugfest

name_title (json)

apple_corporate (json)

NOTE - I cannot even load these records. 



Update a MARC authority record 







Import Harrassowitz invoice
  • Create invoices 
  • Available in real-time for search via Invoices app 
Orchid bugfest
Harrassowitz_folder_1

Import Hein subscription invoice

  • Create invoices 
  • Available in real-time for search via Invoices app 
Orchid bugfest
Hein_folder_1

TBD
  • Import orders in MARC
Orchid bugfest

5_records (.mrc)

~10_records (mrc)



Create MARC holdings 
  • Creates MARC holdings and connected FOLIO holdings record 
  • Available in real-time via Inventory app 

Must create a MARC bib record 


FCBLT Default Inventory UPDATE SRS Record ONLY Using MATCH 999 ff i to 999 ff i (No Override of Protected Fields)

Matches incoming 999 ff $i to existing 999 ff $i

On match updates the marc srs bib

On no match, there's no action

Nolana productionN/A

Note: these files are always 1,000 records, and processing ranges from 2-20 minutes per file. See screenshot for varying duration. 

1000 records (File 1)

1000 records (File 2)

1000 records (File 4) 

1000 records (File 5)

1000 records (File 6) 

1000 records (File 7) 

1000 records (File 8) 

1000 records (File 9)

1000 records (File 3) (Completes with errors - see screenshot

  • 8 errors with message: 
    javax.ws.rs.BadRequestException: Date when processing started is not set, expected snapshot status is PARSING_IN_PROGRESS, actual - FILE_UPLOADED

    or

    system time
  • Reloaded those 8 records with no errors 

34,550 records (Large file) 

Yes5 Colleges
FCBLT Default Inventory UPDATE HOLDINGS Electronic Access ALL Fields (Deletes and Replaces ALL FIELDS) *** [MAKE SURE TO HAVE ADDed the $f for URL relationship to your MRC]***

updating the holdings electronic access field in the holdings record.

Each file has 600 records and the match is to the holdings hrid

Nolana Production

Note: these files have 600 records and times vary from 1-20+ minutes (see screenshot).

600 records (File 1) 

600 records (File 2) 

600 records (File 3) 

600 records (File 4) 

600 records (File 5) 

600 records (File 6) 

600 records (File 7) 

600 records (File 8) 

19,396 records (Large file) 


5 Colleges



  • No labels