[FOLIO-2030] Set up data-loading hosted environment Created: 20/May/19  Updated: 03/Jun/20  Resolved: 24/May/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P3
Reporter: Ann-Marie Breaux (Inactive) Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: ci, data-import, devops, performance, platform-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File CornellFOLIOExemplars_Bibs.mrc    
Issue links:
Relates
relates to MODDATAIMP-124 Measure data-import performance Closed
Sprint: CP: sprint 64
Story Points: 2
Development Team: Core: Platform

 Description   

Purpose: We need an environment to test loading of files and their impact on various apps, OKAPI, and other parts of FOLIO, without impeding the function of folio-testing and folio-snapshot

As a staff person
I want to be able to load files of all record types, starting with MARC files (creating SRS MARC and Inventory Instances)
So that I can ensure batch functionality works as expected, plus test system performance

A small MARC file attached, and here's a link to a large MARC file (143 MB), in case they would be useful: https://drive.google.com/file/d/169Tnsa1KJoVwsHaSZfkML5jj9Zke9ESS/view?usp=sharing

Requirements

  • Create a hosted environment called folio-snapshot-data-loading that mirrors folio-snapshot
  • Consider adding to OKAPI until event-driven handling is implemented (discuss more with Oleksii Kuzminov)
  • Ann-Marie Breaux make sure community knows to use this environment when loading MARC files

Wayne Schneider Per Oleksii Kuzminov, Folijet can set a limiter on the size of MARC file being loaded until we have event-driven handling in place, which will then take the load off of OKAPI. Please advise what we should set as the maximum file size for now.

Per Oleksii Kuzminov, once Kateryna Senchenko's data import performance testing is complete, we'll have a better understanding of current performance.



 Comments   
Comment by Ian Hardy [ 20/May/19 ]

Hi Ann-Marie Breaux, just to clarify, are you looking for a one-off instance that will persist for as long as you need to test this feature, or something that gets rebuilt?

Comment by Ann-Marie Breaux (Inactive) [ 20/May/19 ]

Hi Ian Hardy Something that gets rebuilt, and basically stays in sync with folio-snapshot

Comment by Oleksii Kuzminov [ 21/May/19 ]

Ian Hardy we need to rebuild it like a snapshot env

Comment by Ian Hardy [ 23/May/19 ]

https://folio-snapshot-load.aws.indexdata.com/ is available for loading data or other activities that might be a burden on the other shared environments. Its an equivalent configuration to folio-snapshot. The backend is at https://folio-snapshot-load-okapi.aws.indexdata.com/.

Generated at Thu Feb 08 23:17:40 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.