Migration Tools (UXPROD-850)

[UXPROD-559] Migrate Bibliographic & Holdings data Created: 07/May/18  Updated: 23/Oct/20  Resolved: 23/Oct/20

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: None
Parent: Migration Tools

Type: New Feature Priority: P3
Reporter: Cate Boerema (Inactive) Assignee: Ian Walls
Resolution: Done Votes: 0
Labels: inventory, library_dependent, migration-load, po-mvp, resourceaccess
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
is defined by MODSOURCE-182 Use SQL JOIN in /records query method Closed
is defined by MODSOURCE-183 Add bulk query/streaming support to /... Closed
Relates
relates to UXPROD-47 Batch Importer (Bib/Acq) Analysis Complete
relates to MODINVSTOR-543 Holdings. Holdings statements notes. ... Closed
relates to UIIN-1215 Holdings. Holdings statements notes. ... Closed
relates to UXPROD-2450 Holdings record. Holdings statements ... Closed
Epic Link: Migration Tools
Front End Estimate: Large < 10 days
Front End Estimator: Jakub Skoczen
Back End Estimate: Large < 10 days
Back End Estimator: Jakub Skoczen
Estimation Notes and Assumptions: Not sure what this features involves.
PO Rank: 120
PO Ranking Note: Core functionality for any migration; you have to be able to get the bibliographic and holdings data in there before you can do much else.
Rank: Chalmers (Impl Aut 2019): R5
Rank: Chicago (MVP Sum 2020): R5
Rank: Cornell (Full Sum 2021): R5
Rank: Duke (Full Sum 2021): R5
Rank: 5Colleges (Full Jul 2021): R5
Rank: FLO (MVP Sum 2020): R1
Rank: GBV (MVP Sum 2020): R2
Rank: hbz (TBD): R1
Rank: Hungary (MVP End 2020): R1
Rank: Lehigh (MVP Summer 2020): R2
Rank: Leipzig (Full TBD): R1
Rank: TAMU (MVP Jan 2021): R4
Rank: U of AL (MVP Oct 2020): R2

 Description   

Depends on the pre-existing ILS and 1st libraries moving over.

Documentation: Bib Holdings Instance Migration Loader Requirements



 Comments   
Comment by Tod Olson [ 28/Jun/18 ]

Patty and I are fleshing out requirements for this feature and have a scope question. Comprehensively, migrating bib/holding/item data would include importing to all of the places where that data gets stored (inventory, native MARC storage, MARCcat?) but often we talk about this only in the context of inventory. Which scope should we write for?

Comment by Cate Boerema (Inactive) [ 02/Jul/18 ]

Hi Tod Olson, Ann-Marie Breaux may have some tips on how to talk about MARC import in your requirements. You can probably be pretty specific about expected outcomes in Inventory, since that app already exists. MARCcat expectations will need to be more vague at this point.

This diagram may also help (Ann-Marie Breaux, let us know if this isn't the latest). Thanks!

Comment by Ann-Marie Breaux (Inactive) [ 02/Jul/18 ]

Hi Tod Olson You may want to take a look at UXPROD-47 Analysis Complete and the associated stories. UXPROD-47 Analysis Complete is for ongoing day-to-day batch loading, not initial migration, bu it may give you a feel for what all we'll be covering. In terms of scope, we're aiming to support load of MARC bibs, MARC holdings, MARC authorities, EDIFACT invoices, and delimited files, affecting records in the following areas of FOLIO:
1. mod-source-storage (where the copy of record is retained, passed to other areas of FOLIO, and updated with any changes from other areas)
2. Inventory: instance, holdings, item (maybe package, but we haven't discussed yet)
3. MARCcat: bib record, holdings record, authority record
4. Acquisitions: order record, invoice record

Christie is on the batch loader subgroup, so you may want to check in with her. All our notes and recordings are accessible at https://folio-org.atlassian.net/wiki/display/MM/Batch+Loader+Subgroup. Let me know if it would be helpful to have a chat about any of this, or perhaps for SysOps and the batch loader group to have a joint meeting to discuss at some point (not this week - we lose too many people due to the holidays).

Comment by Hkaplanian [ 17/Jul/18 ]

Adding a link to UXPROD-47 Analysis Complete since they are so similar. I'm trying to understand why this would be built in addition to UXPROD-47 Analysis Complete . Will work with the SIG to gain understanding of differences.

Comment by Ingolf Kuss [ 18/Jul/18 ]

I think we listed the differences in this document [https://docs.google.com/document/d/1oXbEE48zd889lGD87dP7cF3GfuuKwllI_MDp2zGwTRg/edit|BIB HOLDINGS INSTANCE Migration LOADER requirements]. Some requirements are marked "common with batch loader, UXPROD-665", some are not. This document has been approved by the data migration subgroup.
As far as I remember, loading speed was an issue. The migration loader must perform quickly (but this could also be true for the batch loader). We also agreed that both loaders will have the same 'guts', but need different interfaces for initial data migration and day-to-day loading. So, we should specify the features and user stories for the 'guts' only once, at one place.
One requirement that has not been formulated for the batch loader is that administrative metadata for bib, holding, and item records must be preserved (see in the word document mentioned avove).

Comment by Cate Boerema (Inactive) [ 27/Jul/18 ]

Hi Tod Olson I just assigned this to you (assignees in UXPROD are the POs) and marked it In Progress to indicate analysis is underway (that's what In progress means in UXPROD). Just FYI.

Comment by Charlotte Whitt [ 31/Jul/18 ]

Follow-up (on yesterday's meeting with patty.wanninger Tod Olson Cate Boerema:
Theodor Tolstoy (One-Group.se) explained that Chalmers will migrate bibliographic and holdings data with a homegrown migration loader program; therefor Chalmers do not need a FOLIO migration tool.

Comment by patty.wanninger [ 27/Dec/18 ]

Assigned to me as PO, added tag migration-load.

Comment by Ann-Marie Breaux (Inactive) [ 07/Mar/19 ]

HI patty.wanninger would this include loading item record data as well (not circ history, but whatever item record data needs to be loaded)? If so, could you adjust the title of the feature, or is it OK for me to do it? Thank you!

Comment by patty.wanninger [ 07/Mar/19 ]

Ann-Marie Breaux, there's a separate feature for loading circulation data; https://folio-org.atlassian.net/browse/UXPROD-558, rather anemically described at this point.

Comment by Ann-Marie Breaux (Inactive) [ 07/Mar/19 ]

HI patty.wanninger I was assuming UXPROD-558 Open was circ history/policies, etc - not the item records themselves. Since the item records live in Inventory. I was assuming it's better to have them lumped with the other Inventory records (Instances/Holdings). What do you think?

Comment by Anya [ 29/Mar/19 ]

Comment from the March meeting : format = JSON or MARC-21 Bib

Comment by Karen Newbery [ 24/Apr/19 ]

Has it been determined that we're able to use MARC holdings, and does this issue cover loading MARC holdings?

Comment by patty.wanninger [ 24/Apr/19 ]

Karen Newbery You will have some options regarding holdings but FOLIO will support MARC format for holdings and you will be able to load them through a similar process as the one being developed for marc bibs - Marc records are submitted to the Source Data Storage, and then transformed into JSON to be presented to the Inventory. Current the mapping is in draft. Also, most early implementers are experimenting with direct loads into the inventory using custom scripts. DM me for more.

Comment by Ann-Marie Breaux (Inactive) [ 11/Jun/19 ]

Anya Q3 2020 doesn't sound too helpful for a migration feature, if many of the libraries are planning to go live at the start of Q3 2020. Are the Fix versions that you're adding definite, or just a starting point for discussions?

Comment by Anya [ 11/Jun/19 ]

Ann-Marie Breaux Hopefully they are a starting point - I am getting these off the capacity plan - currently, there is no dev team assigned to this - much like all the course reserves stuff

Comment by Ann-Marie Breaux (Inactive) [ 11/Jun/19 ]

Sounds good - thanks, Anya. Definitely will be fodder for conversation next week.

Comment by Charlotte Whitt [ 30/Jul/19 ]

Hi Ian Walls - I noticed that you have only given this 'core function' PO rank "1". That is extremely low rank for a feature all libraries except Chalmers and uChicago define as 'Go live'.

I would have expected maybe PO rank: 130-140 for this feature which you also has given PO-mvp

Comment by Ian Walls [ 30/Jul/19 ]

Apologies; misunderstood the order of PO ranks. Re-ranking

Comment by Holly Mistlebauer [ 11/Sep/19 ]

Ian Walls: Has this feature been completed or is it close to completion? Thanks!

Comment by Ian Walls [ 23/Oct/20 ]

By using the bulk Instance and Holdings endpoints, as well as the (as yet slow) SRS bulk endpoints, this can be accomplished.

Generated at Fri Feb 09 00:08:48 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.