SPIKE: Figure out the details of migrating MARC bib records from Inv to SRS
Description
Environment
Potential Workaround
CSP Request Details
CSP Approved
CSP Rejection Details
Attachments
defines
relates to
Checklist
hideTestRail: Results
Activity
Ann-Marie BreauxJanuary 14, 2019 at 4:18 PMEdited
Hi
No apologies needed!
From Mark: Does this reflect your comment:
When a bibliographic MARC source record changes, these changes need reflecting in the corresponding inventory instance record - YES
When a bibliographic MARC source record changes, these changes need reflecting in the corresponding MARCcat bibliographic record - YES
When a MARCcat bibliographic record changes, these changes need reflecting in the corresponding bibliographic MARC source record - YES, which then in turn updates the corresponding Inv instance record
When an inventory instance record changes, these changes do not need reflecting in any other records (or are inventory records with a corresponding bibliographic MARC source record not editable?) NO - If an Instance is linked to a MARC record, then the bibliograophic data of the Instance is not supposed to be editable in Inventory (though it still is editable right now - not sure if that's a bug or it's just not connected properly yet. Do you know, ? The non-bibliographic portions of the Instance record will remain editable even if a MARC record is linked to it. Those non-bibliographic aspects would include things like tags and reporting codes - data elements that don't have a home in the MARC record.
It needs to be possible to generate a bibliographic MARC source record from an inventory instance - YES, I don't think it has been decided yet whether that is on-demand, or via a default setting across a tenant.
Has it been agreed how the system knows which records correspond with each other? I don't know - good question for tomorrow.
My other comment was only intended to raise awareness of the current technical constraints which may affect the design for these requirements.
Marc JohnsonJanuary 14, 2019 at 12:22 PM
Apologies I could've been more mindful of the origins of the diagram, I wasn't intending to suggest it was wrong or cause offence, only to understand it's meaning better (and how FOLIO's architecture might interact with that).
Thank you for the additional context. I think I am likely getting my terminology mixed up. Does this reflect your comment:
When a bibliographic MARC source record changes, these changes need reflecting in the corresponding inventory instance record
When a bibliographic MARC source record changes, these changes need reflecting in the corresponding MARCcat bibliographic record
When a MARCcat bibliographic record changes, these changes need reflecting in the corresponding bibliographic MARC source record
When an inventory instance record changes, these changes do not need reflecting in any other records (or are inventory records with a corresponding bibliographic MARC source record not editable?)
It needs to be possible to generate a bibliographic MARC source record from an inventory instance
Has it been agreed how the system knows which records correspond with each other?
My other comment was only intended to raise awareness of the current technical constraints which may affect the design for these requirements.
Ann-Marie BreauxJanuary 10, 2019 at 3:56 PM
Sorry - one more comment - there would be a similar relationship between SRS and MARCcat storage. Any changes to SRS records need to be automatically reflected in their MARCcat storage counterparts and vice versa.
Ann-Marie BreauxJanuary 10, 2019 at 3:51 PMEdited
If updates are made to a MARC record in SRS (either by an updated version being imported, or by manual updates via MARCcat), then those changes need to be reflected automatically in the corresponding Inventory Instance record.
For the other direction: if the Instance is underpinned by a MARC record, we would not expect the Instance to be making changes to the SRS record. BUT: if the Instance is not underpinned by MARC, we're aiming for the user to be able to tell FOLIO to create a brief MARC record, which would then be stored in SRS.
So I think that's why you're seeing the bi-directional arrow. Keep in mind this is a diagram that we POs built, not technical people!
Same thing will apply to Inventory Holdings and MARC Holdings once we are able to load and relate MARC holdings records to Inventory holdings records.
Marc JohnsonJanuary 10, 2019 at 11:24 AM
In the attached diagram, there are bidirectional lines between instance storage and MARC bib storage (which I assume is now called source record storage, or is it something else?), and between holding record storage and MARC holdings storage.
At the moment, FOLIO does not allow interactions between storage modules (either at the API or database tiers) and does not support bidirectional dependencies between modules (as this creates circular dependency resolution which stops modules from being activated within Okapi).
These arrows suggest to me some kind of bidirectional data relationship between these modules. Is my understanding incorrect?
Details
Assignee
Taras SpashchenkoTaras SpashchenkoReporter
Ann-Marie BreauxAnn-Marie Breaux(Deactivated)Labels
Priority
P3Story Points
5Sprint
NoneDevelopment Team
FolijetFix versions
TestRail: Cases
Open TestRail: CasesTestRail: Runs
Open TestRail: Runs
Details
Details
Assignee
Taras SpashchenkoReporter
Ann-Marie BreauxLabels
Priority
P3
Currently MARC bib records are being stored in Inventory, and linked to related Inventory Instances. In the future, they will be stored in Source Record Storage (SRS). Once SRS is ready to receive them, then we need to have an understanding of what work will be involved in changing the existing configuration to the new SRS configuration.
Section 3 in this diagram lays out the high level architecture of Data Import, with links to technical diagrams, including SRS:
https://docs.google.com/document/d/1bIBtvixY-t-xNHy-1EnIEei11KvWyLofrllZVaR0CfU/edit#
Stakeholders (set up as watchers)
Inventory: , ,
Data Import: Ann-Marie, Taras, Alex, Tetyana, anyone else?
OAI-PMH: ,
Misc: ,
MARCcat: , , ,
EBSCO dev:
Anyone else?
Details to figure out
When will SRS be ready to accept data?
Does Source Record Manager (SRM) need to be ready as well? If so, when will it be ready?
Questions/dev review of the SRS/SRM APIs that will be used to link other apps to SRS data [ will add details]
What is involved in removing existing MARC Bibs from Inventory?
Per Charlotte, MARC Bibs are loaded to the various FOLIO servers, into mod-inventory-storage, every morning (or every time a FOLIO instance is built). Removing the MARC bibs from mod-inventory-storage is thus achieved by stop loading them.
How will current linkage between Inv MARC and Inv Instance need to change?
Per Charlotte, all Inventory needs is for the data loader to continue to set the property Instance.sourceRecordFormat when it loads a MARC source and creates an Instance in Inventory from SRS. UI-inventory needs this to determine whether to display a "Display source" button on the instance in the UI. See: , and .
Are there any other apps or components interacting with Inv MARC Bibs currently? e.g. OAI-PMH?
Per Craig, OAI-PMH will be retrieving records from SRS. The two teams are working on integrating.already.
There's an inventory setting to suppress output for discovery for all 3 record types (instances, holdings, items). When MARC moves to SRS (instances, then holdings), how will that be handled?
What is the best timing for the change?
Per Charlotte, anytime, but it must be coordinated with
Wayne Schneider who loads the data
Niels Erik Gilvad Nielsen who needs to change the link to the 'View source' button on Instance detail view
Is MARCcat storage built and ready to accept records yet?
Yes, but not yet linked to Inventory or SRS.
Will we need an interim solution and then refactor when MARCcat storage ready to accept records?
How will this affect folio-testing, folio-snapshot, folio-snapshot-stable?
Per Taras, the environments are created from scratch each time, so will need to reload the data to SRS, Inv, and MARcat
Once we have the ID environments working, perhaps try demo.folio.org next, since there are lots of MARC records there?
What happens with the sample data each day when the environments are rebuilt? Do we have to re-load to SRS, or can it just stay there and not have to be re-loaded each time?
Per Taras, the answer is yes, because as far as I know environments are created from scratch. but we need more details for each environment.
Which dev team(s) build which portion(s)?
Build wiki page with details, so that all related devs and POs can track decisions and progress? Or just track here in this spike and whatever Jira stories come from it?
How will we test?
From Taras in the comments section:
The first steps I propose to take to start off:
1. SRS API documentation is available at https://s3.amazonaws.com/foliodocs/api/mod-source-record-storage/source-record-storage.html#
to get MARC records use GET request to the /source-storage/result endpoint https://s3.amazonaws.com/foliodocs/api/mod-source-record-storage/source-record-storage.html#source_storage_result_get
2. SRS module for now does not contain MARC data. There is a story in Folijet backlog https://folio-org.atlassian.net/browse/MODSOURCE-5 “Develop a CLI tool to allow initial filling with data for testing purposes.” To address this.
3. Find the person, who is responsible for the daily MARC loading into mod-inventory storage. Add the same jobs for SRS using SRS-CLI tool.
4. Change mod-inventory implementation for ExternalStorageCollections classes to read MARC records from SRS module instead of mod-inventory-storage. Maybe it make sense to preserve existing implementation as a fallback and make it configurable.