[FOLIO-1273] Define and describe the architecture for seamless push and pull of data between apps (interaction) Created: 07/Jun/18  Updated: 21/Jan/22

Status: Open
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: P1
Reporter: Charlotte Whitt Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: acquisitions, crossplatform, crossrmapps, integration, inventory, receiving
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File 1006.png     PNG File 1007.png    
Issue links:
Cloners
is cloned by FOLIO-1331 Define and describe the architecture ... Open
Relates
relates to FOLIO-1712 PubSub support in FOLIO Open
relates to UXPROD-967 eholdings - Order Integration : Autom... Open
relates to UXPROD-1647 Theme: Ability to maintain relationsh... Open
relates to UXPROD-689 Ability to update item record with ca... Closed
relates to UIU-2082 [BE] Data corruption. When holdings/i... Closed
relates to UXPROD-138 Locally-stored metadata records for e... Closed
relates to UXPROD-187 Receive Item and update details regar... Closed
relates to UXPROD-691 Allow Item status to be set during re... Closed
relates to UXPROD-693 Ability to update item record with Ba... Closed
relates to UXPROD-961 Allow Inventory Instances to generate... Closed
relates to UXPROD-968 eholdings - ERM Integration Closed
relates to UXPROD-1080 Locally-stored metadata records for e... Closed
relates to UXPROD-127 A process is needed to periodically r... Draft
relates to UXPROD-139 Avoid creation of duplicative metadat... Draft
relates to UXPROD-151 Ebook packages - relationship to indi... Draft
relates to UIREQ-589 [BE] Data corruption. When holding/it... Open
relates to UIREQ-650 [BE] When inventory data (e.g. barcod... Open
relates to UXPROD-144 Assign Accession numbers to holdings ... Open
relates to UXPROD-150 Automatic update of Bib records updat... Closed
relates to UXPROD-683 Create Instance Record in inventory f... Closed
relates to UXPROD-684 Ability to create an Item Record in i... Closed
relates to UXPROD-686 Create Item Record in inventory for p... Closed
relates to UXPROD-687 Create Holding Record in inventory fo... Closed
relates to UXPROD-2473 Automatic update of Holdings records ... Closed
relates to UXPROD-2474 Automatic update of Authority records... Closed
relates to UXPROD-3399 When Relevant User Record Attributes ... Draft
Sprint:
Development Team: Core: Platform

 Description   

Purpose: Define seamless exchange (push and pull) of

  • data in given elements in one app to given elements in another app, and also
  • handle excange of a complete record, e.g. being created in one app, and later maintained in another app.

NOTE: This will imply determination of how we make sure data is in sync across apps - FOLIO-1331 Open .

Tech lead documentation: https://folio-org.atlassian.net/wiki/display/DD/Data+consistency+and+message+driven+approach?focusedCommentId=65114576#comment-65114576

This area is relevant for interaction between, e.g.:

  • the Acquisition app and Inventory, e.g. when a brief record is created in the Acquisition app, and then pushed to Inventory when the order is send to the vendor
  • the eHoldings app and Inventory app
  • the ERM app and Inventory app, e.g. selected holdings records
  • the Inventory > Item record and the Check in screen, and e.g. piece number, Piece Description, and condition data as missing and damaged items
  • the Inventory > Item record and the Check out screen, e.g. piece number, Piece Description, and condition data as missing and damaged items
  • the Inventory > Item record and the Request Screen, e.g. Call number, Volume and Enumeration ( UIREQ-81 Closed )
  • the Course Reserves record and Inventory > Item. Course > User
  • the Data Import and various apps (MARCcat, Inventory, Acq (orders/invoices)
  • PO lines and Pieces and their links to Instances, Holdings, and Items
  • exchange of records between Inventory and MARCcat
  • eHoldings app and ERM app
  • eHoldings app and Order app

Documentation:



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 06/Jul/18 ]

Charlotte Whitt Khalilah Gambrell

Would this also cover interactions between:

Batch Loader and various apps (MARCcat, Inventory, Acq (orders/invoices)
Inventory and MARCcat

If this is basically, the internal infrastructure for pushing/pulling data around between apps, then it seems like it would.

Comment by Charlotte Whitt [ 06/Jul/18 ]

Hi Ann-Marie Breaux - this is the basically internal infrastructure - and I don't know if "one size fits all"

In my description I have listed Cross RM app infrastructure features, and Loan data in Inventory etc. But I'll add a couple more bullet point to the description.

Comment by Mike Taylor [ 09/Jul/18 ]

(Note to self: Charlotte and I will figure out who to convene for a small technical-aspects meeting. Once we have a technical handle on the broad problem, we can bring it back to the wider group.)

Comment by Mike Taylor [ 10/Jul/18 ]

Part of the underlying solution to this issue is going to be formalising the notion of an object's type. At present, the type is always implicit – i.e. those things that we deal with in the Users app are users. But once we start passing objects around between apps, those objects will need to know their own types.

This is also a concept that has cropped up in a few other places – for example, when attaching notes to an object, the note needs to know both the type and ID of that object. We presently have some kludges in place that allow this to work, but the time is probably right to address this issue properly, perhaps by introducing an explicit type registry. This would include metadata about each type, including things like:

  • its name (user, instance etc.)
  • the Okapi path used to obtain objects of this type (/users, /inventory-storage/instances, etc.)
  • the preferred app for creating and maintaining objects of this type
  • maybe some indication of what operations can be executed on it
  • maybe other things that we've not thought of yet

I've added Jason Skomorowski and Filip Jakobsen to the watchers list for this issue, because I know that this is something they've both given some thought to.

Comment by Ann-Marie Breaux (Inactive) [ 10/Jul/18 ]

Thanks for this additional info, Mike Taylor. I know UI is a different animal, but there's a similar need on the UI side to what you describe with regards to object types. For example, when we get to the point of central admin of Tags, that app will need to understand the record types using a particular tag and where that type of record is stored/maintained, so that CRUD can happen to the same tag on different record types across various apps. I've been creating a list of record types that each UI app accesses, so that we can figure out whether tags will be allowed for that record type, whether it has an icon to set it apart in mixed search result lists, etc.

Comment by Filip Jakobsen [ 11/Jul/18 ]

I think an important, related, thing to consider when doing the architecture for this, is the notion of embedded views — in e.g. To-do, a very important feature is the ability to show the UI to perform a task right inside the To-do app. However, the view should not be defined by To-do, but by Orders, Users, Inventory, Finances and other apps. I am not sure if this directly ties into this discussion, or only indirectly, but I think it is relevant for everyone to be aware of.

Comment by Filip Jakobsen [ 11/Jul/18 ]

Another notion that I think is directly connected to this discussion, is the notion of record previews—we need this for almost anywhere a list of data is shown from another app. E.g. in the generic Scan application, in Tags, in Check in, Check out, in Users, and more.

Essentially, my vision is that we supply a basic set of requirements for the metadata of the record types registered in the system, and use that data to provide a "record preview" to other apps. Much like websites today define a photo, headline and description for how a link appears when it is shared on social media or in instant messaging apps.

Comment by Filip Jakobsen [ 11/Jul/18 ]

In relation to previews, it might also be that we simply want to set up a standard for how record types and individual records provide their data, and then let e.g. the MCL with its (given time) responsive capabilities handle, based on various variables, including user preferences, whether something is to be shown in a S, M, L cards layout or a list item, etc.

Comment by Mike Taylor [ 11/Jul/18 ]

Thanks, Ann-Marie Breaux, for raising the issue of tags. At present these are implemented in a fundamentally different way from notes – for what were at the time good reasons – but we may need to change that going forward, so that each tag has its own existence. IIRC we briefly discussed this in Durham, but it rather got lost in the flood of other discussions! So, yes, this would then become another candidate area where the concept of object type would become important.

Filip Jakobsen I think the issue of embedded mini-apps is orthogonal to this, and should be handled adequately by the existing Plugin facilities (though we will want to firm up some policies about exactly how to do this).

I think the same is true of your "record previews". The first issue is that of determining what kind of object we're dealing with. Once we know that, we should be able to look up in the type registry to determine which app to use for displaying a preview, and the plugin system can then arrange for the relevant component from that app to be used.

Comment by Charlotte Whitt [ 20/Jul/18 ]

Jakub Skoczen Cate Boerema who can we assign this to?

See also latest Discuss post by Tod Olson (uChicago); with comments from VBar and Jakub Skoczen: https://discuss.folio.org/t/on-distributed-updates-and-eventual-consistency/1966

Comment by Hkaplanian [ 23/Jul/18 ]

When we had this discussion/meeting in Madrid, Jakub and Vince both agreed that the different apps would need to call the API's of the other apps that needed updates or to retrieve data. The specific example of acquisitions needed to update inventory with a holding was used. I'm curious what brought this up again. Has the thinking changed?

Comment by Mike Taylor [ 24/Jul/18 ]

I think the reason it's cropped up again – not just now, but on other occasions – is because we have new people joining the project all the time, and they come with concerns that are not directly addressed by any our documents. We really need a FOLIO FAQ that explains why we made some of the decisions we made, so we can point newcomers to it and avoid repeatedly rehearsing the same issues.

Comment by Ann-Marie Breaux (Inactive) [ 19/Jan/21 ]

Hi Charlotte Whitt I'm not sure how you want to change the description to reflect that MARCcat is no longer being developed, so I left it as-is. Please update however you think is appropriate. Note that quickMARC will be accessing the SRS MARC directly, rather than having separate storage like MARCcat.

Comment by Kelly Drake [ 26/Mar/21 ]

Hi all,  I updated the priority to P1.  After discussion among a number of POs, it was felt that the time has come in FOLIO's maturity to address this issue.

Generated at Thu Feb 08 23:12:06 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.