DCB Integration

Submitted	Jun 15, 2023
Approved
Status	DONE
Impact	high
Arch Ticket	https://folio-org.atlassian.net/browse/ARCH-99
Prod ticket	https://folio-org.atlassian.net/browse/UXPROD-4325

Glossary

Term	Description

Term	Description
DCB	Direct Consortia Borrowing System
DCB Search UI	Consolidated search for items across a consortium
Item	Physical or virtual item
Patron	Patron registered in borrowing institution

Executive Summary

Integration with DCB allows support for cross-ILS circulation of items. FOLIO can participate in cross-ILS circulation in the role of requester institution, owning institution, and pickup location. DCB will act as an orchestrator for coordinating multi-system circulation transactions. The main goal from the perspective of internal integration implementation for FOLIO is to provide circulation for external items without affecting the existing local circulation flow. FOLIO is implemented as an application with microservice architecture and should reuse the same approach for integration with DCB. The current direction of FOLIO's technological stack suggests that the approach should include integration with third-party systems through edge modules and implementation of microservices should be based on a spring-way approach.

One of the main obstacles to the implementation is that FOLIO's inventory contains information regarding items and circulation of items that are out of inventory context boundaries and overlap with circulation context. This is the reason that it is a challenge to create a virtual item for the FOLIO installation so that it wouldn't leak the properties of virtual items in search results of local items. The target solution for this requires the migration of information related to circulation from inventory and moving it to the context of the circulation module.

The circulation flow inside FOLIO is implemented in a "manager/storage" approach as a RAML module and provides additional limitations because of the complexity of introducing new changes to the circulation module. Because of time constraints, the target solution might be difficult to implement. so it is proposed to create an interim solution that can be used as a basis for migration of the circulation module to spring-way in the future. The solution requires the introduction of a circulation item that would abstract the circulation information from the inventory context. The new module will be implemented as a separate circulation-item module that should serve as a proxy/adapter between the circulation module and the inventory module.

Requirements

Functional Requirements

Supported use cases described in https://folio-org.atlassian.net/browse/UXPROD-4325

Business Flow Diagram: https://miro.com/app/board/uXjVP9HDWqw=/?moveToWidget=3458764545695411304&cot=14

Scenario 1: Circulation from Lending library to Pickup Library

Scenario 1: Circulation from Lending library to Borrowing Library

Non-Functional Requirements

Modifiability: The solution should be loose-coupled and support the existing circulation flow
Performance: FOLIO edge modules should answer under <X> ms
Security: FOLIO edge modules should authorize DCB requests through the API keys mechanism
Scalability: new microservices should support a microservice architecture approach and be stateless

Constraints

API communication from DCB to FOLIO should follow a polling approach. FOLIO should provide a set of related REST APIs
Status check requests can happen with high frequency because of the polling mechanism
The integration implementation should be finished inside ECS initial release.

Risks

Possible additional UI work for different modules related to Umbrella approach.

Assumptions

DCB uses a polling approach to communicate with FOLIO. Hence FOLIO should provide REST API endpoints
DCB is able to authenticate patron. Patron identity is supplied by DCB and known to the FOLIO borrowing library.
DCB is able to provide information on the requested item to the lending library. Item information is known through RTAC and reflects up-to-date information from the inventory of FOLIO
The pickup location can be separate from the borrowing library.
In case the lending or pickup institution was changed DCB should create a new transaction
Request canceling can happen at any point in the lifetime of the borrowing/lending request
FOLIO should support borrowing, lending, and pickup role. Implementation should support the following action for requests:
- Creation of virtual patrons in lending and pickup libraries
- Creation of virtual items in borrowing and pickup libraries
- Creation/update/cancel/status for the lending request
- Creation/update/cancel/status for borrowing request
- Creation/update/cancel/status for pickup request

Solution Options

	Solution	Description	Pros	Cons	HLE	Decision

	Solution	Description	Pros	Cons	HLE	Decision
1	Migration of all of mod-circulation to new spring module	Introduction of new circulationItem that would abstract the actual item from the circulation process. Implementation requires moving properties related to the process of circulation from `mod-inventory` to the new circulationItem entity and moving the existing circulation flow to the spring module.	Context of circulation is extracted from the inventory Target technological stack is used for all circulation flow Context boundaries are well-defined and stay inside a single module	The migration of the existing RAML module to the spring-boot module would require significant development effort.
2	Introduction of virtual item/patron in a separate module	Introduction of new circulationItem that would abstract the actual item from the circulation process. Implementation requires moving properties related to the process of circulation from `mod-inventory` to the new circulationItem entity.	Context of circulation is extracted from the inventory Springboot technological stack can be used	Circulation context boundaries are extended to one additional module. Requires proxying requests from `mod-circulation` to `mod-inventory.` Might require additional HTTP request hops due to routing through OKAPI if implemented in a separate module. Concerns related to virtual items (Problem statements 1, 2 and 3).	XXL	Declined. Introduction of virtual item requires additional virtualizing of inventory entities like instance/holding etc.
3	Supporting existing API for the creation of temporary items/patrons	The implementation should use existing API and approach with the creation of temporary patrons/items.	Requires development of an edge module and little to no changes in other parts of the system. The item detail and patron detail will be intact and will be displayed appropriately on the UI.	The item created will be an actual Item in inventory. The circulation context wouldn't be extracted from inventory. Requires implicit exclusion from search results. Current suppression mechanisms have leakage and are not fully addressed in data export. Data integrity could be an issue due to microservice architecture and dependency management between modules	M-L	Declined. Usage of suppression from discovery mechanism involves side-effects in different FOLIO applications like OAI-PMH, mod-search, data-export
4	Implementation of the creation of virtual patrons and items inside of existing `mod-circulation`		Context of `mod-circulation` stays inside existing manager/storage modules related to circulation. Context of circulation is extracted from the inventory.	Due to manager/storage approach in circulation modules logic will be implemented in two modules. Due to the asynchronous approach with vert.x and RMB modules the complexity of the implementation will be significant. Concerns related to virtual items (Problem statements 1, 2 and 3).
5	Implemenation of virtual item with single instance/holding object	Only item is virtualized. Single Instance and holding is created in inventory and used for all DCB Items in lending and pickup libraries.	Little to none effort to remove leakage of virtual instances. Aligned with strategic vision but target solution (proper separation of inventory and circulation context) will require rework.	UX/UI: requires additional UI development to distinct DCB items. Might be confusing for users More dev effort compared to real instances creation, less then full virtual item/instance/holding Edge-patron: might require additional investigation for items/loans edge API’s Extended flow with Renew operations is not analyzed		In Discussion. Aligned with strategic vision

Target Architecture (Solution Option 2)

The solution consists of the following components:

edge-dcb - microservice responsible for providing REST API for DCB and API key authorization.
mod-dcb - microservice responsible for persisting state of DCB transactions and coordination of internal calls inside FOLIO. Should listen for updates from mod-circulation to reflect the state of DCB transactions to external consumers.
mod-circulation-item - should implement an adapter/proxy approach for the virtual item or patron for circulation. The adapter/proxy approach should allow the breaking of dependency between circulation and actual/virtual items. The module should be able to:
1. Create/Get/Update virtual item (circulationItem)
2. Substitute calls to mod-inventory for items and when the request is related to DCB return Virtual Item
Mod-circulation - should not require significant changes

Diagram Source: Volaris-DCB.drawio (2).xml

API Contracts

Open API Description: API Contract

Entity Relationship Diagram

The target solution introduces a new abstraction for an item in circulation context: CirculationItem. This serves as a proxy for items that are present in mod-inventory and, in case of borrowing and pickup libraries, they represent an item that exists in other institutions. DCB Transaction represents a facade object for the synchronization of different entities in FOLIO.

^@startuml

^{entity CirculationTransaction {}
^{*Circulation Transaction ID: UUID}
^--
^{Status: varchar}
^{Created: timestamp}
^{Updated: timestamp}
^}

^{entity CirculationPatron {}
^{*Patron UUID: UUID}
^--
^{Patron Group: varchar}
^{Patron Barcode: varchar}
^{Library Code: varchar}
^{Transaction ID: UUID}
^}

^{entity CirculationItem {}
^{*Item UUID: UUID}
^--
^{Instance Title: varchar}
^{Item Barcode: varchar}
^{Pickup Location: Location ID}
^{Item Material Type: enum}
^{Library Code: varchar}
^{Transaction ID: UUID}
^}

^{entity CirculationProxy {}
^{*Item UUID: UUID}
^--
^{Instance Title: varchar}
^{Item Barcode: varchar}
^{Pickup Location: Location ID}
^{Item Material Type: enum}
^{Library Code: varchar}
^{Transaction ID: UUID}
^}

^{entity CirculationDetails {}
^{*Item UUID}
^--
^Status
^Location
^}

^{entity CirculationNotes {}
^{*Item UUID}
^--
^{Note Type: [CheckIn, Checkout]}
^{Note: varchar}
^{StaffOnly: boolean}
^}

^{entity Loan {}
^{*Loan UUID: UUID}
^--
^{*Patron UUID: <<FK>> Patron}
^{*Item UUID: <<FK>> CirculationProxy}

^}

^{entity Item {}
^{*Item UUID: UUID}
^--
^{...Item Properties}
^}

^{entity Patron {}
^{*Patron UUID: UUID}
^--
^{...Patron Properties}
^}

^{entity Instance {}
^{*Instance UUID: UUID}
^--
^{...Instance Properties}
^}

^{entity Holding {}
^{*Holding UUID: UUID}
^--
^{...Holding Properties}
^}

^{CirculationTransaction ||--|| CirculationItem}
^{CirculationTransaction ||--|| CirculationPatron}
^{CirculationPatron ||--o| Patron}
^{CirculationItem ||--o| CirculationProxy}
^{CirculationProxy ||--o| Item}
^{Patron ||--|{ CirculationProxy}
^{CirculationProxy ||--|{ Loan}
^{CirculationProxy ||--|| CirculationDetails}
^{CirculationProxy ||--|{ CirculationNotes}

^{Instance ||--|{ Holding}
^{Holding ||--|{ Item}

^@enduml

Interaction Sequence Diagram

Coordination of transactions in different libraries is the responsibility of DCB. The communication mechanism is polling from DCB.

^@startuml
^autonumber

^{actor "DCB User" as lu}
^{participant "DCB Search UI" as l}
^{participant "OpenRS" as dcb}
^{box FOLIO}
^{participant "Borrowing Institution" as bor}
^{participant "Lending Institution" as lend}
^{participant "Pickup Institution" as pickup}
^{end box}

^{lu -> l++: Request Item}
^{l -> dcb ++: Request Item}
^{'==0. Prerequisites [] ==}
^{'dcb -> bor++: Create Virtual Item}
^{'return Item Created}
^{'dcb -> lend++: Create Virtual Patron}
^{'return Patron Created}
^{'dcb -> pickup++: Create Virtual Item and Patron}
^{'return Item/Patron Created}

^{==1. Create Transaction==}
^{dcb -> bor++: Create Borrowing DCB Transaction [POST /transactions/{id}]}
^{return DCB Transaction Created}
^{dcb -> lend++: Create Lending DCB Transaction [POST /transactions/{id}]}
^{return DCB Transaction Created}
^{dcb -> pickup ++: Create Pickup DCB Transaction [POST /transactions/{id}]}
^{return DCB Transaction Created}

^{return Item Request Created}
^{return Item Request In Progress}

^{==2. Coordinate Transaction==}

^{==2.1 Coordinate Borrowing==}
^{dcb -> dcb++: On timer:}
^{dcb -> bor++: Borrowing DCB Transaction Status [GET /transactions/{id}/status]}
^{return Status}
^{alt Borrowing DCB Transaction Status Changed}
^{dcb -> lend++: Update Lending DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{dcb -> pickup++: Update Pickup DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{end alt}
^{return ok}

^{==2.2 Coordinate Lending==}
^{dcb -> dcb++: On timer:}
^{dcb -> lend++: Lending DCB Transaction Status [GET /transactions/{id}/status]}
^{return Status}
^{alt Lending DCB Transaction Status Changed}
^{dcb -> bor++: Update Borrowing DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{dcb -> pickup++: Update Pickup DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{end alt}
^{return ok}

^{==2.3 Coordinate Pickup==}
^{dcb -> dcb++: On timer:}
^{dcb -> pickup++: Pickup DCB Transaction Status [GET /transactions/{id}/status]}
^{return Status}
^{alt Pickup DCB Transaction Status Changed}
^{dcb -> bor++: Update Borrowing DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{dcb -> lend++: Update Lending DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{end alt}
^{return ok}

^@enduml

State-Transition Diagram

Scenario: Lending Library to Pickup Library

Scenario: Lending Library to Borrowing Library

Work Breakdown Structure

edge-dcb:
1. Create a module based on edge-common-spring
2. implement API contracts and proxy calls to mod-dcb
mod-dcb:
1. Create a module based on mod-spring-template
2. Create liquibase scripts for database structure for DCBTransaction, DCBItem, and DCBPatron entities
3. Create circulation items in mod-circulation-item
4. Create temporary patron in mod-patron
5. Listen for updates on the circulation process in mod-circulation
mod-circulation-item:
1. Create a module based on mod-spring-template
2. Create a proxy method for fetching items from mod-inventory supporting Item contract schema
3. Create liquibase scripts for database structure for CirculationItem
mod-circulation:
1. Adjust calls for item fetching from mod-inventory to mod-circulation-item using multiple interfaces

Impact Analysis

	Module	Impact	Modification Size Estimation	Risk of Regression

	Module	Impact	Modification Size Estimation	Risk of Regression
1	`mod-circulation`	Change calls for items to proxy `mod-dcb`	S	High
2	`mod-circulation-storage`	Might require changes from `mod-inventory` to `mod-dcb`	S	High
3	`mod-audit`	Might require changes from `mod-inventory` to `mod-dcb`	M	High

Concerns related to virtual items

Problem statement 1 - Creation of Virtual Item without its related data
If the idea is to create only virtual item record with limited columns as mentioned in the design then that will not be sufficient to perform the check-in/checkout process with the existing code of mod-circualtion .

Explanation:
In the existing mod-circulation code the checkin/checkout code fetches the item and then fetches the item's related entities (the entire schemas) based on its holdingsRecordId, materialTypeID, permanentLoanTypeId, and effectiveLocationId. These are required attributes in the inventory-storage item's table.
The related data of an item plays a key role. It has to be present or simulated for further steps of the check-in/checkout process otherwise the process will break.

Concerns:

If the circulation_item table does not contain the related data then how to handle the virtual item's related data? What would be the source for this related data of a virtual item?
Tight coupling of item interface with the new module API's. Will we duplicate the schemas for the items related data in our new module?
If the related item data is simulated then it will lead to hardcoding the data for holding, instance, loan etc. That means these entities would be fake and will be needed to satisfy the checkin/checkout flow.
As of now we have anlayzed use cases like checkin/checkout. But, this will impact other use cases like Renew, Cancellation, Fee/Fines.
The mod-circulation code will need more changes than expected to handle the virtual item's related data for example:
1. The new get circulation item by barcode API to get the item.
2. To get the item's related data either more API's have to be put in place in the new module mod-circulation-item, or, has to be managed in circulation module maybe by hardcoding the values.
3. The new put circulation item API will be called to update the item status and other attributes.
4. After the check-in/check-out, if we click end patron action session, we can see the following error: Failed to send notice for patron action session: referenced item was not found.
5. For the loan section in the UI, there are empty columns because UI calls circulation/loans and it requires a change in the related API.

Problem statement 2 - Virtual Item/Temporary Patron on-screen behavior