DCB Integration

Submitted	Jun 15, 2023
Approved
Status	DONE
Impact	high
Arch Ticket	https://folio-org.atlassian.net/browse/ARCH-99
Prod ticket	https://folio-org.atlassian.net/browse/UXPROD-4325

Glossary

Term	Description

Term	Description
DCB	Direct Consortia Borrowing System
DCB Search UI	Consolidated search for items across a consortium
Item	Physical or virtual item
Patron	Patron registered in borrowing institution

Executive Summary

Integration with DCB allows support for cross-ILS circulation of items. FOLIO can participate in cross-ILS circulation in the role of requester institution, owning institution, and pickup location. DCB will act as an orchestrator for coordinating multi-system circulation transactions. The main goal from the perspective of internal integration implementation for FOLIO is to provide circulation for external items without affecting the existing local circulation flow. FOLIO is implemented as an application with microservice architecture and should reuse the same approach for integration with DCB. The current direction of FOLIO's technological stack suggests that the approach should include integration with third-party systems through edge modules and implementation of microservices should be based on a spring-way approach.

One of the main obstacles to the implementation is that FOLIO's inventory contains information regarding items and circulation of items that are out of inventory context boundaries and overlap with circulation context. This is the reason that it is a challenge to create a virtual item for the FOLIO installation so that it wouldn't leak the properties of virtual items in search results of local items. The target solution for this requires the migration of information related to circulation from inventory and moving it to the context of the circulation module.

The circulation flow inside FOLIO is implemented in a "manager/storage" approach as a RAML module and provides additional limitations because of the complexity of introducing new changes to the circulation module. Because of time constraints, the target solution might be difficult to implement. so it is proposed to create an interim solution that can be used as a basis for migration of the circulation module to spring-way in the future. The solution requires the introduction of a circulation item that would abstract the circulation information from the inventory context. The new module will be implemented as a separate circulation-item module that should serve as a proxy/adapter between the circulation module and the inventory module.

Requirements

Functional Requirements

Supported use cases described in https://folio-org.atlassian.net/browse/UXPROD-4325

Business Flow Diagram: https://miro.com/app/board/uXjVP9HDWqw=/?moveToWidget=3458764545695411304&cot=14

Scenario 1: Circulation from Lending library to Pickup Library

Scenario 1: Circulation from Lending library to Borrowing Library

Non-Functional Requirements

Modifiability: The solution should be loose-coupled and support the existing circulation flow
Performance: FOLIO edge modules should answer under <X> ms
Security: FOLIO edge modules should authorize DCB requests through the API keys mechanism
Scalability: new microservices should support a microservice architecture approach and be stateless

Constraints

API communication from DCB to FOLIO should follow a polling approach. FOLIO should provide a set of related REST APIs
Status check requests can happen with high frequency because of the polling mechanism
The integration implementation should be finished inside ECS initial release.

Risks

Possible additional UI work for different modules related to Umbrella approach.

Assumptions

DCB uses a polling approach to communicate with FOLIO. Hence FOLIO should provide REST API endpoints
DCB is able to authenticate patron. Patron identity is supplied by DCB and known to the FOLIO borrowing library.
DCB is able to provide information on the requested item to the lending library. Item information is known through RTAC and reflects up-to-date information from the inventory of FOLIO
The pickup location can be separate from the borrowing library.
In case the lending or pickup institution was changed DCB should create a new transaction
Request canceling can happen at any point in the lifetime of the borrowing/lending request
FOLIO should support borrowing, lending, and pickup role. Implementation should support the following action for requests:
- Creation of virtual patrons in lending and pickup libraries
- Creation of virtual items in borrowing and pickup libraries
- Creation/update/cancel/status for the lending request
- Creation/update/cancel/status for borrowing request
- Creation/update/cancel/status for pickup request

Solution Options

	Solution	Description	Pros	Cons	HLE	Decision

	Solution	Description	Pros	Cons	HLE	Decision
1	Migration of all of mod-circulation to new spring module	Introduction of new circulationItem that would abstract the actual item from the circulation process. Implementation requires moving properties related to the process of circulation from `mod-inventory` to the new circulationItem entity and moving the existing circulation flow to the spring module.	Context of circulation is extracted from the inventory Target technological stack is used for all circulation flow Context boundaries are well-defined and stay inside a single module	The migration of the existing RAML module to the spring-boot module would require significant development effort.
2	Introduction of virtual item/patron in a separate module	Introduction of new circulationItem that would abstract the actual item from the circulation process. Implementation requires moving properties related to the process of circulation from `mod-inventory` to the new circulationItem entity.	Context of circulation is extracted from the inventory Springboot technological stack can be used	Circulation context boundaries are extended to one additional module. Requires proxying requests from `mod-circulation` to `mod-inventory.` Might require additional HTTP request hops due to routing through OKAPI if implemented in a separate module. Concerns related to virtual items (Problem statements 1, 2 and 3).	XXL	Declined. Introduction of virtual item requires additional virtualizing of inventory entities like instance/holding etc.
3	Supporting existing API for the creation of temporary items/patrons	The implementation should use existing API and approach with the creation of temporary patrons/items.	Requires development of an edge module and little to no changes in other parts of the system. The item detail and patron detail will be intact and will be displayed appropriately on the UI.	The item created will be an actual Item in inventory. The circulation context wouldn't be extracted from inventory. Requires implicit exclusion from search results. Current suppression mechanisms have leakage and are not fully addressed in data export. Data integrity could be an issue due to microservice architecture and dependency management between modules	M-L	Declined. Usage of suppression from discovery mechanism involves side-effects in different FOLIO applications like OAI-PMH, mod-search, data-export
4	Implementation of the creation of virtual patrons and items inside of existing `mod-circulation`		Context of `mod-circulation` stays inside existing manager/storage modules related to circulation. Context of circulation is extracted from the inventory.	Due to manager/storage approach in circulation modules logic will be implemented in two modules. Due to the asynchronous approach with vert.x and RMB modules the complexity of the implementation will be significant. Concerns related to virtual items (Problem statements 1, 2 and 3).
5	Implemenation of virtual item with single instance/holding object	Only item is virtualized. Single Instance and holding is created in inventory and used for all DCB Items in lending and pickup libraries.	Little to none effort to remove leakage of virtual instances. Aligned with strategic vision but target solution (proper separation of inventory and circulation context) will require rework.	UX/UI: requires additional UI development to distinct DCB items. Might be confusing for users More dev effort compared to real instances creation, less then full virtual item/instance/holding Edge-patron: might require additional investigation for items/loans edge API’s Extended flow with Renew operations is not analyzed		In Discussion. Aligned with strategic vision

Target Architecture (Solution Option 2)

The solution consists of the following components:

edge-dcb - microservice responsible for providing REST API for DCB and API key authorization.
mod-dcb - microservice responsible for persisting state of DCB transactions and coordination of internal calls inside FOLIO. Should listen for updates from mod-circulation to reflect the state of DCB transactions to external consumers.
mod-circulation-item - should implement an adapter/proxy approach for the virtual item or patron for circulation. The adapter/proxy approach should allow the breaking of dependency between circulation and actual/virtual items. The module should be able to:
1. Create/Get/Update virtual item (circulationItem)
2. Substitute calls to mod-inventory for items and when the request is related to DCB return Virtual Item
Mod-circulation - should not require significant changes

Diagram Source: Volaris-DCB.drawio (2).xml

API Contracts

Open API Description: API Contract

Entity Relationship Diagram

The target solution introduces a new abstraction for an item in circulation context: CirculationItem. This serves as a proxy for items that are present in mod-inventory and, in case of borrowing and pickup libraries, they represent an item that exists in other institutions. DCB Transaction represents a facade object for the synchronization of different entities in FOLIO.

^@startuml

^{entity CirculationTransaction {}
^{*Circulation Transaction ID: UUID}
^--
^{Status: varchar}
^{Created: timestamp}
^{Updated: timestamp}
^}

^{entity CirculationPatron {}
^{*Patron UUID: UUID}
^--
^{Patron Group: varchar}
^{Patron Barcode: varchar}
^{Library Code: varchar}
^{Transaction ID: UUID}
^}

^{entity CirculationItem {}
^{*Item UUID: UUID}
^--
^{Instance Title: varchar}
^{Item Barcode: varchar}
^{Pickup Location: Location ID}
^{Item Material Type: enum}
^{Library Code: varchar}
^{Transaction ID: UUID}
^}

^{entity CirculationProxy {}
^{*Item UUID: UUID}
^--
^{Instance Title: varchar}
^{Item Barcode: varchar}
^{Pickup Location: Location ID}
^{Item Material Type: enum}
^{Library Code: varchar}
^{Transaction ID: UUID}
^}

^{entity CirculationDetails {}
^{*Item UUID}
^--
^Status
^Location
^}

^{entity CirculationNotes {}
^{*Item UUID}
^--
^{Note Type: [CheckIn, Checkout]}
^{Note: varchar}
^{StaffOnly: boolean}
^}

^{entity Loan {}
^{*Loan UUID: UUID}
^--
^{*Patron UUID: <<FK>> Patron}
^{*Item UUID: <<FK>> CirculationProxy}

^}

^{entity Item {}
^{*Item UUID: UUID}
^--
^{...Item Properties}
^}

^{entity Patron {}
^{*Patron UUID: UUID}
^--
^{...Patron Properties}
^}

^{entity Instance {}
^{*Instance UUID: UUID}
^--
^{...Instance Properties}
^}

^{entity Holding {}
^{*Holding UUID: UUID}
^--
^{...Holding Properties}
^}

^{CirculationTransaction ||--|| CirculationItem}
^{CirculationTransaction ||--|| CirculationPatron}
^{CirculationPatron ||--o| Patron}
^{CirculationItem ||--o| CirculationProxy}
^{CirculationProxy ||--o| Item}
^{Patron ||--|{ CirculationProxy}
^{CirculationProxy ||--|{ Loan}
^{CirculationProxy ||--|| CirculationDetails}
^{CirculationProxy ||--|{ CirculationNotes}

^{Instance ||--|{ Holding}
^{Holding ||--|{ Item}

^@enduml

Interaction Sequence Diagram

Coordination of transactions in different libraries is the responsibility of DCB. The communication mechanism is polling from DCB.

^@startuml
^autonumber

^{actor "DCB User" as lu}
^{participant "DCB Search UI" as l}
^{participant "OpenRS" as dcb}
^{box FOLIO}
^{participant "Borrowing Institution" as bor}
^{participant "Lending Institution" as lend}
^{participant "Pickup Institution" as pickup}
^{end box}

^{lu -> l++: Request Item}
^{l -> dcb ++: Request Item}
^{'==0. Prerequisites [] ==}
^{'dcb -> bor++: Create Virtual Item}
^{'return Item Created}
^{'dcb -> lend++: Create Virtual Patron}
^{'return Patron Created}
^{'dcb -> pickup++: Create Virtual Item and Patron}
^{'return Item/Patron Created}

^{==1. Create Transaction==}
^{dcb -> bor++: Create Borrowing DCB Transaction [POST /transactions/{id}]}
^{return DCB Transaction Created}
^{dcb -> lend++: Create Lending DCB Transaction [POST /transactions/{id}]}
^{return DCB Transaction Created}
^{dcb -> pickup ++: Create Pickup DCB Transaction [POST /transactions/{id}]}
^{return DCB Transaction Created}

^{return Item Request Created}
^{return Item Request In Progress}

^{==2. Coordinate Transaction==}

^{==2.1 Coordinate Borrowing==}
^{dcb -> dcb++: On timer:}
^{dcb -> bor++: Borrowing DCB Transaction Status [GET /transactions/{id}/status]}
^{return Status}
^{alt Borrowing DCB Transaction Status Changed}
^{dcb -> lend++: Update Lending DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{dcb -> pickup++: Update Pickup DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{end alt}
^{return ok}

^{==2.2 Coordinate Lending==}
^{dcb -> dcb++: On timer:}
^{dcb -> lend++: Lending DCB Transaction Status [GET /transactions/{id}/status]}
^{return Status}
^{alt Lending DCB Transaction Status Changed}
^{dcb -> bor++: Update Borrowing DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{dcb -> pickup++: Update Pickup DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{end alt}
^{return ok}

^{==2.3 Coordinate Pickup==}
^{dcb -> dcb++: On timer:}
^{dcb -> pickup++: Pickup DCB Transaction Status [GET /transactions/{id}/status]}
^{return Status}
^{alt Pickup DCB Transaction Status Changed}
^{dcb -> bor++: Update Borrowing DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{dcb -> lend++: Update Lending DCB Transaction [PUT /transactions/{id}/status]}
^{return DCB Transaction Updated}
^{end alt}
^{return ok}

^@enduml

State-Transition Diagram

Scenario: Lending Library to Pickup Library

Scenario: Lending Library to Borrowing Library

Work Breakdown Structure

edge-dcb:
1. Create a module based on edge-common-spring
2. implement API contracts and proxy calls to mod-dcb
mod-dcb:
1. Create a module based on mod-spring-template
2. Create liquibase scripts for database structure for DCBTransaction, DCBItem, and DCBPatron entities
3. Create circulation items in mod-circulation-item
4. Create temporary patron in mod-patron
5. Listen for updates on the circulation process in mod-circulation
mod-circulation-item:
1. Create a module based on mod-spring-template
2. Create a proxy method for fetching items from mod-inventory supporting Item contract schema
3. Create liquibase scripts for database structure for CirculationItem
mod-circulation:
1. Adjust calls for item fetching from mod-inventory to mod-circulation-item using multiple interfaces

Impact Analysis

	Module	Impact	Modification Size Estimation	Risk of Regression

	Module	Impact	Modification Size Estimation	Risk of Regression
1	`mod-circulation`	Change calls for items to proxy `mod-dcb`	S	High
2	`mod-circulation-storage`	Might require changes from `mod-inventory` to `mod-dcb`	S	High
3	`mod-audit`	Might require changes from `mod-inventory` to `mod-dcb`	M	High

Concerns related to virtual items

Problem statement 1 - Creation of Virtual Item without its related data
If the idea is to create only virtual item record with limited columns as mentioned in the design then that will not be sufficient to perform the check-in/checkout process with the existing code of mod-circualtion .

Explanation:
In the existing mod-circulation code the checkin/checkout code fetches the item and then fetches the item's related entities (the entire schemas) based on its holdingsRecordId, materialTypeID, permanentLoanTypeId, and effectiveLocationId. These are required attributes in the inventory-storage item's table.
The related data of an item plays a key role. It has to be present or simulated for further steps of the check-in/checkout process otherwise the process will break.

Concerns:

If the circulation_item table does not contain the related data then how to handle the virtual item's related data? What would be the source for this related data of a virtual item?
Tight coupling of item interface with the new module API's. Will we duplicate the schemas for the items related data in our new module?
If the related item data is simulated then it will lead to hardcoding the data for holding, instance, loan etc. That means these entities would be fake and will be needed to satisfy the checkin/checkout flow.
As of now we have anlayzed use cases like checkin/checkout. But, this will impact other use cases like Renew, Cancellation, Fee/Fines.
The mod-circulation code will need more changes than expected to handle the virtual item's related data for example:
1. The new get circulation item by barcode API to get the item.
2. To get the item's related data either more API's have to be put in place in the new module mod-circulation-item, or, has to be managed in circulation module maybe by hardcoding the values.
3. The new put circulation item API will be called to update the item status and other attributes.
4. After the check-in/check-out, if we click end patron action session, we can see the following error: Failed to send notice for patron action session: referenced item was not found.
5. For the loan section in the UI, there are empty columns because UI calls circulation/loans and it requires a change in the related API.

Problem statement 2 - Virtual Item/Temporary Patron on-screen behavior

Concern:

There are times when a virtual item/virtual patron can be visible on Folio UI. In this case modifications are needed to disable hyperlinks like "item detail". A few columns might not have all the details to display and might display null/blank. This will require effort to identify such scenarios and fix them in appropriate modules.

Problem statement 3 - DCB's material, location, and patron group matchup with folio, and its circulation rules

Concerns:
DCB will be the source of data for its material type, location, patron group, etc. But, how will these mappings be matched with FOLIO's material, location, and patron group, or, will they not match up and be created separately dedicated for DCB? If they will be created for DCB then where and how will they be created and maintained?

Will DCB have a dedicated circulation policy and also be equipped with a fall-back policy? Will every DCB's material type, and patron group have its own circulation rule? How will these rules be created/maintained?

Open Questions/tasks

	Question	Owner	Response

	Question	Owner	Response
1	What is Projected transaction volume?	@Tim Auger	If we define as a transaction as the initiation of a request through a terminal state like "item returned to home library" or "patron cancellation" then it's about 350k/year for MOBIUS. For other consortium it could be significantly higher (1.25M/year). If we are talking about volume of individual message pairs, multiple the number by 5x.
2	What is the frequency of status requests in polling DCB mechanism?	@Ian Ibbotson (Use this one) / @Marc Johnson
3	Provide Groovy scripts	@Ian Ibbotson (Use this one)	#!/usr/bin/env groovy // Use this file by starting up a groovy shell (groovysh) and then issuing :load ./testscript.groovy - the colon is important! import groovy.grape.Grape Grape.addResolver(name:'mvnRepository', root:'https://central.maven.org/maven2/') Grape.addResolver(name:'kint', root:'https://nexus.k-int.com/content/repositories/releases') Grape.grab(group:'org.slf4j', module:'slf4j-api', version:'1.7.25') Grape.grab(group:'net.sf.opencsv', module:'opencsv', version:'2.3') Grape.grab(group:'org.apache.httpcomponents', module:'httpclient', version:'4.5.10') Grape.grab(group:'org.codehaus.groovy.modules.http-builder', module:'http-builder', version:'0.7.1') Grape.grab(group:'org.apache.httpcomponents', module:'httpmime', version:'4.5.10') Grape.grab(group:'org.slf4j', module:'slf4j-api', version:'1.7.6') Grape.grab(group:'org.slf4j', module:'jcl-over-slf4j', version:'1.7.6') Grape.grab(group:'net.sourceforge.nekohtml', module:'nekohtml', version:'1.9.22') Grape.grab(group:'xerces', module:'xercesImpl', version:'2.11.0') Grape.grab(group:'org.ini4j', module:'ini4j', version:'0.5.4') Grape.grab(group:'io.github.http-builder-ng',module:'http-builder-ng-apache',version:'1.0.4') OkapiClient oc = new OkapiClient('snapshot') /* 20230218 10101 / // oc.getInstancePage() // oc.placeHold('ian1234', 'k-int', 'kph', 'kph-suite3', '10101') // oc.placeHold('ian1234', 'k-int', 'kph', 'kph-suite5', '80000') // oc.upsertServicePoint('PHS3'); // oc.upsertServicePoint('PHS5'); String new_instance_id = java.util.UUID.randomUUID().toString(); String instance_id_at_sgcl='1234567114@SGCL'; String item_barcode='3242353' String item_id_at_sgcl='7876499'; oc.login() def institution_data = oc.getInstitutions(); // These should all be LOOKUPORCREATE println("Create Instance"); def new_bib = oc.createInstance([ 'id': new_instance_id, 'hrid': instance_id_at_sgcl, 'title':'The TAO of DCB - a ReShareDCB Temporary Instance Record Vol V', 'instanceTypeId':'3115fc44-70f6-4ee6-b9a8-cc6bafa113b0', 'source':'FOLIO' ]) String new_item_id = java.util.UUID.randomUUID().toString(); String new_holding_id = java.util.UUID.randomUUID().toString(); println("Create holding"); def ch = oc.createHolding([ 'id': new_holding_id, 'instanceId': new_instance_id, 'permanentLocationId': '52ec2938-0aa9-43cd-b0f3-2045b10e457e', 'sourceId': 'f32d531e-df79-46b3-8932-cdd35f7a2264' ]) println("result of create holding: ${ch}") / { "barcode": "ewrtwetwert", "boundWithTitles": [], "holdingsRecordId": "5d4344cb-c321-4f20-b98f-39bacbafe01d", "materialType": { "id": "1a54b431-2e4f-452d-9cae-9cee66c9a892" }, "permanentLoanType": { "id": "2b94c631-fca9-4892-a730-03ee529ffe27" }, "status": { "name": "Available" } } */ Thread.sleep(1000); Map item = [ // 'id': new_item_id, 'hrid': "i${item_id_at_sgcl}@SGCL".toString(), 'barcode': item_barcode, 'boundWithTitles': [], 'holdingsRecordId': new_holding_id, // 'title': 'The TAO of DCB - a ReShareDCB Temporary ITEM Record Vol V', // 'itemIdentifier':"i${item_id_at_sgcl}@SGCL".toString(), 'status':[ 'name':'Available' ], 'materialType':[ id:'1a54b431-2e4f-452d-9cae-9cee66c9a892' ], 'permanentLoanType':[ 'id':'2b94c631-fca9-4892-a730-03ee529ffe27' ] ] println("Create item ${item}"); def new_item = oc.createItem(item); println("Done");
4	Post information on Chalmers temp patron capability	@Tim Auger	I really don't think Chalmers is something we want to copy. It is a different set of use cases.
5	Can you provide transaction ID when POST API creates DCB Transaction (Ian proposed to use internal DCB transaction UUID)?	@Kalibek Turgumbayev	@Ian Ibbotson (Use this one): Yes
6	Can DCB cancel current transaction in case of significant changes and start new one (with new transactionID)? Significant changes include: Change of Item UUID Change of Lending library Change of Pickup Library and/or Pickup Service point	@Kalibek Turgumbayev	@Ian Ibbotson (Use this one): Yes, we can, but this suggests a workflow where either we don't create a patron request until after we have established who will lend it, or we create churn in the FOLIO system by creating and deleting an arbitrary string of transactions - that is pretty awful UX IMO. We can absolutely use the supplier request UUID as a transaction ID and close out the transaction as we move down the rota. I woudn't design a system this way given a choice, but we will adapt to what we are given.
7	What data has DCB besides item title and UUID? Could DCB include following into request to edge-dcb (DCBTransaction in API Contract): Borrowing/Lending/Pickup Library Name and ID Borrowing/Lending/Pickup Service Points	@Kalibek Turgumbayev	@Ian Ibbotson (Use this one): DCB Knows these things but I would prefer to create first class location/service point records and point to them than have circ item level properties.
8	Provide list of material types/ patron groups that exist in DCB	@Kalibek Turgumbayev	@Ian Ibbotson (Use this one): No - that isn't how this works. FOLIO provides US with a list of material types and patron groups and then we map the central data to what a target system has so that what we place in a target system is what YOU know about - FOLIO needs to provide US with this data.
9	Confirm the API Exchange flow: Create DCB Transaction via POST API Poll status via GET API In case of changes update other participants through PUT API	@Kalibek Turgumbayev	@Ian Ibbotson (Use this one): We can work with that - the devil will be in the detail - getting it implemented ASAP Is the important thing.
10	Question from DCB to the Volaris team: Our understanding now is that we will be provided with edge APIs that represent proxies of the existing interfaces and we will NOT write the V1 implementation against the new circ interface. Therefore, the critical path for DCB is on the intermediary Facade pattern interfaces and NOT on the new circ interfaces - is that correct?	@Ian Ibbotson (Use this one)	@Kalibek Turgumbayev: Yes. The implementation of facade gives us opportunity to change internal implementation of circ interfaces independently. The initial integration between DCB and FOLIO will use a brand new API design based around `circulation transactions` DCB will not be interacting with directly items, loans, requests, material types etc No new edge APIs that proxy items, loans etc will be built