Overview

When Data Import runs it consumes a lot of resources, especially via data import modules like mod-source-record-manager (mod-srm), mod-source-record-storage (mod-srs). Additionally, mod-inventory and mod-inventory-storage also carries out Data Import's tasks of creating or updating inventory records.  Therefore, when mod-srs and mod-srm work hard, these inventory modules also work hard as well.  Subsequently, other workflows in FOLIO such as circulation's check in/out, requests, etc.. suffer when Data Import is running in the background because of physical resources starvation since they also require the inventory modules to carry out their tasks.  The solution outlined below is to create another set of mod-inventory and mod-inventory-storage just to do Data Import work and leave the original set of mod-inventory and mod-inventory-storage alone to work on other non-Data Import tasks. 

Deployment Schemes

Current Deployment

All modules communicate to each other via Okapi, denoted by solid arrows.  mod-inventory, mod-srs, and mod-srm also subscribes and publishes messages to Kafka. This is denoted by dashed arrows.

Traffics Diversion

In this traffics diversion scheme, a new set of mod-inventory and mod-inventory-storage services are created. In the diagram below, they are depicted by boxes of "mod-inventory-storage-x" and "mod-inventory-x" in blue font.

1. These additional services do not register with Okapi so no other modules in the ecosystem are aware of these services' existence, thus no messages will be sent to them.  
2. These "x-services" publishes and subscribe to Kafka DI topics for messages as usual in place of the original mod-inventory and mod-inventory-storage services, which stop pubing/subing with Kafka so that they'd exclusively take requests from other modules/Okapi.
3. mod-inventory-x now communicates with mod-inventory-storage-x directly to ask it to create or update records.

With the traffics diversion in place, check-in check-out response times improve by about 60% in Lotus and nearly 100% in Kiwi. mod-inventory-x and mod-inventory-storage-x takes on the brunt of DI traffics leaving mod-inventory and mod-inventory-storage resources to carry out check-in check-out activities.