2024-5-1 Data Import Subgroup meeting

Latest as of 4/24:

• Decision was made to revert Quantity/Location 'Check' & 'Overlay' processes that were introduced in Poppy.
• This will essentially restore Order + Inventory entity Job profile functionality to the way it behaves in Orchid today.

Additional aspect for Subgroup to discuss:

• A similar Check & Overlay process that occurs today is to align Material Type on the Order with Material Type of created Items (when Items are created as part of Data Import job)
  • Current functionality: If Purchase Order Status is set to 'Open', then Material Type field is greyed out in Order mapping. If Items are created as part of Data Import job, then the Material Type values in the Items will be used to fill in the Material Type field in POL.
  • Proposed functionality: If Purchase Order Status is set to 'Open', then Material Type field will not be greyed out and will remain mappable. This will allow the Order to remain source of truth for POL details and aligns with how we will be handling Holdings & Items against Quantity/Location discrepancies.
    • NOTE: In the event that the Material Type of created Items are different from the Material Type mapped in the Order, this could trigger Mod-orders to create additional redundant Items (similar to behavior seen if Order Location & Holdings Locations mismatch). So the responsibility is on the user to ensure that Order details and created Item details match.
• Feedback from subgroup : 
  • Request for documentation on the checks performed by Data Import (why, when, how, etc.)
  • Hard to respond without knowing more about why DI was designed as it is now.
  • Ryan : we are learning together ; thank you for your patience.
  • Ryan : to dig up instructions on how to create multiple items
  • No objections to proposed functionality ; Ryan to move forward and keep everyone in the group as the work progresses. 
    • Proposed functionality centers on previous discussions around the order record being the source of truth.

Proposed next steps to address:

• Given a Data Import Job profile is run to create Orders, Instances, Holdings, & Items
• And the Item quantity within Order details differs from quantity of created Items
• Then FOLIO defers to mod-orders to create Item entities based on Order mapping, which will be linked to the Purchase Order Line (POL)

• Given a Data Import Job profile is run to create Orders, Instances, Holdings, & Items
• And the Location details within the Order differ from those of created Holdings
• Then FOLIO defers to mod-orders to create new Holdings entities based on Order mapping, which will be linked to the Purchase Order Line (POL)

If we move forward with the above, please consider the following details when using Data Import to create Orders with Inventory entities:

• If locations & quantities differ between Order details and the Holdings & Items created by Data Import, then mod-orders will be triggered to create new Holdings & Items that do match the locations & quantities described in the Order. 
• This scenario would lead to creation of erroneous/redundant Holdings or Items that are not linked to the POL of the Order.

Meeting notes from 4/3/2024 - 

Ryan Taylor shared a MIRO diagram for processing Orders via Data Import. 

Q: How are multiple items created as a part of the current process? Ryan - it is dictated by the inventory import profiles. 

Q: What is the end scenario for when there is a job profile that is set to only create an instance and holdings. The end state would be an additional holdings and item would be created because the quantity does not match the number of items. If items are not defined should not be getting an unnecessary item created, but suspicious that an additional holdings would be created. It is thought that if the location matches what is in the profile that a second holdings / item will not be created. Need clarification on whether there will be additional holdings / items created. 

Q: Why does the process not respect the quantity in the order import profile? Why can the order quantity and the location quantity is not be the same? May need to look at this from a different perspective. 

It was noted that the business logic requires the connection of the order to Inventory at the point of creation, so it makes sense for the order to "have control" of the processing. 

It was also noted that the only difference between creating an order in pending vs open is that the mapping of the inventory records should come from the field mapping profiles rather than the defaults in the order import app. 

The Orders (ol and pol) have a certain business logic where the quantity is linked to what is in Inventory in addition to Receiving and Finance apps. Orders should be the source of truth as this would allow for this.

Ryan has some questions to bring back to the dev team. How can we make it simpler? Can you start in Inventory and then create the Order and associate the inventory with the POL?

• 3/27 - Review related scenario raised by Christie in Slack
  Christie presented her DI job profile in her local environment in Poppy CSP 2. The profile worked as expected for single volume monograph orders. This profile creates an order and the quantity physical is being mapped from 980$q, an instance with "uncataloged" status, a holdings, and an item. When there is more than one quantity physical is greater than 1, then in the POL that quantity physical is 1 rather than the number from the mapping in the 984$q.  We can write this up as a separate bug and let the developers determine if it is part of MODDATAIMP-1010 or a separate bug. The fact that only 1 item is being creating needs to be investigated even when the quantity has more than 1. This is different and related to MODDATAIMP-1010. This could be related to an issue in CSP2 where if conditional mapping was used, the number of holdings and items created was incorrect. Could it also be an issue where there is only 1 980 so that there is just 1 holdings and items? Could it be looking for the presence for multiple 980s? This could make sense in regular bib imports but not creating orders. What happens if there is no conditional in the mapping for quantity? Christie will try this out. When there is no conditional, the quantity is still one. When you remove the create items and holdings, then the quantity is what is being mapped. The location maps correctly from the mapping in orders to holdings record.

What happened and how do we move forward? What is the underlying issue? This is a complex scenario where we need to drill into an architecture level. Can we document what we have that can be understood? Does it make sense for real live scenarios? In the immediate time frame, the goal is to make sure we don't get quantity zero. If it is thread and if it is bigger issue, do we have short term workarounds? At the moment, not sure how this will be addressed as there seems to a divide as to how this works. How large of a task is it to investigate and avoid issues that we're seeing for the short term?

We need to talk to Dennis because location and cost quantity are tied together. What would happened if this was severed? This makes sense.

There's a bigger issue that needs to be unpacked. We have the immediacy of continuing to do our work. We need an understanding of what we expect.

If we order 10 copies of something, then the quantity should say 10.

For an order to be open, then you have to create inventory. For an order to be in the pending state, then the Order App does the creation of the inventory records.

In discussing this issue with the Folijet team, I've learned that the described behavior is a result of requirements to help avoid Item duplicates in support of the Multiples enhancements found in Poppy. 

Would the following logic/scenario make sense as a possible path forward?

• If Job profile contains 'Create' Action profiles for Orders, Instance, and/or Holdings, then POL Quantity value should be controlled by mapping.
• If Job profile contains 'Create' Action profiles for Orders, Items, and/or Instance, and/or Holdings , then POL Quantity value should be controlled by the number of Items created.

Discussion :

• Based on previous conversation, assumption is that the first bullet is the ideal logic.
• Questions : what scenario is requiring this complication?  Why wouldn't the quantity in the order always match the quantity in the ingest file?
• Reasoning for current situation is unclear.
• Cost and quantity in an ingest file are related.  Using the # of items instead of the values in the incoming file breaks the logic and expectation of a user.
  • Standing orders are a good example : one (quantity = 1) set for $X dollars instead of $X dollars per item
  • Orders can be for a set or a part; practice varies by library & vendor
  • Controlling the quantity by the mapping leaves these values up to the library ; maximum flexibility
• Confounding variable could be that the quantity ordered must match the quantity by location.
  •  Recommendation to talk to Dennis (PO of Acquisitions) for clarity, further information.
• Question from Ryan : should the number of holdings records created have an impact on quantity?
  • The quantity in the POL should always come from the mapping provided by a library.
  • There is a situation where two items could be ordered, destined for different locations corresponding to multiple holdings records, but both locations aren't known during the time of order.   
  • Locations are often assigned as part of the cataloging process, not the acquisitions process.
  • The order record is the source of truth.  

• Was this behaving differently in Orchid?
• Behavior seen in this bug is that if Create profiles for Holdings or Items are included in the Job profile, then Quantity value is controlled by the number of Items created. So if Holdings profile is included, but not Items, then Quantity will result as 0.
  • Does this make sense to you that Quantity should be controlled by Items created as part of job or should it always be controlled by the Order mapping?

Consensus seems to be that FOLIO should not be de-duplicating within the incoming record unless it is explicitly defined in an import profile.

Q: Is de-duplication something that should be able to be deactivated on a field by field basis? R: Sounds like a reasonable approach. There is also some concern that this would complicate an already complicated situation. 

Possible solution - a tool to deduplicate in another tool rather than within data import instead.

Suggestion to start with the functionality audit. RT can connect with the developers as a part of this audit. 

Q: Are we starting with how we as users expect functionality work or with how the developers expect it to work. R: Really should have both for each feature. Start from perceived / desired functionality of the users and add to it with designed functionality.  Suggestion to provide examples to the developers so that it is clear what we are expecting.

Pilot functionality audit with de-duplication and start with our understanding and then get input from the developers.