Update Inventory records based on POL/VRN matches

UXPROD-3345 - Getting issue details... STATUS

Problem(s):

Vendors often supply cataloging and shelfready information to libraries in MARC bibliographic records at point of shipment or invoicing. That data may be used to update brief Instances, Holdings, and Items created at point of order. The vendor does not have HRID or UUID information for any of the Inventory records. Without that, the best matchpoint is the FOLIO Purchase Order Line number (POL) or the Vendor Reference number (VRN).

Use Cases & Requirements:

Requirement	Status	Use cases
Allow users to update SRS MARC Bibs, instances, holdings, and items based on a POL match.	RESOLVED	Create an SRS MARC Bib for an instance that was previously Source = FOLIO Modify the SRS MARC Bib for an instance that is Source = MARC Update Instance and/or SRS MARC Bib from brief to full cataloging Update holdings with permanent location, call number, URL (shelfready) Update items with barcodes, permanent loan type, In process or Available status (shelfready)
Allow users to update SRS MARC Bibs, instances, holdings, and items based on a VRN match.	RESOLVED	Same as above, except used in cases when there is no FOLIO POL available for matching, e.g. approval plan purchases, DDA purchases

Sample MARC file with POL and VRN data

Firm order (FOLIO POL and Vendor Reference Number)

Note: POL and VRN can be in various MARC fields, usually 9xx, but sometimes 024; below are examples, not all possibilities

LDR 02524cam a2200493 i 4500
001 on1141040024
003 OCoLC
005 20210610144226.0
008 200206t20202020paua\\\\\b\\\s001\0\eng\\
010 \\$a 2020001402
040 \\$aPSt/DLC$beng$erda$cDLC$dYDX$dOCLCO$dOCLCF$dBDX$dPIT$dUKMGB$dYDX$dUPM$dYUS$dFNE$dGYG
019 \\$a1229937418
020 \\$a9780271086040$q(hardcover)
020 \\$a0271086041$q(hardcover)
024 8\$a90030228025
035 \\$a(OCoLC)1141040024$z(OCoLC)1229937418
042 \\$apcc
050 00$aND237.J76$bV36 2020
082 00$a759.13$223
100 1\$aVanDiver, Rebecca,$d1982-$eauthor.
245 10$aDesigning a new tradition :$bLoïs Mailou Jones and the aesthetics of Blackness /$cRebecca VanDiver.
264 \1$aUniversity Park, Pennsylvania :$bThe Pennsylvania State University Press,$c[2020]
264 \4$c©2020
300 \\$axii, 241 pages :$billustrations (chiefly color) ;$c27 cm
336 \\$atext$btxt$2rdacontent
336 \\$astill image$bsti$2rdacontent
337 \\$aunmediated$bn$2rdamedia
338 \\$avolume$bnc$2rdacarrier
386 \\$mGender group:$ngdr$aWomen$2lcdgt
386 \\$mNationality/regional group:$nnat$aAmericans$2lcdgt
386 \\$mOccupational/field of activity group:$nocc$aUniversity and college faculty members$2lcdgt
504 \\$aIncludes bibliographical references (pages 213-223) and index.
505 00$tIntroduction : claiming middle ground --$tSeeking success : school, society, and career aspirations --$tRoutes to roots : from Black Washington to Black Paris --$tDiasporic directions : Haiti, collage, and composite aesthetics --$tIn and out : Africa and the academy --$tConclusion : composite naming practices and art histories.
520 \\$a"A critical analysis of the art and career of African American painter Loïs Mailou Jones (1905-1998). Examines Jones's engagement with African and Afrodiasporic themes as well as the challenges she faced as a black woman artist"--$cProvided by publisher.
600 10$aJones, Lois Mailou$xCriticism and interpretation.
600 17$aJones, Lois Mailou.$2fast$0(OCoLC)fst00051276
650 \0$aAfrican diaspora in art.
650 \7$aAfrican diaspora in art.$2fast$0(OCoLC)fst01902318
655 \7$aCriticism, interpretation, etc.$2fast$0(OCoLC)fst01411635
924 \\$a90030228025
935 \\$a12345-1
949 \1$aND237.J76$bV36 2020$fART$g1$i00053505045$laada$s-$t0$u-$z05000
980 \\$a210622$b6613$d000$e5423$f928288
981 \\$bMHUMAAA$caa

Approval Plan Automatic Purchase (Vendor Reference Number only)

LDR 01690cam a2200373 i 4500
001 1235903375
003 OCoLC
005 20210826141608.5
008 210127s2021\\\\ctu\\\\\\\\\\\000\p\eng\\
010 \\$a 2021003582
040 \\$aDLC$beng$erda$cDLC$dOCLCO$dBDX$dYDX$dOCLCF$dUKMGB$dYDX
020 \\$a9780819580436$qhardcover
020 \\$a0819580430$qhardcover
020 \\$a9780819580429$qtrade paperback
020 \\$a0819580422$qtrade paperback
020 \\$z9780819580443$qelectronic book
035 \\$a(OCoLC)1235903375
042 \\$apcc
050 00$aPS3602.R34288$bB58 2021
082 00$a811/.6$223
100 1\$aBrady, Andrea,$d1974-$eauthor.
245 14$aThe blue split compartments /$cAndrea Brady.
264 \1$aMiddletown, Connecticut :$bWesleyan University Press,$c[2021]
300 \\$a92 pages ;$c23 cm.
336 \\$atext$btxt$2rdacontent
337 \\$aunmediated$bn$2rdamedia
338 \\$avolume$bnc$2rdacarrier
490 1\$aWesleyan poetry
520 \\$a"Combining the history of drone warfare and the consequences of "everywhere war" with personal memory and reflections on the myths and mechanics of prosthetic violence, voyeurism, masculinity, and desire, these poems put their operator in the heads-up display to imagine what happens when targets look back"--$cProvided by publisher.
655 \7$aPoetry.$2fast$0(OCoLC)fst01423828
655 \7$aPoetry.$2lcgft
776 08$iOnline version:$aBrady, Andrea, 1974-$tThe blue split compartments$bFirst.$dMiddletown : Wesleyan University Press, 2021.$z9780819580443$w(DLC) 2021003583
830 \0$aWesleyan poetry.
980 \\$a40030682796$bPOE2$cGen$dJRL$eYBP$g509509$h994720$i210830$j14.36$lPaper$m15.95$q1

Links

Match profile updates for Inventory records
Match profile data elements for Inventory records (add POL/VRN, plus do some housekeeping on existing match data elements that are not used?)
- See the ones in red - should we keep them?
- Decision: Leave them for now and fix as they are tested, if they do not work as expected

SME Questions:

Question	Status	Conclusion	Comments
Can we limit to 0xx and 9xx fields?	RESOLVED	Probably OK	Devs need to confirm if MARC-MARC matching capabilities can be expanded, e.g. 924$a to 924$a
Any different considerations if the match-from and/or match-to field is repeatable?	RESOLVED	So long as matches to single instance, holdings, or item, should be able to update. Recommend that libraries use 9xxs for POLs and VRNs, instead of 024s and 035s, since less likely that there will be other kinds of numbers in specified 9xx fields, but likely to be other kinds of numbers in 024s/035s	From: If copies ordered at the same time, but on separate POLs (for different acq units or locations) To: Instance linked to multiple orders, Holdings linked to multiple orders Is an item ever linked to multiple orders? A-M asking on Acq channel Needs to try each of the froms (if multiples) Needs to try each of the tos (if multiples)
Confirm appropriate test cases for E-to-E automated tests	RESOLVED		See use cases above Test 1: POL, Instance source = FOLIO, update all record types Test 2: POL, Instance source = MARC, update all record types Test 3: VRN, Instance source = FOLIO, update all record types Test 4: VRN, Instance source = MARC, update all record types Add negative test cases - no matches or multiple matches
Do we need to break out vendor ref number types, like we do for Identifiers?	RESOLVED		For now, do not break out the different vendor ref number types; if use case arises in the future, consider breaking out, similar to how the Instance identifiers are broken out
Can we remove some unused match data elements?	RESOLVED		Leave for now; as users test more of the match elements in the future, correct or amend on a case-by-case basis, Besides, that enlarges the scope of this feature
Match on POLs with which statuses?	RESOLVED	Per Christie, would want to be able to choose the POL status for successful matches Per Raegan, this is not a scenario that comes up for them	Open = OK to match Pending = DO NOT MATCH Closed = DO NOT MATCH (no use case identified yet) If multiple are matched (like the same VRN in Law and Main order, leading to the same Instance, but different holdings and items), stop if multiple matches. Or maybe use location as a submatch to get to the right holdings/item Christie: would be rare to want to match based on a closed PO, especially if it has cataloging implications and could result in accidental overlay of previous cataloging. Same for Jennifer E, Leeda, Dung-Lan A-M to talk with Devs on Friday - would it be more complexity to take status into account when matching? a little more, but infrastructure already there Maybe have 4 matching options POL with status = Open POL with status = Open or Closed VRN with status = Open VRN with status = Open or Closed
What about multiple copies?	RESOLVED	For now, we can only do 1 instance/holdings/item update until the multiples feature is developed UXPROD-2741 - Getting issue details... STATUS	Right now, can only create single instance, holdings, item from an incoming MARC Bib. What happens if you're trying to updating multiple holdings or items from the same MARC Bib (if you have multiple holdings/items HRIDs) (find the feature for updating multiples from the same MARC Bib) Write a couple tests to see what happens when trying to update more than 1 item or holdings from the same MARC Bib

Basic workflow

Matching on POL (only accounts for single Instance/Holdings/Item for now)

Data Import job profile:
1. Match POL to Instance UUID
  1. Single match: Update Instance (and Create/Update SRS MARC Bib)
  2. No match or Multiple match: STOP
2. Match POL to Holdings UUID
  1. Single match: Update Holdings
  2. No match or Multiple match: STOP
3. Match POL to Item UUID
  1. Single match: Update Item
  2. No match or Multiple match: STOP
mod-inventory hits mod-orders endpoint to find a POL's related Inventory UUIDs (only for Open, or possibly Open/Closed, not Pending POLs)
If found, follow related Action and Field mapping profiles for SRS MARC Bib create/update and Inventory record updates
1. Questions for SMEs:
  1. would the workflow sometimes be finding the instance and then creating (instead of updating) the holdings and item?
    1. Jenn C - for orders from previous system, creating items during receipt, but that's a time-bounded situation (migration)
    2. Similar for Dung-lan
    3. Leeda - we need to test to see if system can create a holdings and item on the correct bib, if it doesn't find a match on POL - UUID.
  2. would the workflow sometimes be finding the instance and holdings and then creating (instead of updating) the item?

Matching on VRN (only accounts for single Instance/Holdings/Item for now)

Data Import job profile:
1. Match VRN to Instance UUID
  1. Single match: Update Instance (and Create/Update SRS MARC Bib)
  2. No match or Multiple match: STOP
2. Match VRN to Holdings UUID
  1. Single match: Update Holdings
  2. No match or Multiple match: STOP
3. Match VRN to Item UUID
  1. Single match: Update Item
  2. No match or Multiple match: STOP
mod-inventory queries mod-orders by VRN to find related POL (only for Open, not Pending or Closed POLs)
1. If only 1 POL, keep going
2. If multiple POLs, STOP
mod-inventory hits mod-orders endpoint to find a POL's related Inventory UUIDs
If found, follow related Action and Field mapping profiles for SRS MARC Bib create/update and Inventory record updates

What Backend stories are needed?

Thunderjet:
- Create API endpoint that would allow lookup by POL for related Instances, Holdings, Items
- Already have a query for using VRNs to find POLs
- No additional indexes needed, since there are already searches and indexes for POLs and VRNs, which probably will be fast enough
Folijet:
- Spike to review the feature and create BE stories: MODDATAIMP-399
- Create match queries and matching logic for POLs and VRNs
  - Logic: MODDATAIMP-574
- Will need cross-index query if POL/VRN matches need to be limited to specific statuses (e.g. Open, but not Pending or Closed)
- Add stories for happy path and exception Karate tests
Prokopovych:
- No dev work required
Other
- Have PTF help with performance check once dev work is completed
- Analyze MARC-MARC matching for 0xx and 9xx fields (based on the MARC query work previously done)
  - May not be needed if can use MARC-Instance matching to create/update SRS MARC Bib

What UI stories are needed?

Folijet:
- Update UI matching screen to include options for POL and VRNs for each Inventory record type
  - Instance: UIDATIMP-1046
  - Holdings: UIDATIMP-1047
  - Item: UIDATIMP-1048
- Add stories for happy path and exception E-to-E tests: UIDATIMP-1050

Automated Testing Scenarios (for API Karate and UI E-to-E)

Scenario 1: Match on POL and update Instance (source = FOLIO), Holdings, Item (fill in some details about necessary profiles, and supply test data)

Scenario 2: Match on POL, but no match

Scenario 3: Match on POL, but multiple match

Scenario 4: Match on VRN, and update Instance (source = MARC), Holdings, Item

Scenario 5: Match on VRN, but no match

Scenario 6: Match on VRN, but multiple match

Developer Questions:

Question	Status	Conclusion	Comments
What changes are needed to the UI matching screen?	RESOLVED	Add POL and VRN options for each Inventory record type
Any different considerations if the match-from and/or match-to field is repeatable?	RESOLVED	For MARC-to-Inventory matching, not an issue as long as incoming data is in a unique field/Ind1/2/Subfield. Recommend that libraries use 9xxs for POLs and VRNs, instead of 024s and 035s, since less likely that there will be other kinds of numbers in specified 9xx fields, but likely to be other kinds of numbers in 024s/035s	For MARC-to-MARC, still TBD, after spike. Probably not until Morning Glory, but prep for it in Lotus
Where do we pull the POL and VRN from?	IN PROCESS	Pull POL or VRN from the MARC field/subfield designated in the match profile	Need to review the work that was done on MARC Search, to see if the match data can be pulled from 024, 035, and any 9xx field. Kate: add a spike to review the MARC Search. Cannot commit to this for Lotus. For MARC-to-Inventory record match, best if the value in the incoming record is in a unique MARC field (or unique indicators at least), and that field is not used for any other data in the incoming file. (e.g. if POL is in 980$a, have no other 980$a's in the incoming record)
If we only wanted to match on POLs or VRNs for orders that are Open, but NOT Pending or Closed, would that be an issues? (we already use similar logic for EDIFACT invoice matching between Invoice line and POL/VRN)	IN PROCESS	No, not an issue, but confirm with Thunderjet exactly what story/work is needed from them	POs with Ongoing status (in receipt or payment) are included with Open Order status is at the PO level, but POL/VRN are at the POL level. May need to hit a composite endpoint to get the number and the status. Per Siarhei H, can use cross-index query to also find the status of a POL Kate to confirm with Thunderjet: Need to clarify and finalize the story for the consolidated API How to get from a VRN to its associated POL How to ensure that we only retrieve Inventory info for Open POLs
E-to-E automated tests; how many, and happy path only, or negative too?	IN PROCESS	See draft UI test Jira FAT-1107 See draft Karate test Jira: FAT-1105 A-M to clean up the Jiras and add sample data and profiles	For the 4 happy path scenarios Possible negative scenarios No match (instance, holdings, item) Multiple matches (holdings, item; any need for instance?)