Create orders by importing MARC Bibliographic Records

Owned by Ann-Marie Breaux (Deactivated)
Last updated: Oct 12, 2022

Data Import Initial Development Jira:  UXPROD-185 - Getting issue details... STATUS  (Nolana/Orchid, Folijet)

Data Import Refinement Jira: UXPROD-3814 - Getting issue details... STATUS (Scope and release TBD)

Overview slide deck: Click here

Technical/Developer details: Support importing orders from MARC files via Data Import

Rough draft of requirements and focus group meeting notes: Using MARC Bibs and Data Import to create FOLIO POLs

Existing "Lehigh" workaround: Temporary Solution for Creating FOLIO Orders from MARC Data - draft

Problem(s):

  • Some vendors send order-related data in MARC records, so that the receiving library can create orders in their local system from that data. The order data may be sent ay two points in the order workflow
    • Point of order
      • For orders placed on the vendor's ordering system, e.g. firm orders. For these, the vendor does not have a FOLIO POL, since the order is placed in the vendor's system before the order is created in FOLIO (question: would FOLIO need a way for the vendor to send back a POL and have it populate the FOLIO PO/POL fields? Dev preference is no, at least for first iteration, since it would add complexity. Note that this workflow is also covered by the FOLIO Order API process, which is real-time and more efficient than this workflow, at least for creating orders.
      • For selections made in the vendor's ordering system, but not yet ordered. For these pre-order records, the vendor expects to receive back a finalized order from FOLIO, preferably as EDIFACT.
    • Point of receipt/invoicing
      • For materials invoiced automatically by the vendor and purchased by the library. In this flow, orders may only be created if the library wants to track the materials as individual titles and/or pay for the materials as individual titles.
      • Most common materials supplied via this workflow:
        • Approval plan automatic purchases (Physical and/or E) where titles are automatically supplied based on a profile that the library sets up with the vendor. Profiles may be based on subject language, publisher, or combinations of several characteristics.
        • Demand Driven or Patron Driven Acquisitions (Mainly E) where titles are automatically invoiced based on patron usage passing a trigger point that causes titles to be purchased automatically by the library
      • In this flow, there is no opportunity for FOLIO to return a POL to the vendor before invoicing, so the invoice match to the POL much rely on a Vendor Reference Number

Focus Group for drafting requirements:

  • DI PO: Ann-Marie Breaux
  • Acq PO: Dennis Bridges
  • Cornell: Jenn Colt
  • 5C: Jennifer Eustis, Sara Colglazier
  • Michigan State: Kim Wiljanen, Lisa Smith

Use Cases & Requirements: 

Requirement

Status

Notes

Allow users create orders and order lines based on order data supplied in MARC bibliographic records 

CLOSED


Allow users to create field mapping profiles, action profiles, and job profiles to support MARC-based order data

CLOSED


Allow users to create, but not update, POs/POLs via DI

CLOSED

Confirmed by Acq SIG SMEs. No need to consider updating unless significant use case identified in the future
Data import match profiles not needed for Order records

CLOSED

Confirmed by Acq SIG SMEs. Since DI is only creating orders, use case for matching and updating an order does not apply. Use case for matching and then not creating an order: not common, and could only use Product ID or Vendor Ref ID for matching. Revisit if additional use cases identified in the future
Use for one-time orders (physical, electronic, or both). Do not use for ongoing or package orders

CLOSED

Confirmed by Acq SIG SMEs. Revisit if any significant use case identified for Ongoing and/or Package orders
Show all the PO/POL fields in the Field mapping profile, but suppress the PO/POL fields and values that are only used for ongoing or package orders.

CLOSED

Confirmed by Acq SIG SMEs. 
Allow POs to be created as Pending or Open

CLOSED

Confirmed by Acq SIG SMEs.  Use cases for both identified. Will gather that info via a field at the top of the field mapping profile
Allow override to the tenant's default POL limit setting

CLOSED

Confirmed by Acq SIG SMEs. Also, value may need to vary from one vendor to another or from one type of purchases to another. Will gather that info via a field at the top of the field mapping profile
User should have an option to auto-receive

DEFERRED

Confirmed by Acq SIG SMEs. Will gather that info via a field at the top of the field mapping profile. If PO is created as Pending, and then opened manually, can this requirement be included? Or only possible if the PO is created as Open?
Add mapping syntax for TODAY + x days

DEFERRED

Would be useful for fields like Activation date, Receipt date - but per DI SMEs, not important for now. Consider for refinement feature
Honor the Acquisitions unit settings

CLOSED

Confirmed by Acq SIG SMEs. If import user invokes a job profile that creates POs, that user must be a member of the acq unit assigned by the field mapping profile, or the acq unit settings must allow non-members to create POs. This acq unit logic is already in place for Invoices.
Have a way to create POLs for multiple copies with different locations

CLOSED

Confirmed by Acq SIG SMEs. talk with devs about whether the data needs to be configured in a certain way in the MARC file, e.g. 
  • Maybe something like a repeatable 9xx field that has 
  • $a location $b quantity $c fund $d expense class (or $c fund:expense class)
Have a way to create a POL that assigns more than 1 fund and/or expense class to cover the cost

CLOSED

Confirmed by Acq SIG SMEs. See above. Include a way to assign the fund/expense class by % or amount.
Be able to accept fund and expense class whether they are supplied in separate MARC subfields, or in 1 MARC subfield, separated by a colon

CLOSED

Confirmed by Acq SIG SMEs. For example (Fund code = History, Expense class = Electronic)

  • 980 $a History $b Electronic
  • 980 $a History:Electronic
  • Combined fund:expense class DEFERRED since no library is using yet; aim to include in refinement feature
If Orders app is creating Inventory records based on instruction from Data Import (or based on opening orders), follow the tenant's setting for Disable instance matching

CLOSED

Confirmed by Acq SIG SMEs
Instances/Holdings/Items related to the created POL should display proper links in the Acquisition and Receiving history accordions, regardless of whether the POL or the Inventory records were created first

CLOSED

Confirmed by Acq SIG SMEs. Works now when order is opened and Inventory is created by the orders app. Does not work if Inventory records are associated with the POL after it has been opened. Needs further discussion with devs
If a library wants to import a MARC Bib file to create orders, but does NOT want to create any Inventory records, will there be any problem?

DEFERRED

May just require testing of a job profile with Create orders action, but no Create Instance action.
If a library has an Order setting that POs must be approved before opening, will that setting always apply to orders created by Data Import? If yes, need an option to always create the orders as approved, or always create as pending?

CLOSED

Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M asked on Acq Slack channel. If field mapping has approved selected, the user importing the file must have permission to approve orders.
If a library has an Order setting that allows for the FOLIO PO to be edited, will that setting apply to orders created by Data Import?

CLOSED

Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M asked on Acq Slack channel. Does not seem to be needed currently
Will libraries want to map PO numbers from the incoming MARC records?

CLOSED

Discussed in Acq SIG 28 June 2022.  No use case identified (yet) - wait and see if any identified before definite "No", since this decision would require coordination with Settings for editing PO numbers and max POLs per PO. If PO needs to not be mappable, easiest to disallow mapping the field completely on the field mapping screen (like HRID). A-M to ask on Acq Slack channel.
If the PO number is mapped from a MARC field (e.g. 950$p) and the PO varies from one MARC Bib to another, what should FOLIO do?

CLOSED

Wait until above question is resolved. Will not be mapped, so does not need to be dealt with

Discussed, but not requirements

  • Create separate orders per invoice. FOLIO does not currently have the logic to parse MARC files into separate invoices. If this is required, library should work with their materials vendor to get separate MARC files per invoice, and then ensure that the POL line limit in the field matting profile is set larger than the maximum number of lines allowed per invoice from that vendor.
  • Be able to select an order template within the Order field mapping profile:  The Data Import field mapping profile basically acts as a template, gathering default and mapped data for a particular type of file from a particular vendor. Once the field mapping profile is created, it does not have to be adjusted to accommodate each individual import for the same type of file. To tie the field mapping profile to the order templates would require significant development effort, along with problem handling if the field mapping profile and template have different values for the same field.
  • Create invoices from the MARC bib data. Separate feature for that. Cannot be accommodated in the scope of Nolana. For now, create orders based on MARC Bibs, and then create invoices from EDFACT data, via a separate job profile. 
  • Assign order notes to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
  • Assign tags to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
  • Schedule the pickup and import of MARC Bib files, assigning the correct job profile. Not a requirement in Nolana. Added some details into UXPROD-662

Mockup for FOLIO PO/POL Field Mapping Screen

See requirement details in the screens attached to UIDATIMP-294

Sample MARC records with Order data

No FOLIO POL

LDR  01826cam  2200493 i 4500
001  1104214104
003  OCoLC
005  20191218144549.9
008  190426s2020\\\\nyu\\\\\\b\\\\001\0\eng\\
010  \\$a  2019020718
040  \\$aDLC$beng$erda$cDLC$dOCLCO$dBDX$dOCLCF$dYDX
019  \\$a1104218545
020  \\$a9780190068479$qhardcover
020  \\$a0190068477$qhardcover
020  \\$z9780190068509$qelectronic book
020  \\$z9780190068486$quelectronic book
020  \\$z9780190068493$qelectronic publication
035  \\$a(OCoLC)1104214104$z(OCoLC)1104218545
042  \\$apcc
049  \\$aAUMM
050  00$aML3800$b.B75 2020
082  00$a781.1/7$223
100  1\$aBonds, Mark Evan,$eauthor.
245  14$aThe Beethoven syndrome :$bhearing music as autobiography /$cMark Evan Bonds.
264  \1$aNew York, NY :$bOxford University Press,$c[2020]
300  \\$axii, 325 pages ;$c25 cm
336  \\$atext$btxt$2rdacontent
337  \\$aunmediated$bn$2rdamedia
338  \\$avolume$bnc$2rdacarrier
504  \\$aIncludes bibliographical references and index.
600  10$aBeethoven, Ludwig van,$d1770-1827$xAppreciation$xHistory.
600  17$aBeethoven, Ludwig van,$d1770-1827.$2fast$0(OCoLC)fst00042803
650  \0$aMusic$xPhilosophy and aesthetics$xHistory.
650  \0$aExpression (Philosophy)$xHistory.
650  \7$aArt appreciation.$2fast$0(OCoLC)fst00815447
650  \7$aExpression (Philosophy)$2fast$0(OCoLC)fst00918886
650  \7$aMusic$xPhilosophy and aesthetics.$2fast$0(OCoLC)fst01030408
655  \7$aHistory.$2fast$0(OCoLC)fst01411628
949  \1$aML3800$b.B75 2020$e0$gOO$i312066040927765$lKU/CC/DI/M ($a & $b = call number, $e = misc library code, $g = misc library code, $i = barcode number, $l = location) 
980  \\$a200109$b35.00$dMUSIAUAM$e28.88$f40029640970$g1 ($a = invoice date, $b = list price, $d = fund code, $e = net price, $f = vendor reference number (VRN), $g = quantity)

SME Questions:

Question

Status

Conclusion

Comments

If order is created as Open, 1) will the receiving pieces be created and 2) will they show in the Inventory holdings record?

CLOSED

For initial implementation, receiving pieces, holdings, and items will be created by the order being opened

Developer Questions:

Opening orders

How should Data Import handle the opening of orders:

  • Use the Inventory interaction setting from the POL
  • Use the Data import job profile matching and sequencing
  • Allow the user to choose (at the profile level or POL level?)

Job needs to create

  • SRS MARC Bib
  • Source = MARC Instance
  • Holdings (as many as there are locations)
    • If created by Order app, can create multi, with POL link and with limited details
    • If created by DI app, can create single (for now), multi in the future, but with more details
      • Could maybe be added by a secondary Update job profile
  • Item (as many as there are copies)
    • If created by Order app, can create multi, with POL link and with limited details
    • If created by DI app, can create single (for now), multi in the future, but with more details
      • Could maybe be added by a secondary Update job profile

Possible Nolana Job profile sequence (Data Import) 

See video here

  • Create Instance (source = MARC)
  • Create Orders (with data from MARC Bib, Order data in MARC Bib, Order field mapping profile)
    • DI would need to pass the Instance UUID as part of the Create orders info
    • What happens if order is created as Pending?
    • How to create source = MARC
  • Open the order
    • Which creates Holdings and Item based on info the POL knows, and provides the POL link in the Holdings and Item
    • Question about whether opening the PO creates a new holdings or links to an existing holdings in the Instance

Second Nolana Data Import job

  • Match on VRN (to find newly-created Holdings and Item)
  • Action: Update holdings - field mapping profile; only worry about 1 holdings for now
  • Action: Update item - field mapping profile; only worry about holdings for now

Caveats:

  • For Nolana, have to run 2 import jobs
  • VRN must be in the original MARC Bib file
  • Could we run it as 1 job with update actions after the PO is opened?

There are pros and cons to Orders app and DI app creating Holdings and Items

Per Olamide, keep as much business logic in the other apps (Inventory, Orders, Invoices) as possible, instead of Data Import having to understand and replicate that logic

Background:

  • Preference is to use the Orders engine to create the Inventory records
  • Orders can be created as Pending or Open via Data Import
  • Changing an Order’s status from Pending to Open triggers:
    • Creation of encumbrances (if fund and price supplied in POL)
    • Creation of receiving pieces (depending on receiving setting in POL)
    • Creation of Instance, Holdings, Items (depending on Inventory setting in POL)
  • Currently orders feed all data to Inventory to create Instance, Holdings, Item
  • The Order app’s Inventory matching is not as flexible as Data Import’s and is all-or-nothing, depending on the setting
  • POLs are only linked with Instances, Holdings, Items if those Inventory records are created by opening orders
    • Can those POL links be created/changed manually on Inventory records
  • An Inventory record cannot be matched/updated via POL/VRN unless the Inventory record is linked to the POL; this is a very important workflow for libraries
  • The data on an incoming MARC Bib record and in a data import profile can create richer Inventory records than the Orders app can
  • Data import can only create a single holdings and item via an imported MARC Bib at this time, though that will change in Orchid or Poppy; Orders app can create multiple holdings and items


Question

Status

Conclusion

Comments

How to handle the POL limit variation (in the field mapping profile, users can select to use the existing default or override to a number higher or lower than the default)

CLOSED

See logic in field mapping profile - display the existing POL limit setting, but allow override of it
If DI only allows creation of one-time, non-package orders, will there be any problems if we remove all the non-one-time, non-package data elements from the field mapping screen?

CLOSED

Not a problem
We want to use the existing Orders logic as much as possible, but we also want Instance/Holdings/Item linked with correct POL even when created before the POL.

CLOSED

For initial release, create the Instance first via DI, then have the order's opening create the holdings and items. Then update the holdings and item via a second DI job. In the refinement release, aim to combine the above into 1 job
Return an error to DI log if the user does not have permission to create an order for the acq unit specified in the PO

OPEN


Already in place for invoices. Can we reuse that work?
If vendor supplies order date in YYYYMMDD format, can we convert it to FOLIO YYYY-MM-DD default format? 

OPEN



Would we be able to support other order date formats?

OPEN



Test handling of multiple copies for multiple locations in the same POL; how can DI understand and parse properly? 

DEFERRED

Work on this after the multi-copy DI work in Orchid. Initial implementation will be single copy/single locationMARC field mapping syntax expansion required? Aim for repeatable 9xx field that has $a location code $b quantity maybe $c fund $d expense class?
For prices, FOLIO always wants an explicit decimal (e.g. 25.15, not 2516, correct?)

CLOSED

Yes
With invoices, data import does not try to correlate the expense classes allowed by particular funds (just display all the expense classes from Settings in a dropdown list). Should we refine that for Orders? How difficult would it be?

CLOSED

Per SMEs. no need to refine, at least to start with - show all funds and all expense classes in each dropdown.
Would Folijet need to take the receiving app into account? What happens if the Inventory records are created before the Order records, and not triggered by the POL's Inventory setting - would that be an issue?

CLOSED

Not in initial release, since the receiving pieces will be created by opening the order, not by DI
If order is created as Pending, but Instance/Holdings/Item has already been created by DI, what happens when the order is opened manually?

CLOSED

Initial release will be set up so that the Instance is created first, and then the POL linked to it. Then the holdings/items will be linked to the POL via the standard Orders module logic, when the PO is opened. SMEs want the Instance/Holdings/Item to be linked to the appropriate POL on the PO that was manually opened
If library wants no Inventory created, can job profile be set up to import MARC Bibs, but only create orders?

DEFERRED

Not in the initial release. Investigate as a possibility in the refinement releaseMay just take some testing 
From the SMEs: 

Can we assume that actions taken by the system are being done the same way as actions done by a user (except faster, and without any confirmation modals)?

CLOSED


Question for Tjet/Fjet

Yes. Currently there is just a small difference in processing requests if poline.source == "EBSCONET", which  was done for the specific 3rd party system, and it will not affect other flows.

Specific PO/POL data elements 

CLOSED



  • Vendor

CLOSED

Is there an endpoint where DI could do a lookup based on the vendor code? (an exact match on one or the other)

Question for Thunderjet

SN: We can use CQL language to filter organizations/vendors by code for example like this:

/organizations/organizations?query=(((code=AMAZ*)))

NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible

  • Created by

CLOSED

What data should be supplied - user who runs the import job?

Question for Thunderjet

SN: Yes, correct, active signed user. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. 

  • Created on

CLOSED

What data should be supplied - current date/time? Will it default if not supplied?

Question for Thunderjet

SN: Yes, current date/time. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client.

  • Approved by

CLOSED

What data should be supplied - user who runs the import job?Question for Thunderjet

Yes, and  this value will be populated automatically if approval required by tenant configuration
  • Approval date

CLOSED

What data should be supplied - current date/time? Will it default if not supplied?

Question for Thunderjet

Will be populated automatically with current date if approval required by tenant configuration

  • Date ordered

CLOSED

How is this value determined? By when the order is opened? Must it be supplied, or will it default if not supplied?

Question for Thunderjet

SN: Yes, it is set when order is opened, the default value will be supplied as current date/time 

Link to code

  • Product ID and Qualifier

CLOSED

What to do if qualifier in same field as number, e.g. 020$a12345678 (paperback)? Does DI need logic to break apart or can Orders app do it?

Question for Thunderjet

According to acquisition data model productId and qualifier live in different places, so it needs to be splitted before making order request

https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/mod-orders-storage/schemas/product_identifier.json

  • Product identifier type

CLOSED

This is a subset of a tenant's https://folio-snapshot.dev.folio.org/settings/inventory/identifierTypes. How do we know which ones to show in the dropdown list?

Question for Thunderjet

SN: In Orders UI For displaying product identifier types we use this query:

/identifier-types?limit=1000&query=cql.allRecords%3D1%20sortby%20name

So we use all available identifier types

  • Order format

CLOSED

Where does this list of reference values come from?

Question for Thunderjet

SN: It is fetched from acq-models github repository.

Allowed values:

"Electronic Resource",
"P/E Mix",
"Physical Resource",
"Other"

Link to code

  • POL Created on

CLOSED

What data should be supplied - current date/time? Will it default if not supplied?

Question for Thunderjet

SN: It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. 

  • Receipt status

CLOSED

Where does this list of reference values come from?

Question for Thunderjet

SN: It is fetched from acq-models github repository.

Allowed values:

"Awaiting Receipt",
"Cancelled",
"Fully Received",
"Partially Received",
"Pending",
"Receipt Not Required",
"Ongoing"

Link to code

NOTE from A-M: will only display Pending and Receipt not required, since those are the only permitted values when an order is created
  • Payment status

CLOSED

Where does this list of reference values come from?

Question for Thunderjet

SN: It is fetched from acq-models github repository.

Allowed values:

"Awaiting Payment",
"Cancelled",
"Fully Paid",
"Partially Paid",
"Payment Not Required",
"Pending",
"Ongoing"

Link to code

NOTE from A-M: will only display Pending and Payment not required, since those are the only permitted values when an order is created

  • Receiving workflow

CLOSED

Where does this list of reference values come from?

Question for Thunderjet

Based on value of poline.checkinItems

Synchroniz

Synchronized = false
Independent = trued = false
Independent = true
  • Vendor reference number type

CLOSED

Where does this list of reference values come from?

Question for Thunderjet

https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/common/schemas/reference_number_item.json
  • Account number


CLOSED
  1. How to get the list of account numbers to use for a dropdown list for a particular vendor?
  2. If the library transcribed an account number from an incoming record, as long as it matches an account number associated with the selected vendor, would that be acceptable? 
  3. If there are no account numbers associated with a vendor, but an account number is supplied for the POL, will that trigger an error in the POL?

Question for Thunderjet

  1. account numbers are part of organization records https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/mod-orgs/examples/organization_get.sample#L152
  2. -
  3. There is no validation of account numbers. The order can be created with not existing account number, but this can cause issues in future. For example in edifact export flow
  • Currency

CLOSED

Does case matter for the supplied value? (e.g USD vs usd vs Usd)

Question for Thunderjet

Yes, uppercase is mandatory

  • Fund/expense class

DEFERRED

For fund/expense class data, do we have to support fund:expense class from the beginning? Per the SMEs probably not. What do the developers think? OK to leave for POST-NOLANA? We will need to define field mapping syntax to allow for either format

Question for Thunderjet and Folijet

OK to wait until refinement feature per Folijet

  • Material supplier

CLOSED

Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other)

Question for Thunderjet

SN: We can use CQL language to filter organizations/vendors by code for example like this:

/organizations/organizations?query=(((code=AMAZ*)))

NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible

  • Create inventory

CLOSED

  1. Where does this list of reference values come from?
  2. Pay attention to this setting, or use the job profile sequencing instead?

Thunderjet and Folijet

Pay attention to this setting, since the Orders app will use it to create/link various inventory record types when the PO is opened

SN: It is fetched from acq-models github repository.

Allowed values:

"Instance, holding, item",
"Instance, holdings",
"Instance",
"None"

Link to code

  • Access provider

CLOSED

Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other)

Question for Thunderjet

SN: We can use CQL language to filter organizations/vendors by code for example like this:

/organizations/organizations?query=(((code=AMAZ*)))

NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible