Data Import Initial Development Jira: UXPROD-185 (Nolana/Orchid, Folijet)
Data Import Refinement Jira: UXPROD-3814 (Release TBD)
Overview slide deck: Click here
Technical/Developer details: Support importing orders from MARC files via Data Import
Rough draft of requirements and focus group meeting notes: Using MARC Bibs and Data Import to create FOLIO POLs
Existing "Lehigh" workaround: Temporary Solution for Creating FOLIO Orders from MARC Data - draft
Problem(s):
- Some vendors send order-related data in MARC records, so that the receiving library can create orders in their local system from that data. The order data may be sent ay two points in the order workflow
- Point of order
- For orders placed on the vendor's ordering system, e.g. firm orders. For these, the vendor does not have a FOLIO POL, since the order is placed in the vendor's system before the order is created in FOLIO (question: would FOLIO need a way for the vendor to send back a POL and have it populate the FOLIO PO/POL fields? Dev preference is no, at least for first iteration, since it would add complexity. Note that this workflow is also covered by the FOLIO Order API process, which is real-time and more efficient than this workflow, at least for creating orders.
- For selections made in the vendor's ordering system, but not yet ordered. For these pre-order records, the vendor expects to receive back a finalized order from FOLIO, preferably as EDIFACT.
- Point of receipt/invoicing
- For materials invoiced automatically by the vendor and purchased by the library. In this flow, orders may only be created if the library wants to track the materials as individual titles and/or pay for the materials as individual titles.
- Most common materials supplied via this workflow:
- Approval plan automatic purchases (Physical and/or E) where titles are automatically supplied based on a profile that the library sets up with the vendor. Profiles may be based on subject language, publisher, or combinations of several characteristics.
- Demand Driven or Patron Driven Acquisitions (Mainly E) where titles are automatically invoiced based on patron usage passing a trigger point that causes titles to be purchased automatically by the library
- In this flow, there is no opportunity for FOLIO to return a POL to the vendor before invoicing, so the invoice match to the POL much rely on a Vendor Reference Number
- Point of order
Focus Group for drafting requirements:
- DI PO: Ann-Marie Breaux
- Acq PO: Dennis Bridges
- Cornell: Jenn Colt
- 5C: Jennifer Eustis, Sara Colglazier
- Michigan State: Kim Wiljanen, Lisa Smith
Use Cases & Requirements:
Requirement | Status | Notes |
|---|---|---|
| Allow users create orders and order lines based on order data supplied in MARC bibliographic records | CLOSED | |
| Allow users to create field mapping profiles, action profiles, and job profiles to support MARC-based order data | CLOSED | |
| Allow users to create, but not update, POs/POLs via DI | CLOSED | Confirmed by Acq SIG SMEs. No need to consider updating unless significant use case identified in the future |
| Data import match profiles not needed for Order records | CLOSED | Confirmed by Acq SIG SMEs. Since DI is only creating orders, use case for matching and updating an order does not apply. Use case for matching and then not creating an order: not common, and could only use Product ID or Vendor Ref ID for matching. Revisit if additional use cases identified in the future |
| Use for one-time orders (physical, electronic, or both). Do not use for ongoing or package orders | CLOSED | Confirmed by Acq SIG SMEs. Revisit if any significant use case identified for Ongoing and/or Package orders |
| Show all the PO/POL fields in the Field mapping profile, but suppress the PO/POL fields and values that are only used for ongoing or package orders. | CLOSED | Confirmed by Acq SIG SMEs. |
| Allow POs to be created as Pending or Open | CLOSED | Confirmed by Acq SIG SMEs. Use cases for both identified. Will gather that info via a field at the top of the field mapping profile |
| Allow override to the tenant's default POL limit setting | CLOSED | Confirmed by Acq SIG SMEs. Also, value may need to vary from one vendor to another or from one type of purchases to another. Will gather that info via a field at the top of the field mapping profile |
| User should have an option to auto-receive | DEFERRED | Confirmed by Acq SIG SMEs. Will gather that info via a field at the top of the field mapping profile. If PO is created as Pending, and then opened manually, can this requirement be included? Or only possible if the PO is created as Open? |
| Add mapping syntax for TODAY + x days | DEFERRED | Would be useful for fields like Activation date, Receipt date - but per DI SMEs, not important for now. Consider for refinement feature |
| Honor the Acquisitions unit settings | CLOSED | Confirmed by Acq SIG SMEs. If import user invokes a job profile that creates POs, that user must be a member of the acq unit assigned by the field mapping profile, or the acq unit settings must allow non-members to create POs. This acq unit logic is already in place for Invoices. |
| Have a way to create POLs for multiple copies with different locations | CLOSED | Confirmed by Acq SIG SMEs. talk with devs about whether the data needs to be configured in a certain way in the MARC file, e.g.
|
| Have a way to create a POL that assigns more than 1 fund and/or expense class to cover the cost | CLOSED | Confirmed by Acq SIG SMEs. See above. Include a way to assign the fund/expense class by % or amount. |
| Be able to accept fund and expense class whether they are supplied in separate MARC subfields, or in 1 MARC subfield, separated by a colon | CLOSED | Confirmed by Acq SIG SMEs. For example (Fund code = History, Expense class = Electronic)
|
| If Orders app is creating Inventory records based on instruction from Data Import (or based on opening orders), follow the tenant's setting for Disable instance matching | CLOSED | Confirmed by Acq SIG SMEs |
| Instances/Holdings/Items related to the created POL should display proper links in the Acquisition and Receiving history accordions, regardless of whether the POL or the Inventory records were created first | CLOSED | Confirmed by Acq SIG SMEs. Works now when order is opened and Inventory is created by the orders app. Does not work if Inventory records are associated with the POL after it has been opened. Needs further discussion with devs |
| If a library wants to import a MARC Bib file to create orders, but does NOT want to create any Inventory records, will there be any problem? | DEFERRED | May just require testing of a job profile with Create orders action, but no Create Instance action. |
| If a library has an Order setting that POs must be approved before opening, will that setting always apply to orders created by Data Import? If yes, need an option to always create the orders as approved, or always create as pending? | CLOSED | Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M asked on Acq Slack channel. If field mapping has approved selected, the user importing the file must have permission to approve orders. |
| If a library has an Order setting that allows for the FOLIO PO to be edited, will that setting apply to orders created by Data Import? | CLOSED | Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M asked on Acq Slack channel. Does not seem to be needed currently |
| Will libraries want to map PO numbers from the incoming MARC records? | CLOSED | Discussed in Acq SIG 28 June 2022. No use case identified (yet) - wait and see if any identified before definite "No", since this decision would require coordination with Settings for editing PO numbers and max POLs per PO. If PO needs to not be mappable, easiest to disallow mapping the field completely on the field mapping screen (like HRID). A-M to ask on Acq Slack channel. |
| If the PO number is mapped from a MARC field (e.g. 950$p) and the PO varies from one MARC Bib to another, what should FOLIO do? | CLOSED | Wait until above question is resolved. Will not be mapped, so does not need to be dealt with |
Discussed, but not requirements
- Create separate orders per invoice. FOLIO does not currently have the logic to parse MARC files into separate invoices. If this is required, library should work with their materials vendor to get separate MARC files per invoice, and then ensure that the POL line limit in the field matting profile is set larger than the maximum number of lines allowed per invoice from that vendor.
- Be able to select an order template within the Order field mapping profile: The Data Import field mapping profile basically acts as a template, gathering default and mapped data for a particular type of file from a particular vendor. Once the field mapping profile is created, it does not have to be adjusted to accommodate each individual import for the same type of file. To tie the field mapping profile to the order templates would require significant development effort, along with problem handling if the field mapping profile and template have different values for the same field.
- Create invoices from the MARC bib data. Separate feature for that. Cannot be accommodated in the scope of Nolana. For now, create orders based on MARC Bibs, and then create invoices from EDFACT data, via a separate job profile.
- Assign order notes to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
- Assign tags to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
- Schedule the pickup and import of MARC Bib files, assigning the correct job profile. Not a requirement in Nolana. Added some details into UXPROD-662
Mockup for FOLIO PO/POL Field Mapping Screen
See requirement details in the screens attached to UIDATIMP-294
Sample MARC records with Order data
No FOLIO POL
LDR 01826cam 2200493 i 4500
001 1104214104
003 OCoLC
005 20191218144549.9
008 190426s2020\\\\nyu\\\\\\b\\\\001\0\eng\\
010 \\$a 2019020718
040 \\$aDLC$beng$erda$cDLC$dOCLCO$dBDX$dOCLCF$dYDX
019 \\$a1104218545
020 \\$a9780190068479$qhardcover
020 \\$a0190068477$qhardcover
020 \\$z9780190068509$qelectronic book
020 \\$z9780190068486$quelectronic book
020 \\$z9780190068493$qelectronic publication
035 \\$a(OCoLC)1104214104$z(OCoLC)1104218545
042 \\$apcc
049 \\$aAUMM
050 00$aML3800$b.B75 2020
082 00$a781.1/7$223
100 1\$aBonds, Mark Evan,$eauthor.
245 14$aThe Beethoven syndrome :$bhearing music as autobiography /$cMark Evan Bonds.
264 \1$aNew York, NY :$bOxford University Press,$c[2020]
300 \\$axii, 325 pages ;$c25 cm
336 \\$atext$btxt$2rdacontent
337 \\$aunmediated$bn$2rdamedia
338 \\$avolume$bnc$2rdacarrier
504 \\$aIncludes bibliographical references and index.
600 10$aBeethoven, Ludwig van,$d1770-1827$xAppreciation$xHistory.
600 17$aBeethoven, Ludwig van,$d1770-1827.$2fast$0(OCoLC)fst00042803
650 \0$aMusic$xPhilosophy and aesthetics$xHistory.
650 \0$aExpression (Philosophy)$xHistory.
650 \7$aArt appreciation.$2fast$0(OCoLC)fst00815447
650 \7$aExpression (Philosophy)$2fast$0(OCoLC)fst00918886
650 \7$aMusic$xPhilosophy and aesthetics.$2fast$0(OCoLC)fst01030408
655 \7$aHistory.$2fast$0(OCoLC)fst01411628
949 \1$aML3800$b.B75 2020$e0$gOO$i312066040927765$lKU/CC/DI/M ($a & $b = call number, $e = misc library code, $g = misc library code, $i = barcode number, $l = location)
980 \\$a200109$b35.00$dMUSIAUAM$e28.88$f40029640970$g1 ($a = invoice date, $b = list price, $d = fund code, $e = net price, $f = vendor reference number (VRN), $g = quantity)
SME Questions:
Question | Status | Conclusion | Comments |
|---|---|---|---|
| If order is created as Open, 1) will the receiving pieces be created and 2) will they show in the Inventory holdings record? | CLOSED | For initial implementation, receiving pieces, holdings, and items will be created by the order being opened |
Developer Questions:
Opening orders
How should Data Import handle the opening of orders:
- Use the Inventory interaction setting from the POL
- Use the Data import job profile matching and sequencing
- Allow the user to choose (at the profile level or POL level?)
Job needs to create
- SRS MARC Bib
- Source = MARC Instance
- Holdings (as many as there are locations)
- If created by Order app, can create multi, with POL link and with limited details
- If created by DI app, can create single (for now), multi in the future, but with more details
- Could maybe be added by a secondary Update job profile
- Item (as many as there are copies)
- If created by Order app, can create multi, with POL link and with limited details
- If created by DI app, can create single (for now), multi in the future, but with more details
- Could maybe be added by a secondary Update job profile
Possible Nolana Job profile sequence (Data Import)
See video here
- Create Instance (source = MARC)
- Create Orders (with data from MARC Bib, Order data in MARC Bib, Order field mapping profile)
- DI would need to pass the Instance UUID as part of the Create orders info
- What happens if order is created as Pending?
- How to create source = MARC
- Open the order
- Which creates Holdings and Item based on info the POL knows, and provides the POL link in the Holdings and Item
- Question about whether opening the PO creates a new holdings or links to an existing holdings in the Instance
Second Nolana Data Import job
- Match on VRN (to find newly-created Holdings and Item)
- Action: Update holdings - field mapping profile; only worry about 1 holdings for now
- Action: Update item - field mapping profile; only worry about holdings for now
Caveats:
- For Nolana, have to run 2 import jobs
- VRN must be in the original MARC Bib file
- Could we run it as 1 job with update actions after the PO is opened?
There are pros and cons to Orders app and DI app creating Holdings and Items
Per Olamide, keep as much business logic in the other apps (Inventory, Orders, Invoices) as possible, instead of Data Import having to understand and replicate that logic
Background:
- Preference is to use the Orders engine to create the Inventory records
- Orders can be created as Pending or Open via Data Import
- Changing an Order’s status from Pending to Open triggers:
- Creation of encumbrances (if fund and price supplied in POL)
- Creation of receiving pieces (depending on receiving setting in POL)
- Creation of Instance, Holdings, Items (depending on Inventory setting in POL)
- Currently orders feed all data to Inventory to create Instance, Holdings, Item
- The Order app’s Inventory matching is not as flexible as Data Import’s and is all-or-nothing, depending on the setting
- POLs are only linked with Instances, Holdings, Items if those Inventory records are created by opening orders
- Can those POL links be created/changed manually on Inventory records
- An Inventory record cannot be matched/updated via POL/VRN unless the Inventory record is linked to the POL; this is a very important workflow for libraries
- The data on an incoming MARC Bib record and in a data import profile can create richer Inventory records than the Orders app can
- Data import can only create a single holdings and item via an imported MARC Bib at this time, though that will change in Orchid or Poppy; Orders app can create multiple holdings and items
Question | Status | Conclusion | Comments |
|---|---|---|---|
| How to handle the POL limit variation (in the field mapping profile, users can select to use the existing default or override to a number higher or lower than the default) | CLOSED | See logic in field mapping profile - display the existing POL limit setting, but allow override of it | |
| If DI only allows creation of one-time, non-package orders, will there be any problems if we remove all the non-one-time, non-package data elements from the field mapping screen? | CLOSED | Not a problem | |
| We want to use the existing Orders logic as much as possible, but we also want Instance/Holdings/Item linked with correct POL even when created before the POL. | CLOSED | For initial release, create the Instance first via DI, then have the order's opening create the holdings and items. Then update the holdings and item via a second DI job. In the refinement release, aim to combine the above into 1 job | |
| Return an error to DI log if the user does not have permission to create an order for the acq unit specified in the PO | OPEN | Already in place for invoices. Can we reuse that work? | |
| If vendor supplies order date in YYYYMMDD format, can we convert it to FOLIO YYYY-MM-DD default format? | OPEN | ||
| Would we be able to support other order date formats? | OPEN | ||
Test handling of multiple copies for multiple locations in the same POL; how can DI understand and parse properly? | DEFERRED | Work on this after the multi-copy DI work in Orchid. Initial implementation will be single copy/single location | MARC field mapping syntax expansion required? Aim for repeatable 9xx field that has $a location code $b quantity maybe $c fund $d expense class? |
| For prices, FOLIO always wants an explicit decimal (e.g. 25.15, not 2516, correct?) | CLOSED | Yes | |
| With invoices, data import does not try to correlate the expense classes allowed by particular funds (just display all the expense classes from Settings in a dropdown list). Should we refine that for Orders? How difficult would it be? | CLOSED | Per SMEs. no need to refine, at least to start with - show all funds and all expense classes in each dropdown. | |
| Would Folijet need to take the receiving app into account? What happens if the Inventory records are created before the Order records, and not triggered by the POL's Inventory setting - would that be an issue? | CLOSED | Not in initial release, since the receiving pieces will be created by opening the order, not by DI | |
| If order is created as Pending, but Instance/Holdings/Item has already been created by DI, what happens when the order is opened manually? | CLOSED | Initial release will be set up so that the Instance is created first, and then the POL linked to it. Then the holdings/items will be linked to the POL via the standard Orders module logic, when the PO is opened. | SMEs want the Instance/Holdings/Item to be linked to the appropriate POL on the PO that was manually opened |
| If library wants no Inventory created, can job profile be set up to import MARC Bibs, but only create orders? | DEFERRED | Not in the initial release. Investigate as a possibility in the refinement release | May just take some testing |
| From the SMEs: Can we assume that actions taken by the system are being done the same way as actions done by a user (except faster, and without any confirmation modals)? | CLOSED | Question for Tjet/Fjet Yes. Currently there is just a small difference in processing requests if poline.source == "EBSCONET", which was done for the specific 3rd party system, and it will not affect other flows. | |
| Specific PO/POL data elements | CLOSED | ||
| CLOSED | Is there an endpoint where DI could do a lookup based on the vendor code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible |
| CLOSED | What data should be supplied - user who runs the import job? | Question for Thunderjet SN: Yes, correct, active signed user. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data.
|
| CLOSED | What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: Yes, current date/time. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. |
| CLOSED | What data should be supplied - user who runs the import job? | Question for Thunderjet Yes, and this value will be populated automatically if approval required by tenant configuration |
| CLOSED | What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet Will be populated automatically with current date if approval required by tenant configuration |
| CLOSED | How is this value determined? By when the order is opened? Must it be supplied, or will it default if not supplied? | Question for Thunderjet SN: Yes, it is set when order is opened, the default value will be supplied as current date/time |
| CLOSED | What to do if qualifier in same field as number, e.g. 020$a12345678 (paperback)? Does DI need logic to break apart or can Orders app do it? | Question for Thunderjet According to acquisition data model productId and qualifier live in different places, so it needs to be splitted before making order request |
| CLOSED | This is a subset of a tenant's https://folio-snapshot.dev.folio.org/settings/inventory/identifierTypes. How do we know which ones to show in the dropdown list? | Question for Thunderjet SN: In Orders UI For displaying product identifier types we use this query: /identifier-types?limit=1000&query=cql.allRecords%3D1%20sortby%20name So we use all available identifier types |
| CLOSED | Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Electronic Resource", "P/E Mix", "Physical Resource", "Other" |
| CLOSED | What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data.
|
| CLOSED | Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Awaiting Receipt", "Cancelled", "Fully Received", "Partially Received", "Pending", "Receipt Not Required", "Ongoing" |
| CLOSED | Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Awaiting Payment", "Cancelled", "Fully Paid", "Partially Paid", "Payment Not Required", "Pending", "Ongoing" NOTE from A-M: will only display Pending and Payment not required, since those are the only permitted values when an order is created |
| CLOSED | Where does this list of reference values come from? | Question for Thunderjet Based on value of poline.checkinItems Synchroniz Synchronized = false |
| CLOSED | Where does this list of reference values come from? | Question for Thunderjet https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/common/schemas/reference_number_item.json |
|
| Question for Thunderjet
| |
| CLOSED | Does case matter for the supplied value? (e.g USD vs usd vs Usd) | Question for Thunderjet Yes, uppercase is mandatory |
| DEFERRED | For fund/expense class data, do we have to support fund:expense class from the beginning? Per the SMEs probably not. What do the developers think? OK to leave for POST-NOLANA? We will need to define field mapping syntax to allow for either format | Question for Thunderjet and Folijet OK to wait until refinement feature per Folijet |
| CLOSED | Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible |
| CLOSED |
| Thunderjet and Folijet Pay attention to this setting, since the Orders app will use it to create/link various inventory record types when the PO is opened SN: It is fetched from acq-models github repository. Allowed values: "Instance, holding, item", "Instance, holdings", "Instance", "None" |
| CLOSED | Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible |
