Prep Jira: UXPROD-3361 - Prep for Importing Orders in MARC format IN PROGRESS (Morning Glory)
Data Import Dev Jira: UXPROD-185 - Import Orders in MARC format DRAFT (Nolana, Folijet)
Acquisitions Support Jira: TBD (Nolana, Thunderjet)
Overview slide deck: Click here
Technical/Developer details: Support importing orders from MARC files via Data Import
Rough draft of requirements and focus group meeting notes: Using MARC Bibs and Data Import to create FOLIO POLsData Import Initial Development Jira:
(Nolana/Orchid, Folijet) Jira Legacy server System JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key UXPROD-185
Data Import Refinement Jira:
Jira Legacy | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
Overview slide deck: Click here
Technical/Developer details: Support importing orders from MARC files via Data Import
Rough draft of requirements and focus group meeting notes: Using MARC Bibs and Data Import to create FOLIO POLs
Existing "Lehigh" workaround: Temporary Solution for Creating FOLIO Orders from MARC Data - draft
Problem(s):
- Some vendors send order-related data in MARC records, so that the receiving library can create orders in their local system from that data. The order data may be sent ay two points in the order workflow
- Point of order
- For orders placed on the vendor's ordering system, e.g. firm orders. For these, the vendor does not have a FOLIO POL, since the order is placed in the vendor's system before the order is created in FOLIO (question: would FOLIO need a way for the vendor to send back a POL and have it populate the FOLIO PO/POL fields? Dev preference is no, at least for first iteration, since it would add complexity. Note that this workflow is also covered by the FOLIO Order API process, which is real-time and more efficient than this workflow, at least for creating orders.
- For selections made in the vendor's ordering system, but not yet ordered. For these pre-order records, the vendor expects to receive back a finalized order from FOLIO, preferably as EDIFACT.
- Point of receipt/invoicing
- For materials invoiced automatically by the vendor and purchased by the library. In this flow, orders may only be created if the library wants to track the materials as individual titles and/or pay for the materials as individual titles.
- Most common materials supplied via this workflow:
- Approval plan automatic purchases (Physical and/or E) where titles are automatically supplied based on a profile that the library sets up with the vendor. Profiles may be based on subject language, publisher, or combinations of several characteristics.
- Demand Driven or Patron Driven Acquisitions (Mainly E) where titles are automatically invoiced based on patron usage passing a trigger point that causes titles to be purchased automatically by the library
- In this flow, there is no opportunity for FOLIO to return a POL to the vendor before invoicing, so the invoice match to the POL much rely on a Vendor Reference Number
- Point of order
...
- DI PO: Ann-Marie Breaux
- Acq PO: Dennis Bridges
- Cornell: Jenn Colt
- 5C: Jennifer Eustis, Sara Colglazier
- Michigan State: Kim Wiljanen, Lisa Smith
Use Cases & Requirements:
...
Requirement | Status | Notes | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Allow users create orders and order lines based on order data supplied in MARC bibliographic records |
| |||||||||||||||||||
Allow users to create field mapping profiles, action profiles, and job profiles to support MARC-based order data |
| |||||||||||||||||||
Allow users to create, but not update, POs/POLs via DI |
| Confirmed by Acq SIG SMEs. No need to consider updating unless significant use case identified in the future | ||||||||||||||||||
Data import match profiles not needed for Order records |
| Confirmed by Acq SIG SMEs. Since DI is only creating orders, use case for matching and updating an order does not apply. Use case for matching and then not creating an order: not common, and could only use Product ID or Vendor Ref ID for matching. Revisit if additional use cases identified in the future | ||||||||||||||||||
Use for one-time orders (physical, electronic, or both). Do not use for ongoing or package orders |
| Confirmed by Acq SIG SMEs. Revisit if any significant use case identified for Ongoing and/or Package orders | ||||||||||||||||||
Show all the PO/POL fields in the Field mapping profile, but suppress the PO/POL fields and values that are only used for ongoing or package orders. |
| Confirmed by Acq SIG SMEs. | ||||||||||||||||||
Allow POs to be created as Pending or Open |
| Confirmed by Acq SIG SMEs. Use cases for both identified. Will gather that info via a field at the top of the field mapping profile | ||||||||||||||||||
Allow override to the tenant's default POL limit setting |
| Confirmed by Acq SIG SMEs. Also, value may need to vary from one vendor to another or from one type of purchases to another. Will gather that info via a field at the top of the field mapping profile | ||||||||||||||||||
User should have an option to auto-receive |
| Confirmed by Acq SIG SMEs. Will gather that info via a field at the top of the field mapping profile. If PO is created as Pending, and then opened manually, can this requirement be included? Or only possible if the PO is created as Open?PO is created as Open? | ||||||||||||||||||
Add mapping syntax for TODAY + x days |
| Would be useful for fields like Activation date, Receipt date - but per DI SMEs, not important for now. Consider for refinement feature | ||||||||||||||||||
Honor the Acquisitions unit settings |
| Confirmed by Acq SIG SMEs. If import user invokes a job profile that creates POs, that user must be a member of the acq unit assigned by the field mapping profile, or the acq unit settings must allow non-members to create POs. This acq unit logic is already in place for Invoices. | ||||||||||||||||||
Have a way to create POLs for multiple copies with different locations |
| Confirmed by Acq SIG SMEs. talk with devs about whether the data needs to be configured in a certain way in the MARC file, e.g.
| ||||||||||||||||||
Have a way to create a POL that assigns more than 1 fund and/or expense class to cover the cost |
| Confirmed by Acq SIG SMEs. See above. Include a way to assign the fund/expense class by % or amount. | ||||||||||||||||||
Be able to accept fund and expense class whether they are supplied in separate MARC subfields, or in 1 MARC subfield, separated by a colon |
| Confirmed by Acq SIG SMEs. For example (Fund code = History, Expense class = Electronic)
| ||||||||||||||||||
If Orders app is creating Inventory records based on instruction from Data Import (or based on opening orders), follow the tenant's setting for Disable instance matching |
| Confirmed by Acq SIG SMEs | ||||||||||||||||||
Instances/Holdings/Items related to the created POL should display proper links in the Acquisition and Receiving history accordions, regardless of whether the POL or the Inventory records were created first |
| Confirmed by Acq SIG SMEs. Works now when order is opened and Inventory is created by the orders app. Does not work if Inventory records are associated with the POL after it has been opened. Needs further discussion with devs | ||||||||||||||||||
If a library wants to import a MARC Bib file to create orders, but does NOT want to create any Inventory records, will there be any problem? |
| May just require testing of a job profile with Create orders action, but no Create Instance action. | ||||||||||||||||||
If a library has an Order setting that POs must be approved before opening, will that setting always apply to orders created by Data Import? If yes, need an option to always create the orders as approved, or always create as pending? |
| Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M to ask on Acq Slack channelM asked on Acq Slack channel. If field mapping has approved selected, the user importing the file must have permission to approve orders. | ||||||||||||||||||
If a library has an Order setting that allows for the FOLIO PO to be edited, will that setting apply to orders created by Data Import? |
| Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M to ask asked on Acq Slack channel. Does not seem to be needed currently | ||||||||||||||||||
Will libraries want to map PO numbers from the incoming MARC records? |
| Discussed in Acq SIG 28 June 2022. No use case identified (yet) - wait and see if any identified before definite "No", since this decision would require coordination with Settings for editing PO numbers and max POLs per PO. If PO needs to not be mappable, easiest to disallow mapping the field completely on the field mapping screen (like HRID). A-M to ask on Acq Slack channel. | ||||||||||||||||||
If the PO number is mapped from a MARC field (e.g. 950$p) and the PO varies from one MARC Bib to another, what should FOLIO do? |
| Wait until above question is resolved. Will not be mapped, so does not need to be dealt with |
Discussed, but not requirements
- Create separate orders per invoice. FOLIO does not currently have the logic to parse MARC files into separate invoices. If this is required, library should work with their materials vendor to get separate MARC files per invoice, and then ensure that the POL line limit in the field matting profile is set larger than the maximum number of lines allowed per invoice from that vendor.
- Be able to select an order template within the Order field mapping profile: The Data Import field mapping profile basically acts as a template, gathering default and mapped data for a particular type of file from a particular vendor. Once the field mapping profile is created, it does not have to be adjusted to accommodate each individual import for the same type of file. To tie the field mapping profile to the order templates would require significant development effort, along with problem handling if the field mapping profile and template have different values for the same field.
- Create invoices from the MARC bib data. Separate feature for that. Cannot be accommodated in the scope of Nolana. For now, create orders based on MARC Bibs, and then create invoices from EDFACT data, via a separate job profile.
- Assign order notes to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
- Assign tags to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
- Schedule the pickup and import of MARC Bib files, assigning the correct job profile. Not a requirement in Nolana. Added some details into UXPROD-662
Mockup for FOLIO PO/POL Field Mapping Screen
See requirement details in the screens attached to UIDATIMP-294
Sample MARC records with Order data
...
Question | Status | Conclusion | Comments | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
If order is created as Open, 1) will the receiving pieces be created and 2) will they show in the Inventory holdings record? |
| For initial implementation, receiving pieces, holdings, and items will be created by the order being opened |
Developer Questions:
Opening orders
...
Possible Nolana Job profile sequence (Data Import))
See video here
- Create Instance (source = MARC)
- Create Orders (with data from MARC Bib, Order data in MARC Bib, Order field mapping profile)
- DI would need to pass the Instance UUID as part of the Create orders info
- What happens if order is created as Pending?
- How to create source = MARC
- Open the order
- Which creates Holdings and Item based on info the POL knows, and provides the POL link in the Holdings and Item
- Question about whether opening the PO creates a new holdings or links to an existing holdings in the Instance
Second Nolana Data Import job
- Match on ??? VRN (to find newly-created Holdings and Item)
- Action: Update holdings - field mapping profile; only worry about 1 holdings for now
- Action: Update item - field mapping profile; only worry about holdings for nownow
Caveats:
- For Nolana, have to run 2 import jobs
- VRN must be in the original MARC Bib file
- Could we run it as 1 job with update actions after the PO is opened?
There are pros and cons to Orders app and DI app creating Holdings and Items
...
Question | Status | Conclusion | Comments | |||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
How to handle the POL limit variation (in the field mapping profile, users can select to use the existing default or override to a number higher or lower than the default) |
| See logic in field mapping profile - display the existing POL limit setting, but allow override of it | ||||||||||||||||||
If DI only allows creation of one-time, non-package orders, will there be any problems if we remove all the non-one-time, non-package data elements from the field mapping screen? |
| Not a problem | ||||||||||||||||||
We want to use the existing Orders logic as much as possible, but we also want Instance/Holdings/Item linked with correct POL even when created before the POL. |
| How much extra logic would be needed? |
| For initial release, create the Instance first via DI, then have the order's opening create the holdings and items. Then update the holdings and item via a second DI job. In the refinement release, aim to combine the above into 1 job | ||||||||||||||||
Return an error to DI log if the user does not have permission to create an order for the acq unit specified in the PO |
| Already in place for invoices. Can we reuse that work? | ||||||||||||||||||
If vendor supplies order date in YYYYMMDD format, can we convert it to FOLIO YYYY-MM-DD default format? |
| |||||||||||||||||||
Would we be able to support other order date formats? |
| |||||||||||||||||||
Test handling of multiple copies for multiple locations in the same POL; how can DI understand and parse properly? |
| Work on this after the multi-copy DI work in Orchid. Initial implementation will be single copy/single location | MARC field mapping syntax expansion required? Aim for repeatable 9xx field that has $a location code $b quantity maybe $c fund $d expense class? | |||||||||||||||||
For prices, FOLIO always wants an explicit decimal (e.g. 25.15, not 2516, correct?) |
| Question for ThunderjetYes | ||||||||||||||||||
With invoices, data import does not try to correlate the expense classes allowed by particular funds (just display all the expense classes from Settings in a dropdown list). Should we refine that for Orders? How difficult would it be? |
| Per SMEs. no need to refine, at least to start with - show all funds and all expense classes in each dropdown. | ||||||||||||||||||
Would Folijet need to take the receiving app into account? What happens if the Inventory records are created before the Order records, and not triggered by the POL's Inventory setting - would that be an issue? |
| Not in initial release, since the receiving pieces will be created by opening the order, not by DI | ||||||||||||||||||
If order is created as Pending, but Instance/Holdings/Item has already been created by DI, what happens when the order is opened manually? |
| Initial release will be set up so that the Instance is created first, and then the POL linked to it. Then the holdings/items will be linked to the POL via the standard Orders module logic, when the PO is opened. | SMEs want the Instance/Holdings/Item to be linked to the appropriate POL on the PO that was manually opened | |||||||||||||||||
If library wants no Inventory created, can job profile be set up to import MARC Bibs, but only create orders? |
| Not in the initial release. Investigate as a possibility in the refinement release | May just take some testing | |||||||||||||||||
From the SMEs: Can we assume that actions taken by the system are being done the same way as actions done by a user (except faster, and without any confirmation modals)? |
| Question for Tjet/Fjet Yes. Currently there is just a small difference in processing requests if poline.source == "EBSCONET", which was done for the specific 3rd party system, and it will not affect other flows. | ||||||||||||||||||
Specific PO/POL data elements |
| |||||||||||||||||||
| Vendor
| Question for Thunderjet |
| Is there an endpoint where DI could do a lookup based on the vendor code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible | |||||||||||||||
| Created by
| Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other) | Question for Thunderjet |
| What data should be supplied - user who runs the import job? | Question for Thunderjet SN: Yes, correct, active signed user. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. | ||||||||||||||
|
| What data should be supplied - user who runs the import jobbe supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: Yes, correct, active signed usercurrent date/time. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data.Created on | |||||||||||||||||
|
| What data should be supplied - user who runs the import job? | Question for Thunderjet Yes, and this value will be populated automatically if approval required by tenant configuration | |||||||||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet Will be populated automatically with current date if approval required by tenant configuration | |||||||||||||||||
|
| How is this value determined? By when the order is opened? Must it be supplied, or will it default if not supplied? | Question for Thunderjet SN: Yes, current date/time. It it is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Approved bywhen order is opened, the default value will be supplied as current date/time | |||||||||||||||||
|
| What to do if qualifier in same field as number, e.g. 020$a12345678 (paperback)? Does DI need logic to break apart or can Orders app do it? |
| Question for Thunderjet | ||||||||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet | |||||||||||||||||
According to acquisition data model productId and qualifier live in different places, so it needs to be splitted before making order request | ||||||||||||||||||||
|
| This is a subset of a tenant's https://folio-snapshot.dev.folio.org/settings/inventory/identifierTypes. How do we know which ones to show in the dropdown list? | Question for Thunderjet SN: Yes, it is set when order is opened, the defaut value will be supplied as current date/time Product ID and QualifierIn Orders UI For displaying product identifier types we use this query: /identifier-types?limit=1000&query=cql.allRecords%3D1%20sortby%20name So we use all available identifier types | |||||||||||||||||
|
| What to do if qualifier in same field as number, e.g. 020$a12345678 (paperback)? Does DI need logic to break apart or can Orders app do it? | Question for Thunderjet |
|
| Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Electronic Resource",
"P/E Mix",
"Physical Resource",
"Other" | |||||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. | |||||||||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Electronic ResourceAwaiting Receipt", "Cancelled", "P/EFully MixReceived", "PhysicalPartially ResourceReceived", "Pending"Other" | |||||||||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. | |||||||||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet, "Receipt Not Required", "Ongoing" | |||||||||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Awaiting Payment", "Cancelled", "Fully Paid", "Partially Paid", "Payment Not Required", "Pending", "Ongoing" NOTE from A-M: will only display Pending and Payment not required, since those are the only permitted values when an order is created | |||||||||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet Based on value of poline.checkinItems Synchroniz Synchronized = false | |||||||||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/common/schemas/reference_number_item.json | |||||||||||||||||
|
|
| Question for Thunderjetfor Thunderjet
| |||||||||||||||||
|
| Does case matter for the supplied value? (e.g USD vs usd vs Usd) | Question for Thunderjet Yes, uppercase is mandatory | |||||||||||||||||
|
| For fund/expense class data, do we have to support fund:expense class from the beginning? Per the SMEs probably not. What do the developers think? OK to leave for POST-NOLANA? We will need to define field mapping syntax to allow for either format | Question for Thunderjet and Folijet OK to wait until refinement feature per Folijet | |||||||||||||||||
|
| Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other)Question for Thunderjetcode? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible | |||||||||||||||||
|
|
| Thunderjet and Folijet Pay attention to this setting, since the Orders app will use it to create/link various inventory record types when the PO is opened SN: It is fetched from acq-models github repository. Allowed values: "Instance, Holdingholding, Itemitem", "Instance, Holdingholdings", "Instance", "None" | |||||||||||||||||
|
| Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible | |||||||||||||||||