Data Import Initial Development Jira:
(Nolana/Orchid, Folijet)Jira Legacy server System JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key UXPROD-185
Data Import Refinement Jira:
| Jira Legacy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Overview slide deck: Click here
...
Existing "Lehigh" workaround: Temporary Solution for Creating FOLIO Orders from MARC Data - draft
Problem(s):
- Some vendors send order-related data in MARC records, so that the receiving library can create orders in their local system from that data. The order data may be sent ay two points in the order workflow
- Point of order
- For orders placed on the vendor's ordering system, e.g. firm orders. For these, the vendor does not have a FOLIO POL, since the order is placed in the vendor's system before the order is created in FOLIO (question: would FOLIO need a way for the vendor to send back a POL and have it populate the FOLIO PO/POL fields? Dev preference is no, at least for first iteration, since it would add complexity. Note that this workflow is also covered by the FOLIO Order API process, which is real-time and more efficient than this workflow, at least for creating orders.
- For selections made in the vendor's ordering system, but not yet ordered. For these pre-order records, the vendor expects to receive back a finalized order from FOLIO, preferably as EDIFACT.
- Point of receipt/invoicing
- For materials invoiced automatically by the vendor and purchased by the library. In this flow, orders may only be created if the library wants to track the materials as individual titles and/or pay for the materials as individual titles.
- Most common materials supplied via this workflow:
- Approval plan automatic purchases (Physical and/or E) where titles are automatically supplied based on a profile that the library sets up with the vendor. Profiles may be based on subject language, publisher, or combinations of several characteristics.
- Demand Driven or Patron Driven Acquisitions (Mainly E) where titles are automatically invoiced based on patron usage passing a trigger point that causes titles to be purchased automatically by the library
- In this flow, there is no opportunity for FOLIO to return a POL to the vendor before invoicing, so the invoice match to the POL much rely on a Vendor Reference Number
- Point of order
...
- Create separate orders per invoice. FOLIO does not currently have the logic to parse MARC files into separate invoices. If this is required, library should work with their materials vendor to get separate MARC files per invoice, and then ensure that the POL line limit in the field matting profile is set larger than the maximum number of lines allowed per invoice from that vendor.
- Be able to select an order template within the Order field mapping profile: The Data Import field mapping profile basically acts as a template, gathering default and mapped data for a particular type of file from a particular vendor. Once the field mapping profile is created, it does not have to be adjusted to accommodate each individual import for the same type of file. To tie the field mapping profile to the order templates would require significant development effort, along with problem handling if the field mapping profile and template have different values for the same field.
- Create invoices from the MARC bib data. Separate feature for that. Cannot be accommodated in the scope of Nolana. For now, create orders based on MARC Bibs, and then create invoices from EDFACT data, via a separate job profile.
- Assign order notes to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
- Assign tags to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
- Schedule the pickup and import of MARC Bib files, assigning the correct job profile. Not a requirement in Nolana. Added some details into UXPROD-662
Mockup for FOLIO PO/POL Field Mapping Screen
See requirement details in the screens attached to UIDATIMP-294
Sample MARC records with Order data
...
Question | Status | Conclusion | Comments | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| How to handle the POL limit variation (in the field mapping profile, users can select to use the existing default or override to a number higher or lower than the default) |
| See logic in field mapping profile - display the existing POL limit setting, but allow override of it | |||||||||||
| If DI only allows creation of one-time, non-package orders, will there be any problems if we remove all the non-one-time, non-package data elements from the field mapping screen? |
| Not a problem | |||||||||||
| We want to use the existing Orders logic as much as possible, but we also want Instance/Holdings/Item linked with correct POL even when created before the POL. |
| For initial release, create the Instance first via DI, then have the order's opening create the holdings and items. Then update the holdings and item via a second DI job. In the refinement release, aim to combine the above into 1 job | |||||||||||
| Return an error to DI log if the user does not have permission to create an order for the acq unit specified in the PO |
| Already in place for invoices. Can we reuse that work? | |||||||||||
| If vendor supplies order date in YYYYMMDD format, can we convert it to FOLIO YYYY-MM-DD default format? |
| ||||||||||||
| Would we be able to support other order date formats? |
| ||||||||||||
Test handling of multiple copies for multiple locations in the same POL; how can DI understand and parse properly? |
| Work on this after the multi-copy DI work in Orchid. Initial implementation will be single copy/single location | MARC field mapping syntax expansion required? Aim for repeatable 9xx field that has $a location code $b quantity maybe $c fund $d expense class? | ||||||||||
| For prices, FOLIO always wants an explicit decimal (e.g. 25.15, not 2516, correct?) |
| Yes | |||||||||||
| With invoices, data import does not try to correlate the expense classes allowed by particular funds (just display all the expense classes from Settings in a dropdown list). Should we refine that for Orders? How difficult would it be? |
| Per SMEs. no need to refine, at least to start with - show all funds and all expense classes in each dropdown. | |||||||||||
| Would Folijet need to take the receiving app into account? What happens if the Inventory records are created before the Order records, and not triggered by the POL's Inventory setting - would that be an issue? |
| Not in initial release, since the receiving pieces will be created by opening the order, not by DI | |||||||||||
| If order is created as Pending, but Instance/Holdings/Item has already been created by DI, what happens when the order is opened manually? |
| Initial release will be set up so that the Instance is created first, and then the POL linked to it. Then the holdings/items will be linked to the POL via the standard Orders module logic, when the PO is opened. | SMEs want the Instance/Holdings/Item to be linked to the appropriate POL on the PO that was manually opened | ||||||||||
| If library wants no Inventory created, can job profile be set up to import MARC Bibs, but only create orders? |
| Not in the initial release. Investigate as a possibility in the refinement release | May just take some testing | ||||||||||
| From the SMEs: Can we assume that actions taken by the system are being done the same way as actions done by a user (except faster, and without any confirmation modals)? |
| Question for Tjet/Fjet Yes. Currently there is just a small difference in processing requests if poline.source == "EBSCONET", which was done for the specific 3rd party system, and it will not affect other flows. | |||||||||||
| Specific PO/POL data elements |
| ||||||||||||
|
| Is there an endpoint where DI could do a lookup based on the vendor code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible | ||||||||||
|
| What data should be supplied - user who runs the import job? | Question for Thunderjet SN: Yes, correct, active signed user. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data.
| ||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: Yes, current date/time. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. | ||||||||||
|
| What data should be supplied - user who runs the import job? | Question for Thunderjet Yes, and this value will be populated automatically if approval required by tenant configuration | ||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet Will be populated automatically with current date if approval required by tenant configuration | ||||||||||
|
| How is this value determined? By when the order is opened? Must it be supplied, or will it default if not supplied? | Question for Thunderjet SN: Yes, it is set when order is opened, the default value will be supplied as current date/time | ||||||||||
|
| What to do if qualifier in same field as number, e.g. 020$a12345678 (paperback)? Does DI need logic to break apart or can Orders app do it? | Question for Thunderjet According to acquisition data model productId and qualifier live in different places, so it needs to be splitted before making order request | ||||||||||
|
| This is a subset of a tenant's https://folio-snapshot.dev.folio.org/settings/inventory/identifierTypes. How do we know which ones to show in the dropdown list? | Question for Thunderjet SN: In Orders UI For displaying product identifier types we use this query: /identifier-types?limit=1000&query=cql.allRecords%3D1%20sortby%20name So we use all available identifier types | ||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Electronic Resource", "P/E Mix", "Physical Resource", "Other" | ||||||||||
|
| What data should be supplied - current date/time? Will it default if not supplied? | Question for Thunderjet SN: It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data.
| ||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Awaiting Receipt", "Cancelled", "Fully Received", "Partially Received", "Pending", "Receipt Not Required", "Ongoing" | ||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet SN: It is fetched from acq-models github repository. Allowed values: "Awaiting Payment", "Cancelled", "Fully Paid", "Partially Paid", "Payment Not Required", "Pending", "Ongoing" NOTE from A-M: will only display Pending and Payment not required, since those are the only permitted values when an order is created | ||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet Based on value of poline.checkinItems Synchroniz Synchronized = false | ||||||||||
|
| Where does this list of reference values come from? | Question for Thunderjet https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/common/schemas/reference_number_item.json | ||||||||||
|
|
| Question for Thunderjet
| ||||||||||
|
| Does case matter for the supplied value? (e.g USD vs usd vs Usd) | Question for Thunderjet Yes, uppercase is mandatory | ||||||||||
|
| For fund/expense class data, do we have to support fund:expense class from the beginning? Per the SMEs probably not. What do the developers think? OK to leave for POST-NOLANA? We will need to define field mapping syntax to allow for either format | Question for Thunderjet and Folijet OK to wait until refinement feature per Folijet | ||||||||||
|
| Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible | ||||||||||
|
|
| Thunderjet and Folijet Pay attention to this setting, since the Orders app will use it to create/link various inventory record types when the PO is opened SN: It is fetched from acq-models github repository. Allowed values: "Instance, holding, item", "Instance, holdings", "Instance", "None" | ||||||||||
|
| Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other) | Question for Thunderjet SN: We can use CQL language to filter organizations/vendors by code for example like this: /organizations/organizations?query=(((code=AMAZ*))) NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible | ||||||||||
...