Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Prep Jira: Image RemovedUXPROD-3361 - Prep for Importing Orders in MARC format IN PROGRESS (Morning Glory)

Data Import Dev Jira: Image RemovedUXPROD-185 - Import Orders in MARC format DRAFT (Nolana, Folijet)

Acquisitions Support Jira: TBD     (Nolana, ThunderjetData Import Initial Development Jira: 

Jira Legacy
serverSystem JIRA
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyUXPROD-185
 (Nolana/Orchid, Folijet)

Data Import Refinement Jira:

Jira Legacy
serverSystem JIRA
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId01505d01-b853-3c2e-90f1-ee9b165564fc
keyUXPROD-3814
(Scope and release TBD)

Overview slide deck: Click here

...

Rough draft of requirements and focus group meeting notes: Using MARC Bibs and Data Import to create FOLIO POLs Using MARC Bibs and Data Import to create FOLIO POLs

Existing "Lehigh" workaround: Temporary Solution for Creating FOLIO Orders from MARC Data - draft

Problem(s):

  • Some vendors send order-related data in MARC records, so that the receiving library can create orders in their local system from that data. The order data may be sent ay two points in the order workflow
    • Point of order
      • For orders placed on the vendor's ordering system, e.g. firm orders. For these, the vendor does not have a FOLIO POL, since the order is placed in the vendor's system before the order is created in FOLIO (question: would FOLIO need a way for the vendor to send back a POL and have it populate the FOLIO PO/POL fields? Dev preference is no, at least for first iteration, since it would add complexity. Note that this workflow is also covered by the FOLIO Order API process, which is real-time and more efficient than this workflow, at least for creating orders.
      • For selections made in the vendor's ordering system, but not yet ordered. For these pre-order records, the vendor expects to receive back a finalized order from FOLIO, preferably as EDIFACT.
    • Point of receipt/invoicing
      • For materials invoiced automatically by the vendor and purchased by the library. In this flow, orders may only be created if the library wants to track the materials as individual titles and/or pay for the materials as individual titles.
      • Most common materials supplied via this workflow:
        • Approval plan automatic purchases (Physical and/or E) where titles are automatically supplied based on a profile that the library sets up with the vendor. Profiles may be based on subject language, publisher, or combinations of several characteristics.
        • Demand Driven or Patron Driven Acquisitions (Mainly E) where titles are automatically invoiced based on patron usage passing a trigger point that causes titles to be purchased automatically by the library
      • In this flow, there is no opportunity for FOLIO to return a POL to the vendor before invoicing, so the invoice match to the POL much rely on a Vendor Reference Number

...

  • DI PO: Ann-Marie Breaux
  • Acq PO: Dennis Bridges
  • Cornell: Jenn Colt
  • 5C: Jennifer Eustis, Sara Colglazier
  • Michigan State: Kim Wiljanen, Lisa Smith

Use Cases & Requirements:

...

 

Requirement

Status

Notes

Allow users create orders and order lines based on order data supplied in MARC bibliographic records 

Status
subtletrue
colourGreen
titleClosed


Allow users to create field mapping profiles, action profiles, and job profiles to support MARC-based order data

Status
subtletrue
colourGreen
titleclosed


Allow users to create, but not update, POs/POLs via DI

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. No need to consider updating unless significant use case identified in the future
Data import match profiles not needed for Order records

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. Since DI is only creating orders, use case for matching and updating an order does not apply. Use case for matching and then not creating an order: not common, and could only use Product ID or Vendor Ref ID for matching. Revisit if additional use cases identified in the future
Use for one-time orders (physical, electronic, or both). Do not use for ongoing or package orders

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. Revisit if any significant use case identified for Ongoing and/or Package orders
Show all the PO/POL fields in the Field mapping profile, but suppress the PO/POL fields and values that are only used for ongoing or package orders.

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. 
Allow POs to be created as Pending or Open

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs.  Use cases for both identified. Will gather that info via a field at the top of the field mapping profile
Allow override to the tenant's default POL limit setting

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. Also, value may need to vary from one vendor to another or from one type of purchases to another. Will gather that info via a field at the top of the field mapping profile
User should have an option to auto-receive

Status
subtletrue
colourRedYellow
titleIN PROCESSDeferred

Confirmed by Acq SIG SMEs. Will gather that info via a field at the top of the field mapping profile. If PO is created as Pending, and then opened manually, can this requirement be included? Or only possible if the PO is created as Open?PO is created as Open?
Add mapping syntax for TODAY + x days

Status
subtletrue
colourYellow
titleDeferred

Would be useful for fields like Activation date, Receipt date - but per DI SMEs, not important for now. Consider for refinement feature
Honor the Acquisitions unit settings

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. If import user invokes a job profile that creates POs, that user must be a member of the acq unit assigned by the field mapping profile, or the acq unit settings must allow non-members to create POs. This acq unit logic is already in place for Invoices.
Have a way to create POLs for multiple copies with different locations

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. talk with devs about whether the data needs to be configured in a certain way in the MARC file, e.g. 
  • Maybe something like a repeatable 9xx field that has 
  • $a location $b quantity $c fund $d expense class (or $c fund:expense class)
Have a way to create a POL that assigns more than 1 fund and/or expense class to cover the cost

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. See above. Include a way to assign the fund/expense class by % or amount.
Be able to accept fund and expense class whether they are supplied in separate MARC subfields, or in 1 MARC subfield, separated by a colon

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. For example (Fund code = History, Expense class = Electronic)

  • 980 $a History $b Electronic
  • 980 $a History:Electronic
  • Combined fund:expense class 
    Status
    subtletrue
    colourYellow
    titledeferred
    since no library is using yet; aim to include in refinement feature
If Orders app is creating Inventory records based on instruction from Data Import (or based on opening orders), follow the tenant's setting for Disable instance matching

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs
Instances/Holdings/Items related to the created POL should display proper links in the Acquisition and Receiving history accordions, regardless of whether the POL or the Inventory records were created first

Status
subtletrue
colourGreen
titleclosed

Confirmed by Acq SIG SMEs. Works now when order is opened and Inventory is created by the orders app. Does not work if Inventory records are associated with the POL after it has been opened. Needs further discussion with devs
If a library wants to import a MARC Bib file to create orders, but does NOT want to create any Inventory records, will there be any problem?

Status
subtletrue
colourBlueYellow
titleopendeferred

May just require testing of a job profile with Create orders action, but no Create Instance action.
If a library has an Order setting that POs must be approved before opening, will that setting always apply to orders created by Data Import? If yes, need an option to always create the orders as approved, or always create as pending?

Status
subtletrue
colourRedGreen
titlein processclosed

Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M to ask on Acq Slack channelthat setting. A-M asked on Acq Slack channel. If field mapping has approved selected, the user importing the file must have permission to approve orders.
If a library has an Order setting that allows for the FOLIO PO to be edited, will that setting apply to orders created by Data Import?

Status
subtletrue
colourRedGreen
titlein processclosed

Discussed in Acq SIG 28 June 2022; no libraries with that setting. A-M to ask asked on Acq Slack channel. Does not seem to be needed currently
Will libraries want to map PO numbers from the incoming MARC records?

Status
subtletrue
colourRedGreen
titlein processclosed

Discussed in Acq SIG 28 June 2022.  No use case identified (yet) - wait and see if any identified before definite "No", since this decision would require coordination with Settings for editing PO numbers and max POLs per PO. If PO needs to not be mappable, easiest to disallow mapping the field completely on the field mapping screen (like HRID). A-M to ask on Acq Slack channel.
If the PO number is mapped from a MARC field (e.g. 950$p) and the PO varies from one MARC Bib to another, what should FOLIO do?

Status
subtletrue
colourBlueGreen
titleopenclosed

Wait until above question is resolved. Will not be mapped, so does not need to be dealt with

Discussed, but not requirements

  • Create separate orders per invoice. FOLIO does not currently have the logic to parse MARC files into separate invoices. If this is required, library should work with their materials vendor to get separate MARC files per invoice, and then ensure that the POL line limit in the field matting profile is set larger than the maximum number of lines allowed per invoice from that vendor.
  • Be able to select an order template within the Order field mapping profile:  The Data Import field mapping profile basically acts as a template, gathering default and mapped data for a particular type of file from a particular vendor. Once the field mapping profile is created, it does not have to be adjusted to accommodate each individual import for the same type of file. To tie the field mapping profile to the order templates would require significant development effort, along with problem handling if the field mapping profile and template have different values for the same field.
  • Create invoices from the MARC bib data. Separate feature for that. Cannot be accommodated in the scope of Nolana. For now, create orders based on MARC Bibs, and then create invoices from EDFACT data, via a separate job profile. 
  • Assign order notes to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
  • Assign tags to POs and POLs. Not a requirement in Nolana. Currently Data Import does not interact with any helper apps such as Notes or Tags
  • Schedule the pickup and import of MARC Bib files, assigning the correct job profile. Not a requirement in Nolana. Added some details into UXPROD-662

Mockup for FOLIO PO/POL Field Mapping Screen

See requirement details in the screens attached to UIDATIMP-294

Sample MARC records with Order data

...

Question

Status

Conclusion

Comments

If order is created as Open, 1) will the receiving pieces be created and 2) will they show in the Inventory holdings record?

Status
subtletrue
colourBlueGreen
titleOpenclosed

For initial implementation, receiving pieces, holdings, and items will be created by the order being opened


Developer Questions:

Opening orders

...

Possible Nolana Job profile sequence (Data Import) 

See video here

  • Create Instance (source = MARC)
  • Create Orders (with data from MARC Bib, Order data in MARC Bib, Order field mapping profile)
    • DI would need to pass the Instance UUID as part of the Create orders info
    • What happens if order is created as Pending?
    • How to create source = MARC
  • Open the order
    • Which creates Holdings and Item based on info the POL knows, and provides the POL link in the Holdings and Item
    • Question about whether opening the PO creates a new holdings or links to an existing holdings in the Instance

...

Question

Status

Conclusion

Comments

How to handle the POL limit variation (in the field mapping profile, users can select to use the existing default or override to a number higher or lower than the default)

Status
subtletrue
colourBlueGreen
titleOpenclosed

See logic in field mapping profile - display the existing POL limit setting, but allow override of it
If DI only allows creation of one-time, non-package orders, will there be any problems if we remove all the non-one-time, non-package data elements from the field mapping screen?

Status
subtletrue
colourBlueGreen
titleOpenclosed

Not a problem
We want to use the existing Orders logic as much as possible, but we also want Instance/Holdings/Item linked with correct POL even when created before the POL.

Status
subtletrue
colourBlue
titleOpen

How much extra logic would be needed? 

Green
titleclosed

For initial release, create the Instance first via DI, then have the order's opening create the holdings and items. Then update the holdings and item via a second DI job. In the refinement release, aim to combine the above into 1 job
Return an error to DI log if the user does not have permission to create an order for the acq unit specified in the PO

Status
subtletrue
colourBlue
titleOpen


Already in place for invoices. Can we reuse that work?
If vendor supplies order date in YYYYMMDD format, can we convert it to FOLIO YYYY-MM-DD default format? 

Status
subtletrue
colourBlue
titleOpen



Would we be able to support other order date formats?

Status
subtletrue
colourBlue
titleOpen



Test handling of multiple copies for multiple locations in the same POL; how can DI understand and parse properly? 

Status
subtletrue
colourBlueYellow
titleOpen

POST-NOLANA

deferred

Work on this after the multi-copy DI work in Orchid. Initial implementation will be single copy/single locationMARC field mapping syntax expansion required? Aim for repeatable 9xx field that has $a location code $b quantity maybe $c fund $d expense class?
For prices, FOLIO always wants an explicit decimal (e.g. 25.15, not 2516, correct?)

Status
subtletrue
colourBlueGreen
titleOpenclosed

Question for ThunderjetYes
With invoices, data import does not try to correlate the expense classes allowed by particular funds (just display all the expense classes from Settings in a dropdown list). Should we refine that for Orders? How difficult would it be?

Status
subtletrue
colourGreen
titleCLOSED

Per SMEs. no need to refine, at least to start with - show all funds and all expense classes in each dropdown.
Would Folijet need to take the receiving app into account? What happens if the Inventory records are created before the Order records, and not triggered by the POL's Inventory setting - would that be an issue?

Status
subtletrue
colourBlue
titleOpen

Question for Dennis/Thunderjet

Green
titleclosed

Not in initial release, since the receiving pieces will be created by opening the order, not by DI
If order is created as Pending, but Instance/Holdings/Item has already been created by DI, what happens when the order is opened manually?

Status
Open
subtletrue
colourBlueGreen
titleclosed

Initial release will be set up so that the Instance is created first, and then the POL linked to it. Then the holdings/items will be linked to the POL via the standard Orders module logic, when the PO is opened. SMEs want the Instance/Holdings/Item to be linked to the appropriate POL on the PO that was manually opened
If library wants no Inventory created, can job profile be set up to import MARC Bibs, but only create orders?

Status
Open
subtletrue
colourBlueYellow
titledeferred

Not in the initial release. Investigate as a possibility in the refinement releaseMay just take some testing 
From the SMEs: 

Can we assume that actions taken by the system are being done the same way as actions done by a user (except faster, and without any confirmation modals)?

Status
subtletrue
colourBlueGreen
titleOpenCLOSED


Question for Tjet/Fjet

Yes. Currently there is just a small difference in processing requests if poline.source == "EBSCONET", which  was done for the specific 3rd party system, and it will not affect other flows.

Specific PO/POL data elements 

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

Question for Thunderjet

  • Vendor

Status
subtletrue
colourBlueGreen
titleOpenclosed

Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other)

Question for Thunderjet

SN: We can use CQL language to filter organizations/vendors by code for example like this:

/organizations/organizations?query=(((code=AMAZ*)))

NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible

  • Created by

Status
subtletrue
colourBlueGreen
titleOpenclosed

What data should be supplied - user who runs the import job?

Question for Thunderjet

SN: Yes, correct, active signed user. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. 

  • Created on

Status
subtletrue
colourBlueGreen
titleOpenclosed

What data should be supplied - current date/time? Will it default if not supplied?

Question for Thunderjet

SN: Yes, current date/time. It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client.

  • Approved by

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

What data should be supplied - user who runs the import job?Question for Thunderjet

Yes, and  this value will be populated automatically if approval required by tenant configuration
  • Approval date

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

What data should be supplied - current date/time? Will it default if not supplied?

Question for

Thunderjet

Thunderjet

Will be populated automatically with current date if approval required by tenant configuration

  • Date ordered

Status
subtletrue
colourBlueGreen
titleOpenclosed

How is this value determined? By when the order is opened? Must it be supplied, or will it default if not supplied?

Question for Thunderjet

SN: Yes, it is set when order is opened, the defaut default value will be supplied as current date/time 

Link to code

  • Product ID and Qualifier

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

What to do if qualifier in same field as number, e.g. 020$a12345678 (paperback)? Does DI need logic to break apart or can Orders app do it?

Question for Thunderjet

According to acquisition data model productId and qualifier live in different places, so it needs to be splitted before making order request

https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/mod-orders-storage/schemas/product_identifier.json

  • Product identifier type

Status
subtletrue
colourBlueGreen
titleOpenclosed

This is a subset of a tenant's https://folio-snapshot.dev.folio.org/settings/inventory/identifierTypes. How do we know which ones to show in the dropdown list?

Question for Thunderjet

SN: In Orders UI For displaying product identifier types we use this query:

/identifier-types?limit=1000&query=cql.allRecords%3D1%20sortby%20name

So we use all available identifier types

  • Order format

Status
subtletrue
colourBlueGreen
titleOpenclosed

Where does this list of reference values come from?

Question for Thunderjet

SN: It is fetched from acq-models github repository.

Allowed values:

"Electronic Resource",
"P/E Mix",
"Physical Resource",
"Other"

Link to code

  • POL Created on

Status
subtletrue
colourBlueGreen
titleOpenclosed

What data should be supplied - current date/time? Will it default if not supplied?

Question for Thunderjet

SN: It is set automatically by RMB when Verticle receives request. So it is populated by server, should not be populated by the client. Example of metadata payload when creating order with populated by RMB data. 

  • Receipt status

Status
subtletrue
colourBlueGreen
titleOpenclosed

Where does this list of reference values come from?

Question for Thunderjet

SN: It is fetched from acq-models github repository.

Allowed values:

"Awaiting Receipt",
"Cancelled",
"Fully Received",
"Partially Received",
"Pending",
"Receipt Not Required",
"Ongoing"
Link to code Receipt", "Cancelled", "Fully Received", "Partially Received", "Pending", "Receipt Not Required", "Ongoing"

Link to code

NOTE from A-M: will only display Pending and Receipt not required, since those are the only permitted values when an order is created
  • Payment status

Status
subtletrue
colourBlueGreen
titleOpenclosed

Where does this list of reference values come from?

Question for Thunderjet

SN: It is fetched from acq-models github repository.

Allowed values:

"Awaiting Payment",
"Cancelled",
"Fully Paid",
"Partially Paid",
"Payment Not Required",
"Pending",
"Ongoing"

Link to codeto code

NOTE from A-M: will only display Pending and Payment not required, since those are the only permitted values when an order is created

  • Receiving workflow

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

Where does this list of reference values come from?

Question for Thunderjet

Based on value of poline.checkinItems

Synchroniz

Synchronized = false
Independent = trued = false
Independent = true
  • Vendor reference number type

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

Where does this list of reference values come from?

Question for Thunderjet

https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/common/schemas/reference_number_item.json
  • Account number


Status
subtletrue
colourBlueGreen
titleOpenCLOSED
  1. How to get the list of account numbers to use for a dropdown list for a particular vendor?
  2. If the library transcribed an account number from an incoming record, as long as it matches an account number associated with the selected vendor, would that be acceptable? 
  3. If there are no account numbers associated with a vendor, but an account number is supplied for the POL, will that trigger an error in the POL?
Question for Thunderjet

Question for Thunderjet

  1. account numbers are part of organization records https://github.com/folio-org/acq-models/blob/de612d54f94bca8eab6c637f7a7dcb67916b4ba7/mod-orgs/examples/organization_get.sample#L152
  2. -
  3. There is no validation of account numbers. The order can be created with not existing account number, but this can cause issues in future. For example in edifact export flow
  • Currency

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

Does case matter for the supplied value? (e.g USD vs usd vs Usd)

Question for Thunderjet

Yes, uppercase is mandatory

  • Fund/expense class

Status
subtletrue
colourBlueYellow
titleOpendeferred

For fund/expense class data, do we have to support fund:expense class from the beginning? Per the SMEs probably not. What do the developers think? OK to leave for POST-NOLANA? We will need to define field mapping syntax to allow for either format

Question for Thunderjet and Folijet

OK to wait until refinement feature per Folijet

  • Material supplier

Status
subtletrue
colourBlueGreen
titleOpenCLOSED

Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other)Question for Thunderjetor code? (an exact match on one or the other)

Question for Thunderjet

SN: We can use CQL language to filter organizations/vendors by code for example like this:

/organizations/organizations?query=(((code=AMAZ*)))

NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible

  • Create inventory

Status
subtletrue
colourBlueGreen
titleOpenclosed

  1. Where does this list of reference values come from?
  2. Pay attention to this setting, or use the job profile sequencing instead?

Thunderjet and Folijet

Pay attention to this setting, since the Orders app will use it to create/link various inventory record types when the PO is opened

SN: It is fetched from acq-models github repository.

Allowed values:

"Instance, Holdingholding, Itemitem",
"Instance, Holdingholdings",
"Instance",
"None"

Link to code

  • Access provider

Status
subtletrue
colourBlueGreen
titleOpenclosed

Is there an endpoint where DI could do a lookup based on the vendor name or code? (an exact match on one or the other)

Question for Thunderjet

SN: We can use CQL language to filter organizations/vendors by code for example like this:

/organizations/organizations?query=(((code=AMAZ*)))

NOTE FROM A-M: Should be an exact match for the code, NOT a "begins" search, but case-insensitive, if possible