Using MARC Bibs and Data Import to create FOLIO POLs
Focus Group for drafting requirements:
- DI PO: Ann-Marie Breaux
- Acq PO: Dennis Bridges
- Cornell: Jenn Colt
- 5C: Jennifer Eustis, Sara Colglazier
- Michigan State: Kim Wiljanen, Lisa Smith
20 May 2022: A-M, Kim, Sara, Jenn, Dennis
2 June 2022: Kim, Lisa, A-M, Dennis, Jennifer, Jenn
Prep Jira: UXPROD-3361 - Getting issue details... STATUS (Morning Glory)
Dev Jra: UXPROD-185 - Getting issue details... STATUS (Nolana)
Prep work
- Please bring samples of MARC Bibs with 9xx order data and be familiar with placing orders in your library’s FOLIO
Background
- What are your library’s use cases for creating orders from MARC Bib records?
- Approval plans (aka All happens at point of receipt/invoicing)
- Create individual orders for each title, along with Bib/Instances, Holdings, maybe Items - all at the same time
- DDA automatic eBook purchases - may not be creating items for these
- Firm orders (aka Point of order and Point of receipt/invoicing)
- 2 styles, depending on where the user presses the Order button:
- Order in vendor system: usually have to create a copy of that order in FOLIO (either via API, MARC data, or manually); usually the subsequent matching to update Inventory and link Invoices to PO Lines is done via vendor reference number, since the Vendor will not have the FOLIO POL (unless it is returned to the vendor after order creation in FOLIO)
- Order in FOLIO: Order is sent to the vendor in various ways; usually the subsequent matching to update Inventory and link Invoices to PO Lines is done via POL, although it may be via VRN if the library stored the VRN in the POL they created in FOLIO
- Firm orders that are not API - create individual orders/temp Inventory Instances, Holdings, Items, then later finalize Bib/Instance, Holdings, Item
- Lehigh tool workflow from Cornell (which uses MARC Bib data along with the the order API and Data Import)
- Kim: Get records from OCLC and download to local system to create an Instance and Order (ca. 80-90 records per session), but not holdings or item. Holdings and item created when the materials are processed, which may only be a couple days after the orders were created. (EastView for Russian materials, MARCnow does cataloging and adds to OCLC; library gets records from OCLC)
- 2 styles, depending on where the user presses the Order button:
- Multi-line POs instead of Single-line POs (Cornell and Mich State)
- Mich State likes 1 POL per PO for firm orders, but approvals are multi-line; settings are 999, which works for AP, but not for firm orders
- Approval plans (aka All happens at point of receipt/invoicing)
- Walk through one library’s details of the top use case (Approvals)
Approvals: Cornell Aux Amateurs
- Separate MARC file for each invoice; POL limit setting is 999
- Holdings, Item, Invoice data in 9xx fields
- Create the order as Open
- Be sure that the VRN is mapped into the POL
- Inventory interaction for the POL: Create Instance, Holdings, Item
- When the order is opened,
- Orders app creates brief Instance, Holdings, Item
- Creates expected receiving pieces, with synchronized receiving
- Receiving pieces are created
- Cornell does minimal receiving for approvals; just click received and do not include barcode number or other item data
- Holdings and Item will be updated separately via Data Import
- Library could also default to receipt not required, but
- Lose some visual clues in that case, that the POL was fully received (which Michigan State observed)
- Receiving Piece would not be linked to the holdings or item and would not populate the Receiving history accordion
- Receiving pieces are created
- Encumbers the funds
- Lehigh app triggers Single record import that
- Overlays each of the Instances
- Changes Source to MARC
- Updates the Holdings and Item
- Import stops if Instance Source = MARC already, so Library can review
- If no Lehigh app, and there is a file of 8 MARC Bibs
- Import the MARC file
- Matching on VRN for Instance, Holdings, Item
- Updating Instance, Holdings, Item based on the update field mapping profiles
- Approval Q&A:
- Template needed? Probably not, since DI field mapping profile acts similarly; invoking a template might be overly-complicated
- Do libraries add extra lines to the approval POL manually? Usually no
- Allow the POs to be created as Pending or Open, since some libraries may want to review before opening them
- Also make sure that Receiving option is in the Field mapping profile, e.g. Receipt not required
- Does the Orders app have to create brief Instance, Holdings, Item, or could DI do it? If DI does it, is there a way to link the POL to the Inventory records? A-M and Dennis discuss
Firm orders
Placed outside of FOLIO and then need to be copied into FOLIO - can't count on the vendor having FOLIO POL
- Ways these could get into FOLIO:
- API
- Lehigh app (what is the new name)
- MARC Bibs with order data (after this dev work is done)
- Manual (after order is sent to vendor by e-mail, mail, or placed on their site, if there is no automated way to get the data back into FOLIO)
- Sara
- Point of order
- Place orders on vendor site
- Get daily MARC file (from vendor, with OCLC numbers and Vendor reference number)
- Pick up the file (FTP)
- Get a daily e-mail that the file was picked up and loaded
- Imported MARC file creates bib, item, order in current system (does not create the holdings yet - this is in their former system, not FOLIO)
- Order created in open status
- Point of receipt
- All matches by Vendor Ref Number (VRN)
- MARC file to update the bib
- Manual receiving (could FOLIO automate?)
- Manual create holdings and update item (could FOLIO automate?)
- EDIFACT invoice
- Point of order
- Ways these could get into FOLIO:
- FOLIO Approval workflow job profile
- Create order - figure out where receiving fits
- Rely on the POL setting for which inventory records are created
- Rely on the Tenant setting for whether it matches to Instance or not (what if library wants to override?)
- Rely on the Tenant setting for max lines on PO (what if library wants to override?)
- Field mapping profile controls whether order is created as Pending or Open
- Open the PO
- Creates receiving pieces
- Maybe option to immediately receive or Create as received? (sets the receipt date and updates item status)
- Creates/matches to Instance (source = FOLIO, if create)
- Creates holdings
- Creates item
- Instance/Holdings/Item all have VRN, and are linked to the POL that was just created
- Creates receiving pieces
- Meanwhile, back in DI land - FOLIO has complete info for the Instance, Holdings, Item
- Update instance, matching on VRN - changes to Source = MARC
- Need to keep this action from happening until Orders app confirms that the Order has been created and opened, so that DI can be sure that the Inventory records with links to POL are in place
- Update holdings, matching on VRN
- Update item, matching on VRN
- Create order - figure out where receiving fits
Placed in FOLIO and sent to the vendor - vendor would have the POL
- How?
- May be creating pre-orders from MARC Bibs, and then finalize and send to vendor (after this dev work is done)
- Manually key
- Jenn/Cornell
- Point of order - Cornell uses external selection tool to gather requests and recommendations from selectors and vendors
- Selectors may decide on candidate titles, which triggers MARC to be sent
- Vendor sends MARC Bibs for candidate titles
- Other records come from other vendors, and may be selected at this point - all is external to FOLIO
- Automation to check the file, maybe change vendor, assign funds
- Produces a MARC record with order data
- Weird ones go into review file
- Then pre-order MARC file is loaded into FOLIO - using the Lehigh app - all settings configured in the tool, so not using order templates
- Creates the order as Open
- When the order is opened, does its regular process: create the receiving pieces, Inventory instances, holdings, items
- Then run a data import to create SRS MARC, change Instance source, update the holdings with call number ON ORDER, no changes to item
- Only for 1 vendor automated; rest are manual
- Export the order info in csv and send to vendor
- Vendor creates orders, with the FOLIO POL
- Point of receipt
- Get a MARC file from the vendor
- Add the Instance HRID into the MARC, and then use that for matching
- Use "On Order" in the holdings call number field as a static match to update the correct holdings (or holdings HRID begins 1)
- Use "On Order" or "In process" item status to find the correct item and update it
- Updates the Instance, Holdings, Item (call number, location, barcode)
- EDI invoicing, probably using FOLIO POL for matchpoint
- Point of order - Cornell uses external selection tool to gather requests and recommendations from selectors and vendors
- How?
- A-M create videos of matching by POL/VRN and updating related Instance/Holdings/Item (delivered in Morning Glory)
Possible Order field mapping UI
- At top
- Use template or not - what happens if there is a disconnect between the template and the import field mapping profile
- Use settings POL max or not (system level setting)
- Create orders as Pending or Open
- Option to auto-receive or not (just click the receive button); rest of the holdings/item updates will be handled via import
- Override the system-level setting for trying to match to instance? (would always create new instance, holdings, item)
- Then all the template data
Things we don't want to forget/Misc Info and Q&A
- All 3 Inventory record types linked to POL as of Kiwi, but can only see the POLs in holdings; Lotus: can see the POL links on all 3 Inventory record types
- What happens with POL links to holdings/items before the POL in holdings was implemented? Are they there retroactively? Is there a script that the library can run to identify them and populate them?
- Does POL know about Holdings it's associated with? No for Juniper and before; Yes for Kiwi and after; currently no request for any way to marry pre-Kiwi POLs and Holdings with each other
- Does POL know about Item it's associated with? Yes, even though it didn't show in the UI until Lotus
- Does POL know about Instance it's associated with? Yes, even though it didn't show in the UI until Lotus
- What happens with POL links to holdings/items before the POL in holdings was implemented? Are they there retroactively? Is there a script that the library can run to identify them and populate them?
- Be able to assign a template as part of Order field mapping profile or no?
- Maybe start with a default profile that users can copy and adapt, similar to the invoice field mapping profiles
- Including the Order templates would create more complexity, since we would need to deal with restricting edit of the related field mapping fields, or deal with template/field mapping conflicts for specific fields
- Decision: Not at this time, since the field mapping profile basically acts as a template
- POL maximum setting OK to follow? Maybe NO, if library does multi-line for APs and single-line for FOs or vice versa; have a way to override the default in the Order field mapping profile?
- If set it as part of the DI profile, then will have problems if library tries to unopen/reopen the order; maybe just document this; if order can be created as Pending, maybe there would be less need to unopen/reopen order?
- Would libraries ever want different POL maximums per import job? If so, probably easier to store the value in the DI field mapping profile instead of Order settings
- Decision: Allow POL max to vary per field mapping profile (i.e. per job profile); otherwise, follow the settings default
- Can we confirm that things done manually happen the same way when they are done mechanically?
- Ask the developers for some sort of language? Is the system basically doing the same thing as a person, only faster? Need to allow for the user to make some decisions, and edit
- Work through scenarios where multiple libraries share the same instance, but separate holdings and items - 5 Colleges
- Does Inventory creation happen via Orders app (Instances would be Source = FOLIO) or Data Import app (Instances would be Source = MARC)? If DI, can we trigger POL links to be created?
- Via Opening Orders
- Order generates automatically if there's not already a reference in the POL
- Will connect to existing instance, if matches on Product ID
- May or may not match to an existing holdings
- If no Holdings ID in the POL (has location ID only), it creates a new holdings
- If Holdings ID in the POL, it matches to existing holdings (but the POL has to know about the holdings before the PO is opened)
- Always creates a new item
- During receiving, can correct the linkage of the item from an incorrect or duplicate holdings to the correct one, and delete the holdings that no longer has an item
- Approvals doesn't know POL, but knows VRN
- Order generates automatically if there's not already a reference in the POL
- Via Data Import
- Hard to get the proper links in place
- Maybe use the match capabilities of DI Match profile, e.g. from POL to Instance field or to MARC field
- What happens if vendor cannot provide the POL or VRN in the MARC Bib, or the OCLC number/ISBN is not a good match
- Via Opening Orders
- Could orders be created and automatically transition to received, especially if DI is populating the details like barcode into the Item record? Would be helpful
- Should Inventory be created first, and then the POLs? Or vice versa
- Option 1
- SRS MARC Bib
- Instance
- Holdings
- Item
- Order/Order line - Open order
- Matches to the Inventory record types (instead of creating new) 2 locations, 4 copies - could Opening the order handle the extra holdings and items until DI can?
- Receiving pieces
- Option 2
- Order/Order line - Opening order
- Receiving pieces
- Instance
- SRS MARC Bib
- Holdings
- Item
- Order/Order line - Opening order
- Option 3 - So far seems to be the best, but still evolving
- SRS MARC Bib
- Instance
- Order/Order line - Opening order
- Match to Instance - precoordinate with Instance UUID
- Holdings
- Item
- Receiving pieces (with option to receive all)
- FOLIO takes care of updating pieces, instead of an explicit import step?
- Update holdings
- Update item
- Does DI need to create pieces? Not if the orders app does its work first
- Only 1 holdings/1 item can be updated by DI right now; what would have to change in the above workflows when DI can support multiples?
- UXPROD-2741 - Getting issue details... STATUS targeted for Orchid
- Fund and Expense class - do we need special field mapping syntax to split if they come from the vendor as 1 value?
- DI field mapping needs to support combined and separate,
- e.g. 980 $b HIST:ELEC <=== Talk with Ruslan about best mapping syntax for splitting this combined info into 2 POL fields
- or 980 $bHIST $cELEC
- DI field mapping needs to support combined and separate,
- If library uses 1 order location (e.g. Chicago always uses "Order" location), and then change to a different permanent location
- If POL is connected to the holdings, and Perm Loc in holdings is changed - does not affect the order
- If POL has a location instead of holdings, and the holdings Perm Loc is changed - does not affect the order
- Do some testing with this to see if any breakdown
- How does this dovetail with the Lehigh app - does this replace it or coexist with it?
- Not necessary to replace the Lehigh app, but Data Import needs to support this basic functionality as well
- At some point, review the Lehigh app customizations to see if they suggest any additional functionality that the DI Orders feature would need to support
- Would a library ever want to create POLs from the incoming MARC, but not want to create Instances at all? (similar to EDIFACT invoices creating invoices but then not being linked to any Inventory record)
- Yes. Sometimes get a spreadsheet of eContent
- Would want an individual POL (maybe all on same PO or maybe each on separate PO) for each spreadsheet row, but would NOT want any Inventory content; all the individual titles are being tracked through eHoldings/KB
- Question for devs: can job profile be set up to just create orders, and no Inventory?
- Similar to the DI features for
- Import of delimited data
- Import MARC Bibs to create/update Inventory, but do not link to SRS records
- Yes. Sometimes get a spreadsheet of eContent
- Ongoing orders - how necessary or not is MARC Order 9xx data?
- Decision: For first iteration, assume no
- Journals - don't need Inventory created
- New orders are infrequent enough that we can set aside for the first iteration
- Big packages
- Usually 1 POL (unless different funds, perhaps)
- Pay all on the main POL
- Have lots of new titles all at once
- Manage titles through discovery service package info
- If MARC records for them, maybe put the POL into the Instance/Holdings as a default Admin note? Put in as a local identifier? Also want the Inventory record(s) linked to the Agreement; also need Inventory Containers
- Titles move from one package to another
- Usually 1 POL (unless different funds, perhaps)
- For POs/POLs, do we need update, or just create?
- Decision: Start with create, then see if any compelling use case for update
- If only create, would match profile ever be needed? (Match allows you to find an existing record and update it OR determine there is no existing record and create it)
- What about PO/POL notes (using the notes app) - support import for that data or no?
- Decision: Not yet (current Data Import does not interact with the Notes app)
- What about tags?
- Decision: Not yet? (current Data Import does not interact with the Tags app)
Technical/Dev questions/issues:
- How to handle PO line limit variation?
- Return an error to DI log if user does not have permission to create orders for acq unit specified in the field mapping
- How to handle Fund/Expense info sometimes being sent by vendor in one field (e.g. History:Electronic) and sometimes 2 fields?
- For Invoices, import does not try to align which expense classes are allowed with which funds. Do we need to change that for orders, or just discard and return error?
- Would it be more work or less work to limit the DI Imports to one time orders only? (which means hiding/turning off fields related to ongoing orders)
- For Instance, Holdings, Item to be linked to the POL, they must be created by opening the PO. If they were created by DI instead, would it be possible to link them to the POL in a different way?
- Test handling of multiple copies for multiple locations in the same POL; how can DI understand? Aim for repeatable 9xx field that has $a location code $b quantity maybe $c fund $d expense class?
- Can we assume that actions taken by the system are being done the same way as actions done by a user (except faster, and without any confirmation modals)?
- If order created as pending, but Instance/Holdings/Item has already been created by DI, what happens when the order is opened?
- If library wants no Inventory created, can job profile be set up to import MARC Bibs, but only create orders?