2022-12-14 Data Import Subgroup meeting

Recordings are posted Here (2022+) and Here (pre-2022) Slack channel for Q&A, discussion between meetings

Requirements details Here Additional discussion topics in Subgroup parking lot

Attendees: Ann-Marie Breaux (Deactivated) , Kim , Taylor Smith , Lisa Smith Raegan Wiechert Jennifer Eustis leeda.adkins@duke.edu Jennifer Eustis Jeanette Kalchik

Current development (Nolana and Orchid)

Agenda:

- Nolana Bugfest and Bugfixes
  - Met with Performance Task Force to review their Nolana import stats for large files versus Nolana Bugfest import stats. Will be doing additional analysis in the next couple of weeks.
  - Per Jennifer E, 5C was importing 3K, but now doing 600 due to the difficulty of of cleaning up problems; it is not sustainable; can usually load 3 600-record files simultaneously
  - Per Jenn C, they do 5K, but may have larger configurations; also confirms it's problematic to fix things when they go wrong
- Punctuation!
  - https://folio-snapshot.dev.folio.org/inventory/view/f8310c03-f41d-436e-9f77-1d8bf2d5a4bc
  - Ending punctuation for 1xx/7xx names
    - Periods: remove via the default MARC-to-Instance map
    - Commas: do not remove via the default MARC-to-Instance map, which leads to split indexes; OK to start removing as of Orchid?
    - Hyphens: do not remove, and not planning to remove
    - Closing parentheses: do not remove, and not planning to remove
    - Any other ending punctuation for names?
    - Question - how would it be handled if there is an ending period and comma, e.g. Jr.,
    - Other examples:
      - 700 1# $a Ward, Humphrey, $c Mrs.
      - 100 1# $a Seuss, $c Dr.
  - Punctuation on 1xx/7xx $e/$j relator text
    - Periods: cause the term not to be recognized as equivalent to the code
    - Semi-colons: cause the term not to be recognized as equivalent to the code
    - Commas: cause the term not to be recognized as equivalent to the code
- Importing to create POs/POLs
  - We are finalizing the backend work and plan to start connecting everything next sprint
    - PO line limit logic
    - Interaction of Approval required checkbox, Order status, and permissions of user doing the import
      - What if the importer does not have that permission; the order may have already been placed, so maybe it should be created as Open? Different case than manually creating an approval-required order in FOLIO
      - A-M post on Slack; maybe have a separate permission?
    - Interaction of Receiving note and Must acknowledge receiving note checkbox
    - Interaction of the Import job profile and the POL Create inventory fields, especially for 1) Pending orders and 2) P/E mix orders where the POL Create inventory fields have different settings
  - You can view and interact with the order field mapping profile on Snapshot and Snapshot 2
    - You CANNOT import yet, but you can create/edit save Order field mapping profiles
  - Job profile
    - Create inventory will be controlled by the actions in the job profile, not the POL Create inventory data elements; still sorting out some of those details, especially with regards to orders that are created as Pending, then opened in the Orders app
    - Job profile structure
      - MARC modification (optional)
      - Create order
      - Match SRS MARC or Instance (optional)
        Create Instance (always Source = MARC at this time), Update Instance, or Stop
      - Match Inventory Holdings (optional)
        Create Holdings, or Stop (since orders are single holdings, single items, update action unlikely)
      - Match Inventory Items (optional)
        Create Item, or Stop (since orders are single holdings, single items, update action unlikely)
      - MARC modification (optional)
    - If job profile does not contain Inventory actions, order will be created and then the SRS MARC Bib will be ignored (like imported EDIFACT invoice files, or MARC Bibs that only create/update holdings/items)
    - For version 1, if records in the same file need to be on different POs for any reason other than PO line limit, file must be split before import (vendor, fund, account, location, etc)
  - Homework from last week:
    - Before next week, review and test filling in the field mapping profile (NOT importing); any questions or comments about it?
    - Remember that order creation will only work properly for 1 copy POLs for now (until multi support is implemented in Poppy)
    - Mapping of multiple funds for the same POL
    - true/false defaults or MARC values for some of the check boxes
    - A-M to post on Slack DI and Acq channels, and create a feedback document
- DEFERRED Question: posted to DI Slack channel as well
  - Is anyone using MARC 005 or other matchpoint/data element to determine whether or not an existing SRS/Inventory record has been changed, and thus whether an incoming record should update it or not?
    - ETL records vs Records from external sources
    - MARC vs Inventory records
    - Source = MARC Instances vs Source = FOLIO Holdings and Items
  - Leeda/Duke using something to prevent update of the weekly authority bib processing if the records were updated after the extract, and by a user other than the authority vendor; this would be a good example to document/talk through

Upcoming meetings/agenda topics:

No meetings on 21 Dec and 28 Dec 2022
Next meeting: 4 Dec 2023

Chat

From Jennifer Eustis (she/her) to Everyone 01:04 PM
I wish

From Lisa Smith, Mich State to Everyone 01:04 PM
lol

From Leeda Adkins to Everyone 01:05 PM
I'm actually wrapping things between meetings

From Jennifer Eustis (she/her) to Everyone 01:11 PM
Those are just widely different from what we get.

From Jenn Colt to Everyone 01:13 PM
We do 5k but I believe EBSCO has really cranked up our hosting

From Lynne Fors to Everyone 01:20 PM
I'm okay with removing the commas

From Leeda Adkins to Everyone 01:20 PM
agreed

From Jennifer Eustis (she/her) to Everyone 01:22 PM
I'm good with the ending commas being removed

From jeanette kalchik to Everyone 01:22 PM
how would it be handled if both a period and a comma? Like Jr.,

From Lynne Fors to Everyone 01:23 PM
700 1# $a Ward, Humphrey, $c Mrs.

From Jennifer Eustis (she/her) to Everyone 01:24 PM
It is the browse searching

From Lynne Fors to Everyone 01:29 PM
100 1# $a Seuss, $c Dr.

what if there is more than one relator term?

From Jennifer Eustis (she/her) to Everyone 01:32 PM
Sounds good

From Lynne Fors to Everyone 01:33 PM
some records have commas between multiple $e

From Jennifer Eustis (she/her) to Everyone 01:41 PM
We run into that with EDI invoices

From Jenn Colt to Everyone 01:42 PM
We do...

From Taylor Smith to Everyone 02:07 PM
have a good holiday, everyone