Page Comparison

We want to ensure that MARC-MARC matching works properly for repeatable and non-repeatable fields, especially 0XX/9XX fields, and that they can pair well with Inventory submatches.

In scope:

• MARC-MARC matches that result in multiple possible hits can be narrowed to single records with MARC-Inventory or static value submatches

• Review any existing matching bugs and plan to resolve as part of this feature

Out of scope:

• After a MARC-MARC or MARC-Instance match, a user can include both Instance and MARC Bib actions afterwards (need examples from users)

• Confirm that MARC matches are working properly using indicator wildcards (asterisks) versus blanks

• Currently ISBN matches do not translate 10-digit and 13-digit so that they can be matched against each other, Include in this feature, or handle as a separate feature in the future? See MODSOURMAN-269

• Should we add a bug for not being able to have an override action for field protections under a MARC-Instance match?

• What else?

Use case(s):

• SMEs: Please add examples*
   * MARC-MARC match on OCLC number and then submatch by Instance status

• MARC-MARC match on 001 and then submatch for holdings by permanent location

• Need a use case that results in multiple SRS hits that then need to be narrowed down by Inventory match

Steps to Reproduce:

1. Log into any environment on Orchid or Nolana.

2. Identify any record in inventory and export the MARC record.

3. Add two exact duplicate 856 fields to the record.

   1. Exact duplicates including indicators and subfields.  Copy & paste.

4. Import using a simple overlay job profile matching on a subfield of the 999 to update the SRS.  Do not use MARC modifications.

Expected Results: The SRS record contains duplicate 856 fields.
Actual Results: The SRS record contains only one of the duplicate 856 fields.
**

Additional Information:  We know that Data Import does not de-dupe the 903 field, for example, during an update but it does the 856 field.  Data Import jobs which create new SRS records includes the duplicate 856 fields.  This raises several questions:

• Why does the system de-dupe during an update without explicit instructions from the user?  

• Which fields does the system de-dupe and when?

• Is this a bug?  Or by design?  

From testing, there appears to be no difference between de-duping of the 856 when field protections are applied or not.

Steps to Reproduce:

1. Log into Morning Glory bugfest

2. Go to inventory and select import using the "OCLC with MARC modifications" option and import OCLC number 31934425

3. The record should import successfully and the title will have received an obvious MARC modification

4. Attempt to overlay the same record with the same OCLC number.

Expected Results:

Overlaying the record works. The MARC is modified as described in the profile and the instance and SRS are updated.

Actual Results:

The instance is not updated. A modified SRS record is created but still has the original OCLC 001 and 003. QuickMARC will not work on the record.

Additional Information:
URL:  Update profile: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/1c351fa4-d578-434f-a02a-7ff46af16f06?query=single&sort=name

An example of an instance with this issue: https://bugfest-mg.int.aws.folio.org/inventory/view/53e28701-dccc-49be-a01d-9adaa15f4cb6?query=neuromancer&sort=title&xidtype=0dd718cf-a09a-4f1c-be6a-0cf0de58b424

Job profiles

• Morning Glory

  • ISRI Create: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/d8f1c585-b56a-4024-b82a-930be5569396?query=jenn%20single&sort=name

  • ISRI Update: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/1c351fa4-d578-434f-a02a-7ff46af16f06?query=jenn%20single&sort=name

• Nolana

  • ISRI Create: https://bugfest-nolana.int.aws.folio.org/settings/data-import/job-profiles/view/f44ee72c-f396-4742-b16d-3393c78623ca?query=jenn%20single&sort=name

    • Job completed: https://bugfest-nolana.int.aws.folio.org/data-import/job-summary/be62c707-4c8e-4a75-baed-b640df515d14

  • ISRI Update: https://bugfest-nolana.int.aws.folio.org/settings/data-import/job-profiles/view/127f27f8-5711-43fb-8114-393f6bb478f1?query=jenn%20single&sort=name

    • Job completed with errors (no error message): https://bugfest-nolana.int.aws.folio.org/data-import/job-summary/aacede86-7b50-449c-9d95-c5dbfb91c0d6

• Orchid

  • ISRI Create: https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/29cf9029-8e70-40ee-a972-fb35a11b257b?query=jenn%20single&sort=name

    • Job completed (but 2 entries in the log, 1 with title before MARC MOD, and 1 with title after MARC MOD): https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/829f4a43-5dec-41a7-8bae-d94ef941ebe7

  • ISRI Update: https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/4447c1dc-39ff-45cf-8b15-74dde79869dd?query=jenn%20single&sort=name

    • Job completed with errors (no error message): https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/3731a505-0f79-4506-ba80-12cdf3e9c3fc

BE Notes:

• First Modify action saves the record in SRS, subsequent Match on MARC Bib in SRS returns multiple match error, because it finds not only the original record, but also the one that was just saved.

• During modify action post-processing an attempt appears to change instance hrid because of incoming record doesn't contain actual instance hrid in 001
  that causes error on instance update.

Note for QAs:
When this bug is fixed, create a new TestRail for modification, followed by match, followed by update action, as outlined in the repro steps above. Then unlink this Jira from TestRail C350914

Steps to Reproduce:

1. Log into some FOLIO snapshot environment 

2. Go to Settings/Data Import

3. Duplicate the field mappings for Inventory Single Record - Default for the update marc bib and update instance

4. Create a new field mapping for a marc modification where the fields 029, 506, 856, 583 are deleted

5. Create action profiles linked to the field mappings you just created

6. Duplicate the inventory single record match for no srs and existing srs

7. In field protections, make sure that you add 583 with subfield 5 and data MU in addition to 856 with subfield 5 and data MU.

8. Create a job with a match to existing marc srs. On matches update marc srs. On no matches, another match to no srs and then under that update instance. Then at the level of the 1st match, add the modify marc action (see screenshot)

9. Bring in a record and add a 583 with a $5 MU and a 856 $5 MU

10. Overlay

Expected Results: The existing and protected fields should remain in the record.

Actual Results: The protected fields have been removed.

Additional Information:
URL: The job profile on orchid bugfest is https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/b590fc78-f069-42a1-bfdd-3988c7d6be00?query=fc%20test&sort=name. 

Steps to Reproduce:

1. Log into bugfest-Orchid 

2. Open Inventory and search for an instance (random records were chosen and the behavior was duplicated each time) 

3. From Actions, choose 'Overlay source bibliographic record':

   1. External target = OCLC WorldCat

   2. Profile = Inventory single record - default update instance (Default)

   3. Enter the OCLC# of the instance currently being viewed

4. Click 'Import'

Expected Results: The bibliographic record and instance are updated with the latest OCLC version.
Actual Results: The bibliographic record and instance are updated but with duplicate 035 fields for those OCLC#s with a prefix.  
Additional Information: Testing shows that this happens on the initial overlay, but subsequent overlays do not continue to 'add' duplicates.  

Records tested in bugfest-orchid: in523951 and in2486915 (screenshots of before overlay and after are attached) 

Duplicate data causes issues with integrations and other functions that rely on the OCLC# as a match point.  

However, when I went to add the Match profile (Inventory Single Record - Default match for existing SRS record) which is a system generated profile, it did not appear as one of the options. When I look at the Action profiles, I am able to choose system generated ones, so the problem is only with the Match profiles. We are on Orchid-SP-5. I was also able to recreate the issue on Snapshot. (I don't have Bugfest access so I did not try that.)

Steps to Reproduce:

1. Log into FOLIO Snapshot

2. Go to Settings–>Data Import–>Job Profiles–>Actions–>New Job Profile

3. Under Overview, click on the plus sign

4. Choose Match

Expected Results: A list of all Match Profiles (both system provided and locally created) is shown

Actual Results: Only locally created Match Profiles are shown (if there are none, you get "no results found")

Additional Information:
When I did my original creation of the job profiles in May, our test tenant would have been on Orchid-SP-3 or 4, so this seems to be something that was introduced in one of the patches since then.

To a librarian wishing to overlay existing FOLIO records with matching incoming MARC record, it is a source of great confusion that, when there is a match and an underlying SRS record, the status of the SRS record is sometimes reported as Updated and sometimes as Created – even if you are matching on the same field, overlaying the same record. This may cause enough uncertainty about whether Data Import works as expected, overlaying the right records when it should, to make the librarian reluctant about using Data Import.

It seems that either

• there is something wrong in how FOLIO creates/updates existing SRS records when matching on anything but the UUID in 999i
  or

• the information about SRS update/creation displayed in the log is incorrect or designed in a way that is more confusing than useful to end users (librarians)

It would be great, if as a first step, someone with insight into how Data Import works could review the current behaviour to assess whether librarians can "safely" use Data Import with match profiles even though the log shows this ambiguous result.

Steps to Reproduce:

To test, you need:

• a FOLIO instance with an underlying SRS record

• three .mrc files: one with the UUID of the instance in 999i, one without a 999i field but with the ISBN of the instance in 020a, and one with a 999i field that contains a value which is not an instance UUID.

Use the following job profile:
https://bugfest-kiwi.folio.ebsco.com/settings/data-import/job-profiles/view/5d105836-e127-48f4-a942-dec42ef265e6

Try the following three:

1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.

2. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.

3. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"

Expected Results:

1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.
   The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

1. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.
   *The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"
   The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

Steps to Reproduce:

1. Log into folio-snapshot as diku_admin.

2. Go to settings > Data import > field protections and add the 541 tag as a protected field. 

3. In settings > data import create a new update instance mapping and action profile. The profile can be empty or the status / statistical code/ cataloged date can be changed. 

   1. Create a new match profile on the 999ff$i to instance uuid.

   2. Create a new job using the new match profile and instance uuid.

4. Use the default import profile to create a new record in the system that has a 541 tag. 

5. Go to inventory and locate the newly created record. Select the record and then use the Action menu to export the instance.

6. Open the newly created and edit the 541 in QuickMarc so that you will know that this was the existing 541 record. Save the record. 

7. Use data import to overlay the FOLIO record. There should be two 541s in the record. The existing 541 (protected) will be the first 5xx field in the record and the newly imported 541 will be in its existing place in the record.

Expected Results: Both 541 fields will be next to each other filed according to their previous position. I think it can be assumed that protected 541 will be the first 541 in the new record where both 541 fields are present.

Actual Results: The protected 541 is filed at the beginning of all 5xx fields in the record and the new 541 is in place as it was in the incoming file.

From 2024-01-31 chat: Technically, not true Lynne. I accidentally deleted a job profile without unlinking stuff first (which I was not stopped from doing). I was then not allowed to delete the associated profiles because they needed to be unlinked first, which I could not do because I could no longer get to the job profile to unlink them.

Steps to Reproduce:

1. Log into Poppy Bugfest as a user with admin permissions.

2. Create a job profile using the steps here: https://foliotest.testrail.io//index.php?/cases/view/388505

   1. On the holding field mapping profile, update the Permanent location mapping to be 945$h; else "1 hour earlier (1he)"

Case 1:

1. Use the .mrc file attached to the Testrails and import using the job profile created in Step 2.

Expected Results: Multiple holdings and items are created.

Actual Results: Only a single holdings/item representing the first 945 field is created.

Case 2:

1. Modify holdings mapping profile to contain Permanent location mapping 945$h

2. Import attached NoLocationMarcField.mrc file

Expected Results: Job finished with status “Completed with errors“, error log for holdings says that permanent location should be not null

Actual Results: Job finished with status “Completed with errors“, error log for holdings show StackTraceThrowable exception

Additional Information: Removing the ; else “LOCCODE” produced expected results. See my tests by looking at the logs for 12177 (successful creation of multiple holdings/items) and 12180 (only created single holdings/item). The MARC records used for these examples and the job profiles are listed within the log.

Current workaround: No workaround

Steps to Reproduce:

1. Log into Orchid bug-fest 

2. Create brief pending order records using POL numbers and Vendor Reference numbers from MARC file

3. Use Import job profile: GOBI API eBooks - Full cataloging

4. Upload records and run job

5. Vendor Reference Numbers and POL numbers must be unique. If running the import job multiple times, you must change these data points in the MARC records.

Expected Results: Match on the PO Line and then the Vendor Order Reference number before updating the Instance record with full cataloging record and the holdings item with the correct holdings type and permanent location

Actual Results: Records are all discarded