Page Comparison

We want to ensure that MARC-MARC matching works properly for repeatable and non-repeatable fields, especially 0XX/9XX fields, and that they can pair well with Inventory submatches.

In scope:

• MARC-MARC matches that result in multiple possible hits can be narrowed to single records with MARC-Inventory or static value submatches

• Review any existing matching bugs and plan to resolve as part of this feature

Out of scope:

• After a MARC-MARC or MARC-Instance match, a user can include both Instance and MARC Bib actions afterwards (need examples from users)

• Confirm that MARC matches are working properly using indicator wildcards (asterisks) versus blanks

• Currently ISBN matches do not translate 10-digit and 13-digit so that they can be matched against each other, Include in this feature, or handle as a separate feature in the future? See MODSOURMAN-269

• Should we add a bug for not being able to have an override action for field protections under a MARC-Instance match?

• What else?

Use case(s):

• SMEs: Please add examples*
   * MARC-MARC match on OCLC number and then submatch by Instance status

• MARC-MARC match on 001 and then submatch for holdings by permanent location

• Need a use case that results in multiple SRS hits that then need to be narrowed down by Inventory match

Steps to Reproduce:

1. Log into any environment on Orchid or Nolana.

2. Identify any record in inventory and export the MARC record.

3. Add two exact duplicate 856 fields to the record.

   1. Exact duplicates including indicators and subfields.  Copy & paste.

4. Import using a simple overlay job profile matching on a subfield of the 999 to update the SRS.  Do not use MARC modifications.

Expected Results: The SRS record contains duplicate 856 fields.
Actual Results: The SRS record contains only one of the duplicate 856 fields.
**

Additional Information:  We know that Data Import does not de-dupe the 903 field, for example, during an update but it does the 856 field.  Data Import jobs which create new SRS records includes the duplicate 856 fields.  This raises several questions:

• Why does the system de-dupe during an update without explicit instructions from the user?  

• Which fields does the system de-dupe and when?

• Is this a bug?  Or by design?  

From testing, there appears to be no difference between de-duping of the 856 when field protections are applied or not.

Steps to Reproduce:

1. Log into Morning Glory bugfest

2. Go to inventory and select import using the "OCLC with MARC modifications" option and import OCLC number 31934425

3. The record should import successfully and the title will have received an obvious MARC modification

4. Attempt to overlay the same record with the same OCLC number.

Expected Results:

Overlaying the record works. The MARC is modified as described in the profile and the instance and SRS are updated.

Actual Results:

The instance is not updated. A modified SRS record is created but still has the original OCLC 001 and 003. QuickMARC will not work on the record.

Additional Information:
URL:  Update profile: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/1c351fa4-d578-434f-a02a-7ff46af16f06?query=single&sort=name

An example of an instance with this issue: https://bugfest-mg.int.aws.folio.org/inventory/view/53e28701-dccc-49be-a01d-9adaa15f4cb6?query=neuromancer&sort=title&xidtype=0dd718cf-a09a-4f1c-be6a-0cf0de58b424

Job profiles

• Morning Glory

  • ISRI Create: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/d8f1c585-b56a-4024-b82a-930be5569396?query=jenn%20single&sort=name

  • ISRI Update: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/1c351fa4-d578-434f-a02a-7ff46af16f06?query=jenn%20single&sort=name

• Nolana

  • ISRI Create: https://bugfest-nolana.int.aws.folio.org/settings/data-import/job-profiles/view/f44ee72c-f396-4742-b16d-3393c78623ca?query=jenn%20single&sort=name

    • Job completed: https://bugfest-nolana.int.aws.folio.org/data-import/job-summary/be62c707-4c8e-4a75-baed-b640df515d14

  • ISRI Update: https://bugfest-nolana.int.aws.folio.org/settings/data-import/job-profiles/view/127f27f8-5711-43fb-8114-393f6bb478f1?query=jenn%20single&sort=name

    • Job completed with errors (no error message): https://bugfest-nolana.int.aws.folio.org/data-import/job-summary/aacede86-7b50-449c-9d95-c5dbfb91c0d6

• Orchid

  • ISRI Create: https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/29cf9029-8e70-40ee-a972-fb35a11b257b?query=jenn%20single&sort=name

    • Job completed (but 2 entries in the log, 1 with title before MARC MOD, and 1 with title after MARC MOD): https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/829f4a43-5dec-41a7-8bae-d94ef941ebe7

  • ISRI Update: https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/4447c1dc-39ff-45cf-8b15-74dde79869dd?query=jenn%20single&sort=name

    • Job completed with errors (no error message): https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/3731a505-0f79-4506-ba80-12cdf3e9c3fc

BE Notes:

• First Modify action saves the record in SRS, subsequent Match on MARC Bib in SRS returns multiple match error, because it finds not only the original record, but also the one that was just saved.

• During modify action post-processing an attempt appears to change instance hrid because of incoming record doesn't contain actual instance hrid in 001
  that causes error on instance update.

Note for QAs:
When this bug is fixed, create a new TestRail for modification, followed by match, followed by update action, as outlined in the repro steps above. Then unlink this Jira from TestRail C350914

Steps to Reproduce:

1. Log into some FOLIO snapshot environment 

2. Go to Settings/Data Import

3. Duplicate the field mappings for Inventory Single Record - Default for the update marc bib and update instance

4. Create a new field mapping for a marc modification where the fields 029, 506, 856, 583 are deleted

5. Create action profiles linked to the field mappings you just created

6. Duplicate the inventory single record match for no srs and existing srs

7. In field protections, make sure that you add 583 with subfield 5 and data MU in addition to 856 with subfield 5 and data MU.

8. Create a job with a match to existing marc srs. On matches update marc srs. On no matches, another match to no srs and then under that update instance. Then at the level of the 1st match, add the modify marc action (see screenshot)

9. Bring in a record and add a 583 with a $5 MU and a 856 $5 MU

10. Overlay

Expected Results: The existing and protected fields should remain in the record.

Actual Results: The protected fields have been removed.

Additional Information:
URL: The job profile on orchid bugfest is https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/b590fc78-f069-42a1-bfdd-3988c7d6be00?query=fc%20test&sort=name. 

Steps to Reproduce:

1. Log into bugfest-Orchid 

2. Open Inventory and search for an instance (random records were chosen and the behavior was duplicated each time) 

3. From Actions, choose 'Overlay source bibliographic record':

   1. External target = OCLC WorldCat

   2. Profile = Inventory single record - default update instance (Default)

   3. Enter the OCLC# of the instance currently being viewed

4. Click 'Import'

Expected Results: The bibliographic record and instance are updated with the latest OCLC version.
Actual Results: The bibliographic record and instance are updated but with duplicate 035 fields for those OCLC#s with a prefix.  
Additional Information: Testing shows that this happens on the initial overlay, but subsequent overlays do not continue to 'add' duplicates.  

Records tested in bugfest-orchid: in523951 and in2486915 (screenshots of before overlay and after are attached) 

Duplicate data causes issues with integrations and other functions that rely on the OCLC# as a match point.  

If we wanted to remove the old records, how complicated would that be, and what might we need to take into consideration?

KS: there are also lots of "trash" data saved to SRS as a result of failed or stopped imports (records linked to a snapshot/jobExecution that is Cancelled or records that don't have 999 ff i UUIDs) - consider ways to clean up that data as well.

• Developers

  • If UI import log related to the previous version of a record has not yet been deleted, what would happen if an SRS record related to the log was deleted. Would it break the log? Can we prevent SRS records from being deleted if they are still connected to an import log in the UI?

  • Should we plan to keep the current and most recent previous? (would that be helpful for when we implement the rollback feature?)

  • Would there be any issues related to the various UUIDs assigned during imports?

  • Would there be any issues related to

    • quickMARC updates?

    • LDP data extracts?

    • Data export

    • OAI-PMH

  • OK for it to apply to all SRS records? all 3 MARC types, EDIFACT invoices?

  • How often to run the cleanup? Make it variable in the MOD or UI settings?

  • How much effort would this be? T-shirt sizes for UI and BE

• SMEs

  • Would this be helpful? Do we have any way to measure or estimate the impact on SRS/Import performance before implementing?

  • Any requirements?

  • Any questions or concerns?

Results of this spike

• Wiki page with design

• All required Jira stories

• T-shirt sizes for UI and MOD

• Decide if this is a separate feature or just a couple of stories

However, when I went to add the Match profile (Inventory Single Record - Default match for existing SRS record) which is a system generated profile, it did not appear as one of the options. When I look at the Action profiles, I am able to choose system generated ones, so the problem is only with the Match profiles. We are on Orchid-SP-5. I was also able to recreate the issue on Snapshot. (I don't have Bugfest access so I did not try that.)

Steps to Reproduce:

1. Log into FOLIO Snapshot

2. Go to Settings–>Data Import–>Job Profiles–>Actions–>New Job Profile

3. Under Overview, click on the plus sign

4. Choose Match

Expected Results: A list of all Match Profiles (both system provided and locally created) is shown

Actual Results: Only locally created Match Profiles are shown (if there are none, you get "no results found")

Additional Information:
When I did my original creation of the job profiles in May, our test tenant would have been on Orchid-SP-3 or 4, so this seems to be something that was introduced in one of the patches since then.

To a librarian wishing to overlay existing FOLIO records with matching incoming MARC record, it is a source of great confusion that, when there is a match and an underlying SRS record, the status of the SRS record is sometimes reported as Updated and sometimes as Created – even if you are matching on the same field, overlaying the same record. This may cause enough uncertainty about whether Data Import works as expected, overlaying the right records when it should, to make the librarian reluctant about using Data Import.

It seems that either

• there is something wrong in how FOLIO creates/updates existing SRS records when matching on anything but the UUID in 999i
  or

• the information about SRS update/creation displayed in the log is incorrect or designed in a way that is more confusing than useful to end users (librarians)

It would be great, if as a first step, someone with insight into how Data Import works could review the current behaviour to assess whether librarians can "safely" use Data Import with match profiles even though the log shows this ambiguous result.

Steps to Reproduce:

To test, you need:

• a FOLIO instance with an underlying SRS record

• three .mrc files: one with the UUID of the instance in 999i, one without a 999i field but with the ISBN of the instance in 020a, and one with a 999i field that contains a value which is not an instance UUID.

Use the following job profile:
https://bugfest-kiwi.folio.ebsco.com/settings/data-import/job-profiles/view/5d105836-e127-48f4-a942-dec42ef265e6

Try the following three:

1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.

2. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.

3. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"

Expected Results:

1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with a FOLIO identifier.
   The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

1. Using a job profile that matches on identifier ISBN, import a MARC record that does not contain a 999i field with a FOLIO identifier.
   *The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

1. Using a job profile that matches on identifier ISBN, import a MARC record that contains a 999i field with the value "catsarecute"
   The incoming 020 of the incoming MARC record successfully matches on the ISBN of the instance. In the log, instance and SRS have status Updated.

Preconditions:

• The job profile for updating existing "MARC bib" records should be created for "Data import" app: " Update MARC Bib by matching 999 ff $s subfield value (830 - update all) ". Related field mapping profile should have following rule in "Field mapping" accordion:
    Field: 830
    In.1: *
    In.2: *
    Subfield: *.

• The system must have at least 1 "MARC Bibliographic" record which has fields linked to at least 1 "MARC Authority" record.

• In Orchid Bugfest

  • Authority record for Cambridge tracts in mathematics: https://bugfest-orchid.int.aws.folio.org/marc-authorities/authorities/0f35c46e-34b6-4a5d-a752-fe7d23ca2719?authRefType=Authorized&headingRef=Cambridge%20tracts%20in%20mathematics%20and%20mathematical%20physics%20no.%2019.%20english%20England&qindex=uniformTitle&query=cambridge%20tracts%20in%20math&segment=browse
    You can use files from the attachments:
      1) "MARC Authority" record (Import it via "Data import" app using "Default - Create SRS MARC Authority" job profile):
    MARC_Auth_130(Cambridge).mrc
      2) "MARC Bibliographic" records (Import it via "Data import" app using "Default - Create instance and SRS MARC Bib" job profile):
    MARC_Bib_830(Cambridge).mrc

• The "830" MARC field with "$a" value "Cambridge tracts in mathematics" of imported "Instance" record must be linked to the "100" field of imported "MARC Authority" record ("Cambridge tracts in mathematics and mathematical physics").

• Authorized user with the following permissions:
    Data import: Can upload files, import, and view logs
    Inventory: All permissions
    MARC Authority: View MARC authority record
    quickMARC: Can Link/unlink authority records to bib records
    quickMARC: View, edit MARC bibliographic record
    UI: Data export module is enabled

• User is on "Inventory" pane with search results for "Instance" record which was linked with "MARC Authority" (see Precondition, e.g.: "The algebraic theory of modular systems / by F.S. Macaulay.").

Steps to Reproduce:

1. Check the checkbox that is displayed next to the "Instance" record which is linked to "MARC Authority" records on the search result pane.
   For example: "The algebraic theory of modular systems / by F.S. Macaulay.".

2. Click on the "Actions" button on the second pane and select "Export Instances (MARC)" option.

3. Go to "Data export" app.

4. Download exported ".mrc" file by clicking on the "File name" column value.

5. Open downloaded ".mrc" file via "MarcEdit" (or any similar app).

6. Delete the linked field (see precondtions): "830" field

7. Save the edited ".mrc" file.

8. Go to the "Data import" app >> Click on the "or select files" button >> Select the updated ".mrc" file (see previous step) >> click the "Open" button

9. Click on the created in precondition "Update MARC Bib by matching 999 ff $s subfield value (830 - update all)" job profile >> Click on the "Actions" in the appeared third pane >> Select "Run" option >> Click on the "Run" button in appeared modal.

10. Find updated record in "Inventory" app

11. Click on the "Actions" button and select "Edit MARC bibliographic record" option from the expanded menu.

Expected Results: Deleted "830" field (see step 6) is not shown.

Actual Results: Deleted "830" field (see step 6) is shown and has divided boxes (see attached screencast).

Steps to Reproduce:

1. Log into bugfest-poppy.

2. Open Data Import and look at Job 10638.

   1. Job Profile = CAH Update SRS MARC on 035$a match w/ qualifier

      1. This profile matches MARC to MARC on the incoming 035$a with a qualifier exactly matching the 035$a of the existing record : (MiFhGG)galncbln000092 -> galncbln000092.

      2. For matches, the record is updated overriding the protection on the 856. 

   2. Sample test records = 51356_test_records.mrc

Expected Results: The job matches the incoming records to the SRS records associated with in10783235 and in10783236 and updates these records with the new 856 in the incoming MARC record.

Actual Results: The incoming records are not matched, the log for SRS MARC says 'No action', and the SRS record is not updated.  

Additional Information: I tested the Field Mapping Profile without the 856 protection and it still failed (Job 10641).  

When reviewing logs on an internal system, the error messages given note that a match is not found.

Additional testing was done on changing the 'Match criterion' of the existing record and no value was found to make the Match profile successful.  

Original testing done in an Orchid environment.

Steps to Reproduce:

1. Log into folio-snapshot as diku_admin.

2. Go to settings > Data import > field protections and add the 541 tag as a protected field. 

3. In settings > data import create a new update instance mapping and action profile. The profile can be empty or the status / statistical code/ cataloged date can be changed. 

   1. Create a new match profile on the 999ff$i to instance uuid.

   2. Create a new job using the new match profile and instance uuid.

4. Use the default import profile to create a new record in the system that has a 541 tag. 

5. Go to inventory and locate the newly created record. Select the record and then use the Action menu to export the instance.

6. Open the newly created and edit the 541 in QuickMarc so that you will know that this was the existing 541 record. Save the record. 

7. Use data import to overlay the FOLIO record. There should be two 541s in the record. The existing 541 (protected) will be the first 5xx field in the record and the newly imported 541 will be in its existing place in the record.

Expected Results: Both 541 fields will be next to each other filed according to their previous position. I think it can be assumed that protected 541 will be the first 541 in the new record where both 541 fields are present.

Actual Results: The protected 541 is filed at the beginning of all 5xx fields in the record and the new 541 is in place as it was in the incoming file.

Preconditions:

• The job profile should be created for "Data import" app: "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)"

• "Match profile" should have following specified:

Field: 999
Ind 1: f
Ind 2: f
Subfield: s

• Action profile should be Update MARC Bib

• "Field mapping profile" should have following rules specified:

Update specific fields only

Field: *
In.1: *
In.2: *
Subfield: 1.

Field: *
In.1: *
In.2: *
Subfield: 2.

No Field protection overrides

• Attached "MARC Bib" record should be imported to the system via "Default - Create instance and SRS MARC Bib" job profile (it has subfields "$1", "$2" in two fields)

Steps to Reproduce:

1. Export previously imported "MARC Bib" record from Preconditions (For preconditions, use MARC_bib_Black_panther_five_nonRepeatable_linkable_fields.mrc file attached to this Jira, and import using the default Create MARC Bib and Instance job profile)

2. Open exported .mrc file in "MarcEdit" or similar editor

3. In 130 field: delete "$2" subfield and update "$1" subfield value (e.g., change "$1" value to "new")

4. In 240 field: delete "$2" subfield and add one more "$1" subfield (e.g., add "$1onemore")

5. Save updated file

6. Import updated file using "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)" job profile created in Preconditions

7. Open updated "MARC Bib" record for edit ("Actions" → "Edit MARC bibliographic record") 

Expected Results: "$2" removed from both edited fields. Following "$1"subfield values are shown in fields:

•  "130" field: "$1new"

• "240" field: "$1original $1onemore"

Actual Results: "$2" subfields are not removed from edited fields. "$1" subfields updated/added as expected

Preconditions:

• The job profile should be created for "Data import" app: "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)"

• "Match profile" should have following specified:

Field: 999
Ind 1: f
Ind 2: f
Subfield: s

• Action profile should be: Update MARC Bibs

• "Field mapping profile" should have following rules specified:

Update specific fields:

Field: *
In.1: *
In.2: *
Subfield: 1.

Field: *
In.1: *
In.2: *
Subfield: 2.

No overrides to MARC field protection

• Attached "MARC Bib" record should be imported to the system via "Default - Create instance and SRS MARC Bib" job profile

Steps to Reproduce:

1. Export previously imported "MARC Bib" record from Preconditions 

2. Open exported .mrc file in "MarcEdit" or similar editor

3. Add "$1" subfield values in with no "$1" (e,g.:

   1. add "$1555555" to "035" field

   2. add "$1test" to "264" field

4. Save updated file

5. Import updated file using "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)" job profile created in Preconditions

6. Open updated "MARC Bib" record for edit ("Actions" → "Edit MARC bibliographic record") 

Expected Results: "$1" added to edited "035", "264" fields. Fields were not duplicated 

Actual Results: "035", "264" fields were duplicated. First copy of each remains unchanged (no added subfield), while second copies contain added "$1" subfield

Preconditions:

• The job profile should be created for "Data import" app: "Update MARC Bib records by matching 999 ff $s subfield value (240, 600)"

• "Match profile" should have following specified:

Field: 999
Ind 1: f
Ind 2: f
Subfield: s

• Actin profile should be: Update MARC Bibs

• "Field mapping profile" should have following rules specified:

Update specific MARC fields

Field: 240
In.1: *
In.2: *
Subfield: 1.

Field: 240
In.1: *
In.2: *
Subfield: 2.

Field: 600
In.1: *
In.2: *
Subfield: 1.

Field: 600
In.1: *
In.2: *
Subfield: 2.

No overrides to the existing field protections

• Attached "MARC Bib" record should be imported to the system via "Default - Create instance and SRS MARC Bib" job profile

Steps to Reproduce:

1. Export previously imported "MARC Bib" record from Preconditions 

2. Open exported .mrc file in "MarcEdit" or similar editor

3. Add "$1" and "$2" subfields to "240" field (e.g., add "$1test$2testing")

4. Add "$1" and "$2" subfields to one of the "600" fields (e.g., add "$1test$2testing" to the third "600" field)

5. Save updated file

6. Import updated file using "Update MARC Bib records by matching 999 ff $s subfield value (240, 600)" job profile created in Preconditions

7. Open updated "MARC Bib" record for edit ("Actions" → "Edit MARC bibliographic record") 

Expected Results: "$1", "$2" added to edited "240", "600" fields. Fields were not duplicated 

Actual Results: "240", "600" fields were duplicated. First copy of each remains unchanged (no added subfields), while second copies contain added "$1", "$2" subfields

From 2024-01-31 chat: Technically, not true Lynne. I accidentally deleted a job profile without unlinking stuff first (which I was not stopped from doing). I was then not allowed to delete the associated profiles because they needed to be unlinked first, which I could not do because I could no longer get to the job profile to unlink them.

Steps to Reproduce:

1. Log into Poppy Bugfest as a user with admin permissions.

2. Create a job profile using the steps here: https://foliotest.testrail.io//index.php?/cases/view/388505

   1. On the holding field mapping profile, update the Permanent location mapping to be 945$h; else "1 hour earlier (1he)"

Case 1:

1. Use the .mrc file attached to the Testrails and import using the job profile created in Step 2.

Expected Results: Multiple holdings and items are created.

Actual Results: Only a single holdings/item representing the first 945 field is created.

Case 2:

1. Modify holdings mapping profile to contain Permanent location mapping 945$h

2. Import attached NoLocationMarcField.mrc file

Expected Results: Job finished with status “Completed with errors“, error log for holdings says that permanent location should be not null

Actual Results: Job finished with status “Completed with errors“, error log for holdings show StackTraceThrowable exception

Additional Information: Removing the ; else “LOCCODE” produced expected results. See my tests by looking at the logs for 12177 (successful creation of multiple holdings/items) and 12180 (only created single holdings/item). The MARC records used for these examples and the job profiles are listed within the log.

Current workaround: No workaround

Steps to Reproduce:

1. Log into Orchid bug-fest 

2. Create brief pending order records using POL numbers and Vendor Reference numbers from MARC file

3. Use Import job profile: GOBI API eBooks - Full cataloging

4. Upload records and run job

5. Vendor Reference Numbers and POL numbers must be unique. If running the import job multiple times, you must change these data points in the MARC records.

Expected Results: Match on the PO Line and then the Vendor Order Reference number before updating the Instance record with full cataloging record and the holdings item with the correct holdings type and permanent location

Actual Results: Records are all discarded