Archived Data Import Topics

Introduction

This page is meant to track archived topics that are now closed. When a topic from The Data Import Tracker closes, then remove it from that page of current issues to this spreadsheet below.

How to contribute to other people's discussion topics:

  1. Do not add detail to closed or discussed topics as your comments may be overlooked. In this situation, it might be best to Add your details as a new topic and reference the previous topic.

  2. To contribute to an existing topic. Add a new paragraph to the description column.

  3. @mention yourself at the beginning of the paragraph

How to indicate you are also interested in a topic:

  1. @mention yourself in the "Interested parties" column and add your institution name

How are topics archived:

When a topic status is set to closed by it's "Owner". The topic must also be moved to the Data Import Topic Tracker Archive.

  1. Copy the topic and paste it at the top of the Archived topics page that is nested under this page

  2. Delete the topic from this page

Closed Topics

Status

Topic

Description/use case

Date Added

Provided By (Name/Institution)

Interested Parties

Has Been Discussed (Link to agenda/minutes)

Jira Link

Action Required

Status

Topic

Description/use case

Date Added

Provided By (Name/Institution)

Interested Parties

Has Been Discussed (Link to agenda/minutes)

Jira Link

Action Required

1

CLOSED

MARC-MARC Matching Enhancements

MARC-MARC matches and MARC-Inventory matches have differing use cases. Pairing a MARC-MARC match with a more specific MARC-Instance or MARC–Holdings or MARC–Item match allows for identifying a specific record to be updated, or confirms that a new record is needed.

 

We want to ensure that MARC-MARC matching works properly for repeatable and non-repeatable fields, especially 0XX/9XX fields, and that they can pair well with Inventory submatches.

In scope:

  • MARC-MARC matches that result in multiple possible hits can be narrowed to single records with MARC-Inventory or static value submatches

  • Review any existing matching bugs and plan to resolve as part of this feature

Out of scope:

  • After a MARC-MARC or MARC-Instance match, a user can include both Instance and MARC Bib actions afterwards (need examples from users)

  • Confirm that MARC matches are working properly using indicator wildcards (asterisks) versus blanks

  • Currently ISBN matches do not translate 10-digit and 13-digit so that they can be matched against each other, Include in this feature, or handle as a separate feature in the future? See MODSOURMAN-269

  • Should we add a bug for not being able to have an override action for field protections under a MARC-Instance match?

  • What else?

Use case(s):

  • SMEs: Please add examples*
     * MARC-MARC match on OCLC number and then submatch by Instance status

  • MARC-MARC match on 001 and then submatch for holdings by permanent location

  • Need a use case that results in multiple SRS hits that then need to be narrowed down by Inventory match

 

2020-05-13

All

All

2024-1-17 Data Import Subgroup meeting

UXPROD-2742: MARC-MARC matching enhancements: Narrowing multiple matches & Bug workClosed

More Use Cases and scenarios

2

closed

Data Import removes duplicate 856s in SRS

Overview:  When updating an SRS record via Data Import, some MARC fields are duplicated while others are de-duped without notification or guidelines.    
**

Steps to Reproduce:

  1. Log into any environment on Orchid or Nolana.

  2. Identify any record in inventory and export the MARC record.

  3. Add two exact duplicate 856 fields to the record.

    1. Exact duplicates including indicators and subfields.  Copy & paste.

  4. Import using a simple overlay job profile matching on a subfield of the 999 to update the SRS.  Do not use MARC modifications.

Expected Results: The SRS record contains duplicate 856 fields.
Actual Results: The SRS record contains only one of the duplicate 856 fields.
**

Additional Information:  We know that Data Import does not de-dupe the 903 field, for example, during an update but it does the 856 field.  Data Import jobs which create new SRS records includes the duplicate 856 fields.  This raises several questions:

  • Why does the system de-dupe during an update without explicit instructions from the user?  

  • Which fields does the system de-dupe and when?

  • Is this a bug?  Or by design?  

From testing, there appears to be no difference between de-duping of the 856 when field protections are applied or not.

 

 

2023-06-13

@Corrie Hutchinson (Unlicensed) 

All

2024-1-17 Data Import Subgroup meeting

2024-1-10 Data Import Subgroup meeting

 

https://folio-org.atlassian.net/browse/MODDATAIMP-879

Need to define what deduplication means

Clarify expectations
Deduping in the UI vs SRS
Deduping the incoming record
3

CLOSED

Adding MARC modifications to imports with update actions creates broken records

Overview: 

Steps to Reproduce:

  1. Log into Morning Glory bugfest

  2. Go to inventory and select import using the "OCLC with MARC modifications" option and import OCLC number 31934425

  3. The record should import successfully and the title will have received an obvious MARC modification

  4. Attempt to overlay the same record with the same OCLC number.

Expected Results:

Overlaying the record works. The MARC is modified as described in the profile and the instance and SRS are updated.

Actual Results:

The instance is not updated. A modified SRS record is created but still has the original OCLC 001 and 003. QuickMARC will not work on the record.

Additional Information:
URL:  Update profile: https://bugfest-mg.int.aws.folio.org/settings/data-import/job-profiles/view/1c351fa4-d578-434f-a02a-7ff46af16f06?query=single&sort=name

An example of an instance with this issue: https://bugfest-mg.int.aws.folio.org/inventory/view/53e28701-dccc-49be-a01d-9adaa15f4cb6?query=neuromancer&sort=title&xidtype=0dd718cf-a09a-4f1c-be6a-0cf0de58b424

Job profiles

BE Notes:

  • First Modify action saves the record in SRS, subsequent Match on MARC Bib in SRS returns multiple match error, because it finds not only the original record, but also the one that was just saved.

  • During modify action post-processing an attempt appears to change instance hrid because of incoming record doesn't contain actual instance hrid in 001
    that causes error on instance update.

Note for QAs:
When this bug is fixed, create a new TestRail for modification, followed by match, followed by update action, as outlined in the repro steps above. Then unlink this Jira from TestRail C350914

 

2022-08-04

@Jenn Colt 

All

 

MODINV-994: Adding MARC modifications to imports with update actions creates broken recordsClosed

 

4

CLOSED

Adding MARC modifications to single record overlay doesn't respect field protections

Overview: When a MARC modification action is added to the end of the single record overlay job, protected fields from the existing MARC SRS are removed rather than from the incoming file.

Steps to Reproduce:

  1. Log into some FOLIO snapshot environment 

  2. Go to Settings/Data Import

  3. Duplicate the field mappings for Inventory Single Record - Default for the update marc bib and update instance

  4. Create a new field mapping for a marc modification where the fields 029, 506, 856, 583 are deleted

  5. Create action profiles linked to the field mappings you just created

  6. Duplicate the inventory single record match for no srs and existing srs

  7. In field protections, make sure that you add 583 with subfield 5 and data MU in addition to 856 with subfield 5 and data MU.

  8. Create a job with a match to existing marc srs. On matches update marc srs. On no matches, another match to no srs and then under that update instance. Then at the level of the 1st match, add the modify marc action (see screenshot)

  9. Bring in a record and add a 583 with a $5 MU and a 856 $5 MU

  10. Overlay

Expected Results: The existing and protected fields should remain in the record.

Actual Results: The protected fields have been removed.

Additional Information:
URL: The job profile on orchid bugfest is https://bugfest-orchid.int.aws.folio.org/settings/data-import/job-profiles/view/b590fc78-f069-42a1-bfdd-3988c7d6be00?query=fc%20test&sort=name. 

 

2023-08-23

@Jennifer Eustis 

All

2024-2-21 Data Import Subgroup meeting

MODDATAIMP-897: Adding MARC modifications to single record overlay doesn't respect field protectionsClosed

Functionality audit being prepared in DI lab

5

CLOSED

Single record overlay creates duplicate OCLC # / 035

Overview:  When "Overlay source bibliographic record' is employed for the first time, duplicate 035 fields are created.

Steps to Reproduce:

  1. Log into bugfest-Orchid 

  2. Open Inventory and search for an instance (random records were chosen and the behavior was duplicated each time) 

  3. From Actions, choose 'Overlay source bibliographic record':

    1. External target = OCLC WorldCat

    2. Profile = Inventory single record - default update instance (Default)

    3. Enter the OCLC# of the instance currently being viewed

  4. Click 'Import'

Expected Results: The bibliographic record and instance are updated with the latest OCLC version.
Actual Results: The bibliographic record and instance are updated but with duplicate 035 fields for those OCLC#s with a prefix.  
Additional Information: Testing shows that this happens on the initial overlay, but subsequent overlays do not continue to 'add' duplicates.  

Records tested in bugfest-orchid: in523951 and in2486915 (screenshots of before overlay and after are attached) 

Duplicate data causes issues with integrations and other functions that rely on the OCLC# as a match point.  

 

2023-04-12

@Corrie Hutchinson (Unlicensed) 

All

 

MODSOURCE-767: Single record overlay creates duplicate OCLC#/035Closed

 

6

CLOSED

Not able to use the system generated match profiles

I am attempting to create a new job profile for single record import to get rid of the junk fields. I was able to do this a while back on our test tenant and today I was finally able to get around to creating it on our production tenant. 

 

2023-09-14

 

 

 

UIDATIMP-1521: Not able to use system generated Match profilesClosed

 

7

CLOSED

Data Import log displays ambiguous information for successful matches on anything but 999i/instance uuid

f your match profile matches incoming MARC 020 on instance ISBN, and the incoming MARC record matches an instance, the reported status in the log of the SRS record will be Updated if the MARC record also contains a 999-ff-i matching the instance's UUID, but Created if the incoming MARC record does not contain a 999-ff-i. In both cases the status reported for the instance in the log is Updated.

 

2022-05-30

 

 

 

MODSOURMAN-848: Data Import log displays disconcertingly ambiguous information for successful matches on anything but 999i/instance UUID Closed

 

8

closed

data import sorts protected fields out of order after update

Field protections moves the protected field to the first Nxx field. For instance, if the protected field is a 541, the protected 541 becomes the first of all 5xx tags. (see screen shot attached.)

 

2023-05-08

 

 

 

MODDICORE-358: Data import sorts protected fields out of order after updateClosed

 

9

CLOSED

Job summary: error column does not display errors

When there is any error related to an instance/authority/orders/invoice, the error column does not display it.

2024-01-25

 

 

 

UIDATIMP-1590: Job summary: error column does not display errorsClosed

 

10

CLOSED

Field mapping profiles: state of the final form fields is not set

When switching between Folio record types fields with the same name do not reset the state (value, dirty, etc.), although the field values are equal to the initial.

Current workaround: start over/refresh page

2021-06-30

 

 

 

https://folio-org.atlassian.net/browse/UIDATIMP-947

 

11

CLOSED

Incorrect quantity is displayed in the cell of no action and error rows at the individual import job's log

The '1' number of Instance is displayed in cell in the row with the 'No action' and 'Error' rows header in the 'Summary table' at the individual import job's log.

2024-01-19

 

 

 

https://folio-org.atlassian.net/browse/MODSOURMAN-1117

 

12

CLOSED

The status of srs marc is created after match+modify action

Expected Results: The status of SRS MARC is 'Updated' in the Import log after uploading MARC file for update.

2023-03-07

 

 

 

MODDATAIMP-1008: Document scenarios for Modify used with Update action and verify the behaviorClosed

 

13

CLOSED

PMSystem displayed as source in quickmarc view when record was created by non matches action of job profile

"PMSystem" displayed as source (instead of User's last and first name) in "Edit MARC authority record" view when record was created by "Non-matches" action of job profile.

2023-03-07

 

 

 

MODSOURCE-608: "PMSystem" displayed as source in "quickmarc" view when record was created by "Non-matches" action of job profileClosed

 

14

CLOSED

Single record overlay creates duplicate oclc #/035

When "Overlay source bibliographic record' is employed for the first time, duplicate 035 fields are created.   **  

2023-04-12

 

 

 

MODSOURCE-767: Single record overlay creates duplicate OCLC#/035Closed

 

15

CLOSED

Invoice level adjustments do not work

When loading an EDIFACT invoice using a field mapping profile with invoice-level adjustments, the adjustments error

2021-03-29

 

@Kimberly Pamplin 

 

MODDICORE-125: Invoice level adjustments do not workClosed

 

16

CLOSED

Invoice line level adjustments don't work

When loading an EDIFACT invoice using a field mapping profile with invoice-level adjustments, the adjustments error

2021-03-29

 

@Kimberly Pamplin 

 

https://folio-org.atlassian.net/browse/MODDICORE-124

 

17

CLOSED

Data import incorrectly maps Resource type for no display constant generated

See steps in JIRA

2023-04-19

 

 

 

https://folio-org.atlassian.net/browse/MODDATAIMP-804

 

18

CLOSED

Status descending sort on Data Import view all page not working

In Honeysuckle Bugfest, on the Data Import View all, the status sort ascending works, but not descending

2020-12-09

 

 

 

https://folio-org.atlassian.net/browse/UIDATIMP-798

 

19

CLOSED

MARC holdings update log has additional empty row

MARC Holdings update log has additional empty row. If two "MARC Holdings" records are updated by one job, then 2 additional empty rows will be displayed.

2023-05-01

 

 

 

https://folio-org.atlassian.net/browse/MODSOURMAN-983

 

20

CLOSED

Data Import field mapping profile is saved with data deleted from the system

The user can save a mapping profile with data that has been deleted from the system.

2022-08-19

 

 

 

https://folio-org.atlassian.net/browse/MODDICONV-290

 

21

closed

Alert modal with error message is displayed on page after entering '##*' characters and clicking on search button

see steps in JIRA

2022-06-10

 

 

 

https://folio-org.atlassian.net/browse/MODDICONV-250

 

22

closed

DI Log: Title missing but status reads updated

see JIRA

2023-10-24

 

 

 

https://folio-org.atlassian.net/browse/MODSOURMAN-1069

 

23

CLOSED

RRT - Invoices don't display fund codes

Institution specific - MI State Univ./ Library of Michigan

2024-01-04

 

 

 

https://folio-org.atlassian.net/browse/MODINVOSTO-173

 

24

closed

Job profile with POL/VRN match cascade does not finish properly

Some Inventory records (Instances/Holdings/Items) get created when orders are opened, depending on the Inventory setting in the POL. During testing of POL/VRN matching, I noticed that some holdings being updated by importing MARC Bibs were having their source changed from FOLIO to MARC. We need to ensure this DOES NOT happen.

2022-05-25

 

 

 

https://folio-org.atlassian.net/browse/MODINV-709

 

25

CLOSED

Poppy data import log does not include links to records for SRS updates.

For SRS updates with no instance update, the changes to the import logs have resulted in no record links in the log. Before the log indicated that the instance was updated and included a link to the instance. Now, the log indicates that the SRS record is updated, but there is no link. I am guessing it is because there is no SRS record per se in FOLIO. Can we revisit the decision to not display the instance update status with a link? Or add a link to the instance to the SRS updated status?  PPT from Data Import subgroup work:

From this spreadsheet it appears that the instance should also be updated and provide the record link: .

 

 

 

 

 

https://folio-org.atlassian.net/browse/MODDATAIMP-984

https://folio-org.atlassian.net/browse/MODSOURMAN-1106

 

26

CLOSED

 RRT, 5C match bug

Problem of match that didn't match

2023-09-28

 

 

 

https://folio-org.atlassian.net/browse/MODINV-882

 

27

CLOSED

Duplicate records in incoming file causes problems after overlay process with no error reported

When overlaying instance records, if the incoming file has duplicate records and therefore multiple incoming matches for one match in FOLIO, the record that was overlaid in FOLIO cannot be opened using quickMARC. Note: In this scenario the incoming file has the duplicate records and therefore duplicate match points. FOLIO does not have duplicates.

2021-12-15

 

 

 

https://folio-org.atlassian.net/browse/MODSOURCE-530

 

28

CLOSED

Enforce an order to deletion for Data Import profiles

There needs to be an enforced order to deletion for Data import profiles to prevent this or there should be a confirmation window that lets you delete associated match/action profiles with the job profile if they aren't in use in another Job profile

 

2024-01-31

 

 

2024-2-28 Data Import Subgroup meeting

2024-7-10 Data Import Subgroup meeting

https://folio-org.atlassian.net/browse/MODDATAIMP-1026

https://folio-org.atlassian.net/browse/MODDICONV-388

Developers will look into this

29

Closed

There needs to be a warning or error to stop the job when a job contains no action profiles.

If a job doesn't have any actions, nothing happens and there is a risk that the records are corrupted.

2021-02-08

@Sara Colglazier

@Jenn Colt

@Christie Thomas

 

2024-2-28 Data Import Subgroup meeting

2024-7-10 Data Import Subgroup meeting

https://folio-org.atlassian.net/browse/MODDICONV-373

Ryan will ask what the behavior is for when actions are missing and you try to run a job or if you can edit a job with no actions.

Behavior now in FOLIO snapshot

30

CLOSED

When importing EDIFACT files, invoices lines aren't in order

When importing an Edifact file, the invoices lines aren't in order.

2024-02-06

Corrie Hutchinson, Jennifer Eustis

@Kimberly Pamplin

 

https://folio-org.atlassian.net/browse/MODDATAIMP-995

type key summary assignee reporter priority status resolution created updated due
Loading...
Refresh