Data Import Implementers Topic Tracker

Steps to Reproduce:

1. Log into Orchid bugfest

2. Use data import to begin a file of 5000 records with a moderately complex import profile

3. Cancel the data import job when it hits around 2%

4. Immediately go to Inventory and trigger an Inventory Single record import by selecting OCLC and using record # 1234567

5. Then do a few more Inventory single record imports

6. Watch the log to confirm that the stopped job actually stops and the single record imports are happening quickly

Expected Results:

The job cancels and stops processing. Data import jobs started after the cancellation act normally.

Actual Results:

Single record imports started after canceling a job like this are slow. One single record import after a cancelation took 17 minutes.

Additional Information:

In Nolana, canceled jobs created a large number of error messages that seemed to affect performance. Have the logs been checked to be sure this isn't still happening? Is something else causing this behavior?

Attached file and profile "cornell ebz" can be used to replicate.

BE notes (possible solution):

1. Create cache in mod-source-record-manager with jobs that are IN_PROGRESS. When job is marked as CANCELLED, COMPLETED, ERROR - update the cache and send Kafka event (probably, we only need the event for CANCELLED status)

2. mod-source-record-storage, mod-inventory, mod-orders, mod-invoice should also have the same cache as in mod-source-record-manager. Subscribe these modules to the status update event. When Job_Cancelled event is received - update cache, set status of that job to CANCELLED. 

3. In mentioned above 4 modules, when event for record processing is received, before processing - check JobExecution status in the cache. If it is not cancelled - proceed with processing. If there is no record for the given JobExecution, call mod-source-record-manager endpoint to retrieve info for the cache (alternatively call mod-source-record-manager to get the cache data only on module startup). 

The file has 18 invoices and 1104 invoice lines.

Steps to Reproduce:

1. Admin user is logged in.

Steps to Reproduce:

1. Duplicate the "Default – GOBI monograph invoice" profile.

2. Update next fields into the copied profile:

• • Name: any unique name

  • Incoming record type: EDIFACT Invoice

  • FOLIO record type: Invoice

  • Description: clean-up the field

  • Details block:
    1) Batch group: any option from the dropdown list
    2) Vendor name: use Organization lookup to find and select GOBI Library Solutions (which will also automatically fill in the Accounting code)
    3)Payment method: any option from the dropdown list.

       3. Create a new "Action Profile" with a unique valid name and with the next properties in it:

• • Action: Create

  • FOLIO record type: Invoice

  • Link the field mapping profile from step#2.

       4. Create a new "Job profile" with a unique valid name in it and with the next properties in it:

• • Accepted data type: EDIFACT

  • Link an action profile from step#3.

       5. Upload a valid EDIFACT file using Job profile from the previous step.

       6. Wait till the file is uploaded.

       7. Check log UI and summary to see the record with the upload result.

       8. Pay attention on the 'Invoice' column in the 'Created' row.

Expected Results: The '0' number of created invoices is displayed in cells in the row with the 'Created' row in the 'Summary' table in logs.
Actual Results: The '18' number of created invoices is displayed in cells in the row with the 'Created' row in the 'Summary' table in logs.

NOTE: Recreated on Poppy Bugfest:

• Job profile: https://bugfest-poppy.int.aws.folio.org/settings/data-import/job-profiles/view/22c61a1b-3d03-4ddd-bdc8-f885ceab3c61?query=gobi&sort=name (did not include an EDIFACT mapping in the Invoice line description field, so every invoice line errored)

• Log: https://bugfest-poppy.int.aws.folio.org/data-import/job-summary/c541334a-15c7-40e7-89a4-e135d87e11f6

If we wanted to remove the old records, how complicated would that be, and what might we need to take into consideration?

KS: there are also lots of "trash" data saved to SRS as a result of failed or stopped imports (records linked to a snapshot/jobExecution that is Cancelled or records that don't have 999 ff i UUIDs) - consider ways to clean up that data as well.

• Developers

  • If UI import log related to the previous version of a record has not yet been deleted, what would happen if an SRS record related to the log was deleted. Would it break the log? Can we prevent SRS records from being deleted if they are still connected to an import log in the UI?

  • Should we plan to keep the current and most recent previous? (would that be helpful for when we implement the rollback feature?)

  • Would there be any issues related to the various UUIDs assigned during imports?

  • Would there be any issues related to

    • quickMARC updates?

    • LDP data extracts?

    • Data export

    • OAI-PMH

  • OK for it to apply to all SRS records? all 3 MARC types, EDIFACT invoices?

  • How often to run the cleanup? Make it variable in the MOD or UI settings?

  • How much effort would this be? T-shirt sizes for UI and BE

• SMEs

  • Would this be helpful? Do we have any way to measure or estimate the impact on SRS/Import performance before implementing?

  • Any requirements?

  • Any questions or concerns?

Results of this spike

• Wiki page with design

• All required Jira stories

• T-shirt sizes for UI and MOD

• Decide if this is a separate feature or just a couple of stories

Preconditions:

• The job profile for updating existing "MARC bib" records should be created for "Data import" app: " Update MARC Bib by matching 999 ff $s subfield value (830 - update all) ". Related field mapping profile should have following rule in "Field mapping" accordion:
    Field: 830
    In.1: *
    In.2: *
    Subfield: *.

• The system must have at least 1 "MARC Bibliographic" record which has fields linked to at least 1 "MARC Authority" record.

• In Orchid Bugfest

  • Authority record for Cambridge tracts in mathematics: https://bugfest-orchid.int.aws.folio.org/marc-authorities/authorities/0f35c46e-34b6-4a5d-a752-fe7d23ca2719?authRefType=Authorized&headingRef=Cambridge%20tracts%20in%20mathematics%20and%20mathematical%20physics%20no.%2019.%20english%20England&qindex=uniformTitle&query=cambridge%20tracts%20in%20math&segment=browse
    You can use files from the attachments:
      1) "MARC Authority" record (Import it via "Data import" app using "Default - Create SRS MARC Authority" job profile):
    MARC_Auth_130(Cambridge).mrc
      2) "MARC Bibliographic" records (Import it via "Data import" app using "Default - Create instance and SRS MARC Bib" job profile):
    MARC_Bib_830(Cambridge).mrc

• The "830" MARC field with "$a" value "Cambridge tracts in mathematics" of imported "Instance" record must be linked to the "100" field of imported "MARC Authority" record ("Cambridge tracts in mathematics and mathematical physics").

• Authorized user with the following permissions:
    Data import: Can upload files, import, and view logs
    Inventory: All permissions
    MARC Authority: View MARC authority record
    quickMARC: Can Link/unlink authority records to bib records
    quickMARC: View, edit MARC bibliographic record
    UI: Data export module is enabled

• User is on "Inventory" pane with search results for "Instance" record which was linked with "MARC Authority" (see Precondition, e.g.: "The algebraic theory of modular systems / by F.S. Macaulay.").

Steps to Reproduce:

1. Check the checkbox that is displayed next to the "Instance" record which is linked to "MARC Authority" records on the search result pane.
   For example: "The algebraic theory of modular systems / by F.S. Macaulay.".

2. Click on the "Actions" button on the second pane and select "Export Instances (MARC)" option.

3. Go to "Data export" app.

4. Download exported ".mrc" file by clicking on the "File name" column value.

5. Open downloaded ".mrc" file via "MarcEdit" (or any similar app).

6. Delete the linked field (see precondtions): "830" field

7. Save the edited ".mrc" file.

8. Go to the "Data import" app >> Click on the "or select files" button >> Select the updated ".mrc" file (see previous step) >> click the "Open" button

9. Click on the created in precondition "Update MARC Bib by matching 999 ff $s subfield value (830 - update all)" job profile >> Click on the "Actions" in the appeared third pane >> Select "Run" option >> Click on the "Run" button in appeared modal.

10. Find updated record in "Inventory" app

11. Click on the "Actions" button and select "Edit MARC bibliographic record" option from the expanded menu.

Expected Results: Deleted "830" field (see step 6) is not shown.

Actual Results: Deleted "830" field (see step 6) is shown and has divided boxes (see attached screencast).

Steps to Reproduce:

1. Log into bugfest-poppy.

2. Open Data Import and look at Job 10638.

   1. Job Profile = CAH Update SRS MARC on 035$a match w/ qualifier

      1. This profile matches MARC to MARC on the incoming 035$a with a qualifier exactly matching the 035$a of the existing record : (MiFhGG)galncbln000092 -> galncbln000092.

      2. For matches, the record is updated overriding the protection on the 856. 

   2. Sample test records = 51356_test_records.mrc

Expected Results: The job matches the incoming records to the SRS records associated with in10783235 and in10783236 and updates these records with the new 856 in the incoming MARC record.

Actual Results: The incoming records are not matched, the log for SRS MARC says 'No action', and the SRS record is not updated.  

Additional Information: I tested the Field Mapping Profile without the 856 protection and it still failed (Job 10641).  

When reviewing logs on an internal system, the error messages given note that a match is not found.

Additional testing was done on changing the 'Match criterion' of the existing record and no value was found to make the Match profile successful.  

Original testing done in an Orchid environment.

Preconditions:

• The job profile should be created for "Data import" app: "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)"

• "Match profile" should have following specified:

Field: 999
Ind 1: f
Ind 2: f
Subfield: s

• Action profile should be Update MARC Bib

• "Field mapping profile" should have following rules specified:

Update specific fields only

Field: *
In.1: *
In.2: *
Subfield: 1.

Field: *
In.1: *
In.2: *
Subfield: 2.

No Field protection overrides

• Attached "MARC Bib" record should be imported to the system via "Default - Create instance and SRS MARC Bib" job profile (it has subfields "$1", "$2" in two fields)

Steps to Reproduce:

1. Export previously imported "MARC Bib" record from Preconditions (For preconditions, use MARC_bib_Black_panther_five_nonRepeatable_linkable_fields.mrc file attached to this Jira, and import using the default Create MARC Bib and Instance job profile)

2. Open exported .mrc file in "MarcEdit" or similar editor

3. In 130 field: delete "$2" subfield and update "$1" subfield value (e.g., change "$1" value to "new")

4. In 240 field: delete "$2" subfield and add one more "$1" subfield (e.g., add "$1onemore")

5. Save updated file

6. Import updated file using "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)" job profile created in Preconditions

7. Open updated "MARC Bib" record for edit ("Actions" → "Edit MARC bibliographic record") 

Expected Results: "$2" removed from both edited fields. Following "$1"subfield values are shown in fields:

•  "130" field: "$1new"

• "240" field: "$1original $1onemore"

Actual Results: "$2" subfields are not removed from edited fields. "$1" subfields updated/added as expected

Preconditions:

• The job profile should be created for "Data import" app: "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)"

• "Match profile" should have following specified:

Field: 999
Ind 1: f
Ind 2: f
Subfield: s

• Action profile should be: Update MARC Bibs

• "Field mapping profile" should have following rules specified:

Update specific fields:

Field: *
In.1: *
In.2: *
Subfield: 1.

Field: *
In.1: *
In.2: *
Subfield: 2.

No overrides to MARC field protection

• Attached "MARC Bib" record should be imported to the system via "Default - Create instance and SRS MARC Bib" job profile

Steps to Reproduce:

1. Export previously imported "MARC Bib" record from Preconditions 

2. Open exported .mrc file in "MarcEdit" or similar editor

3. Add "$1" subfield values in with no "$1" (e,g.:

   1. add "$1555555" to "035" field

   2. add "$1test" to "264" field

4. Save updated file

5. Import updated file using "Update MARC Bib records by matching 999 ff $s subfield value (subfields 1, 2)" job profile created in Preconditions

6. Open updated "MARC Bib" record for edit ("Actions" → "Edit MARC bibliographic record") 

Expected Results: "$1" added to edited "035", "264" fields. Fields were not duplicated 

Actual Results: "035", "264" fields were duplicated. First copy of each remains unchanged (no added subfield), while second copies contain added "$1" subfield

Preconditions:

• The job profile should be created for "Data import" app: "Update MARC Bib records by matching 999 ff $s subfield value (240, 600)"

• "Match profile" should have following specified:

Field: 999
Ind 1: f
Ind 2: f
Subfield: s

• Actin profile should be: Update MARC Bibs

• "Field mapping profile" should have following rules specified:

Update specific MARC fields

Field: 240
In.1: *
In.2: *
Subfield: 1.

Field: 240
In.1: *
In.2: *
Subfield: 2.

Field: 600
In.1: *
In.2: *
Subfield: 1.

Field: 600
In.1: *
In.2: *
Subfield: 2.

No overrides to the existing field protections

• Attached "MARC Bib" record should be imported to the system via "Default - Create instance and SRS MARC Bib" job profile

Steps to Reproduce:

1. Export previously imported "MARC Bib" record from Preconditions 

2. Open exported .mrc file in "MarcEdit" or similar editor

3. Add "$1" and "$2" subfields to "240" field (e.g., add "$1test$2testing")

4. Add "$1" and "$2" subfields to one of the "600" fields (e.g., add "$1test$2testing" to the third "600" field)

5. Save updated file

6. Import updated file using "Update MARC Bib records by matching 999 ff $s subfield value (240, 600)" job profile created in Preconditions

7. Open updated "MARC Bib" record for edit ("Actions" → "Edit MARC bibliographic record") 

Expected Results: "$1", "$2" added to edited "240", "600" fields. Fields were not duplicated 

Actual Results: "240", "600" fields were duplicated. First copy of each remains unchanged (no added subfields), while second copies contain added "$1", "$2" subfields