Batch Importer (Bib/Acq) (UXPROD-47)

[MODINVSTOR-1093] Data import: circulation notes created via import cause blank references, rendering items uneditable within the UI (short term fix) (Orchid CSP 6 Clone) Created: 11/May/23  Updated: 13/Oct/23  Resolved: 21/Sep/23

Status: Closed
Project: mod-inventory-storage
Components: None
Affects versions: None
Fix versions: 26.0.1, 27.0.0
Parent: Batch Importer (Bib/Acq)

Type: Bug Priority: P2
Reporter: Molly Driscoll Assignee: Maksym Ishchenko
Resolution: Done Votes: 0
Labels: 2, data-import, epam-folijet, has-testrail, known-issue-nolana, orchid-patch-approved, support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File TestCircNotes.mrc     File circnotetesting2.mrc     PNG File image-2023-08-28-11-56-09-093.png     PNG File image-2023-08-28-12-59-28-228.png     PNG File image-2023-08-28-13-00-20-750.png     PNG File image-2023-08-28-13-01-01-148.png    
Issue links:
Blocks
blocks MODINVSTOR-1096 Items with multiple circulation notes... Closed
blocks FAT-7602 Review of C409419 Closed
blocks MODINVSTOR-1107 Release v26.0.1 (Orchid R1 2023 CSP #6) Closed
Cloners
is cloned by MODINVSTOR-1110 Data import: circulation notes create... Closed
Continues
is continued by UIDATIMP-1384 Provide "required" field for subfield... Closed
is continued by UIDATIMP-1516 Provide "required" field for Receivin... Draft
Defines
defines UXPROD-3838 NFR: Data Import & Inventory Support ... In Review
Relates
relates to MODDICONV-297 Add migration script for profiles wit... Closed
Requires
is required by FAT-8240 Review of C410938 Closed
Sprint: Folijet Sprint 172, Folijet Sprint 174, Folijet Sprint 173
Story Points: 2
Development Team: Folijet
Release: Orchid (R1 2023) Service Patch #6
CSP Approved: Yes
CSP Request Details: Orchid CSP requested 1 Sept 2023 on Slack. Followed up 5, 12, 19 Sept. Approved 21 Sept by ...
Potential Workaround: From Molly: Labeled a P2 because the workaround is onerous – must remove notes via data import or API.
Affected Institution:
!!!ALL!!!
Epic Link: Batch Importer (Bib/Acq)
RCA Group: Implementation coding issue
Affected releases:
Orchid (R1 2023), Nolana (R3 2022)

 Description   

Overview: Data import field mapping profiles for items allow the addition/update/removal of circulation notes. However, when adding circulation notes to items via data import, null ID values are added to the circulation notes. This prevents records from being editable in the UI after the fact. Once the circulation notes are removed via import or API, the records are editable again.

Here is a sample JSON response:

"circulationNotes": [
       {
           "id": null,
           "noteType": "Check in",
           "note": "Check for discs.",
           "staffOnly": true,
           "source":

{                "id": null,                "personal": null            }

,
           "date": null
       },
       {
           "id": null,
           "noteType": "Check out",
           "note": "Check for discs and confirm number with patron.",
           "staffOnly": true,
           "source":

{                "id": null,                "personal": null            }

,
           "date": null
       }
   ],

Steps to Reproduce:

  1. Log into https://bugfest-orchid.int.aws.folio.org/ as a user with admin permissions.
  2. Open the Inventory app to the item search segment.
  3. Narrow to material type = molly
    1. Two records should appear.
  4. Click Actions > Save instance UUIDs
    1. A CSV file will be downloaded to your machine containing two instance UUIDs.
  5. Open the Data export app.
  6. Upload the CSV file that downloaded from step 4.
  7. Run using the SRS + Item HRID data export job profile (0e0599dc-1ac9-41ae-9814-c6ca7aa3e05c) and indicating you've provided instance UUIDs.
  8. Download the resulting .mrc file.
    1. This should have item HRIDs in the 900ff$a.
  9. Open the Data import app.
  10. Upload the .mrc file that was saved to your computer during step 8.
  11. Run using the Molly Circ Notes Test profile (3a0c92b1-1e0d-40a4-9fa4-00e118d652c3).
    1. This profile matches incoming 900ff$a to the item HRID with a static submatch to 'molly' as the material type, then adds check out and checkin notes to the item records when a match is found.
  12. Once the job completes, open the log, and click on one of the updated items.
    1. This will route you to the item record in Inventory.
    2. You should see the circulation notes on the record.
  13. Click Actions > Edit.
  14. Try to make a change to the record, then Save & close.

Expected Results: Item record saves with edits.

Actual Results: The save & close button is grayed out for a time, but then resets and the record is not saved. 

The dev tools network tab shows a canceled status for the PUT request with "Failed to load response data: No data found for resource with given identifier.

The console shows request timeout errors.

Additional Information: Originally discovered in a Nolana environment, escalated to EBSCO hosting who diagnosed the null values.

Replicated on Orchid Bugfest, so escalating to Support SIG for resolution within the Community.

BE: When Item is created/updated in inventory-storage - check for circ notes and generate UUIDs for each note that does not have the id assigned

Test info

  1. Test 1: Create 2 items, 1 with mapped circ notes, 1 without
    • Instance: use default create instance profile
    • Holdings field mapping: default a permanent location, no other data
    • Item field mapping: default Material type, perm loan type, status
    • Map the Circulation notes from the incoming file
      • Default: Note type: default to Check-out note type, 901$aCirc note text, default the staff only only checkbox
    • In the MARC file, have 2 records
      • Record 1: has 1 901 fields: with 901$a only
      • Record 2: has NO 901 field
    • Import to create instance, holdings, item
    • Results
      • Record 1: should have a check-out note, with text from 901$a, and marked staff only
      • Record 2: should not have any circulation note: no circ note type, no text, no staff only checkbox
  2. Test 2: Update those 2 items
    • Export those 2 instances, with item HRID in 900$a
    • Edit the export records
      • Record 1: remove the 901
      • Record 2: add a 901$a with check-in text
    • Match 900$a to item record HRID
    • Item action: Update item
    • Item field mapping:
      • Circ notes: Remove existing and add new
      • Mapping: Note type: default to check-in note, with text from 901$a, staff only: do NOT mark the checkbox
    • Create import profile
      • Match to item
      • Update item
    • Results:
      • Item 1: Check-out note is removed, and there's no new circ note added to the item, because no 901$a in the incoming record
      • Item 2: No note to remove, but new check-in note added to the item, using defaults and 901$a data

ORCHID Critical service patch details

  1. Describe issue impact on business If a field mapping references a MARC field as one with notes data, and there is no notes data, it will create a note field in the item with a blank note, AND make the item record un-editable via the UI. That is a significant data issue. This fix allows the item to be editable via the UI, but we're making a more significant fix in Poppy that will not create the item note field at all, if there is no actual note text to populate into it. So this is a short-term fix, which creates the item circ note field (with note type and staff only values, but no note text) and allows the item to be edited.
  2. What institutions are affected? (field “Affected Institutions” in Jira to be populated) All
  3. What is the workaround if exists? None, without editing the item in the backed via API
  4. What areas will be impacted by fix (i.e. what areas need to be retested) Data Import smoke and critical path tests, plus the manual test associated with this Jira
  5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high). Work is complete and level effort is low. Very minimal risk.
  6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO. Test cases already exist and level of effort is .5 day
  7. What is the roll back plan in case the fix does not work? Revert to previous version


 Comments   
Comment by Molly Driscoll [ 11/May/23 ]

Labeled a P2 because the workaround is onerous – must remove notes via data import or API. You can see one of the resulting records here: https://bugfest-orchid.int.aws.folio.org/inventory/view/7e9a8185-eb23-49e1-a899-0647bc90f7d0/144bdd28-bca0-4772-a137-ffaec78cd652/3175df77-3023-440f-a492-f631b4edabcd

Comment by Anya [ 15/May/23 ]

Support: Ann-Marie Breaux  please be aware of this ticket 

Comment by Ann-Marie Breaux (Inactive) [ 15/May/23 ]

Hi Molly Driscoll  Thank you for the clear explanation and example in Orchid BF! Kateryna Senchenko  and Mariia Aloshyna Please review and determine if this is a UI or BE bug. Thank you!

Comment by Molly Driscoll [ 15/May/23 ]

Ann-Marie Breaux my pleasure! I'm happy to help re-test whenever you're ready!

Comment by Debra Howell [ 03/Jul/23 ]

From SUPPOR SIG: Ann-Marie Breaux Would you please give us an update on this ticket? Thanks!

Comment by Debra Howell [ 07/Aug/23 ]

From SUPPOR SIG: Ann-Marie Breaux Would you please give us an update on this ticket? Thanks!

Comment by Maksym Ishchenko [ 22/Aug/23 ]

Hello Ann-Marie Breaux 

Ticket verified on snapshot env. Now items with circ rules created and can be edited without errors. Moving ticket to In Review

Screencasts you can find in related FAT ticket

Comment by Ann-Marie Breaux (Inactive) [ 28/Aug/23 ]

Hi Maksym Ishchenko Could you create additional TestRails that follow the Test steps outlined above? Thank you!

Comment by Ann-Marie Breaux (Inactive) [ 28/Aug/23 ]

Hi Molly Driscoll Could you test this on Snapshot and see what you think? Is it working as expected now on Snapshot? Maksym Ishchenko and @kate we should maybe look at this together at standup before closing it. Thank you!

Comment by Molly Driscoll [ 28/Aug/23 ]

Ann-Marie Breaux just tested Test#1 on Snapshot – item 1 behaves as expected. Circ notes are created and edits to the item in the UI after import are saved as expected.

Item 2 does not behave as expected. Circ notes are added with the staff only box checked, but no note text included:

Please see job 29: https://folio-snapshot.dev.folio.org/data-import/job-summary/1f7e5b5f-f48a-41a2-98b0-b4ddd54b156a

Comment by Molly Driscoll [ 28/Aug/23 ]

Here is the JSON for item 1:

{ "id": "fc57c7ca-82ff-4d9c-99de-2e9582648c37", "_version": "2", "status":

{ "name": "Available", "date": "2023-08-28T15:52:03.202+00:00" }

, "administrativeNotes": [], "title": "Absolute time : rifts in early modern British metaphysics, Part 2 / Emily Thomas.", "hrid": "it00000006562", "contributorNames": [

{ "name": "Thomas, Emily, 1985-" }

], "formerIds": [], "discoverySuppress": null, "holdingsRecordId": "815f4977-8900-4bff-8ad0-a4a983191409", "notes": [], "circulationNotes": [ { "id": "f05f2b59-f326-46b2-ac1e-87547921de11", "noteType": "Check in", "note": "This is a check in note.", "staffOnly": true, "source":

{ "id": null, "personal": null }

, "date": "2023-08-28T15:52:03.140+0000" }, { "id": "cc54b6a1-4443-45a9-973c-5a02b4d23bfe", "noteType": "Check out", "note": "This is a checkout note.", "staffOnly": true, "source":

{ "id": null, "personal": null }

, "date": "2023-08-28T15:52:03.140+0000" } ], "tags":

{ "tagList": [] }

, "yearCaption": [], "electronicAccess": [], "statisticalCodeIds": [], "purchaseOrderLineIdentifier": null, "materialType":

{ "id": "5ee11d91-f7e8-481d-b079-65d708582ccc", "name": "dvd" }

, "permanentLoanType":

{ "id": "2b94c631-fca9-4892-a730-03ee529ffe27", "name": "Can circulate" }

, "metadata":

{ "createdDate": "2023-08-28T15:52:03.202+00:00", "createdByUserId": "526f3066-8b26-579b-8cdf-3237fd477164", "updatedDate": "2023-08-28T15:52:56.512+00:00", "updatedByUserId": "526f3066-8b26-579b-8cdf-3237fd477164" }

, "effectiveCallNumberComponents":

{ "callNumber": null, "prefix": null, "suffix": null, "typeId": null }

, "isBoundWith": false, "effectiveLocation":

{ "id": "fcd64ce1-6995-48f0-840e-89ffa2288371", "name": "Main Library" }

}

 

Here is the JSON for item 2:

{ "id": "68b87a42-0717-46fc-8234-2a00aaa29337", "_version": "1", "status":

{ "name": "Available", "date": "2023-08-28T15:52:03.154+00:00" }

, "administrativeNotes": [], "title": "Absolute time : rifts in early modern British metaphysics, Part 3 / Emily Thomas.", "hrid": "it00000006561", "contributorNames": [

{ "name": "Thomas, Emily, 1985-" }

], "formerIds": [], "discoverySuppress": null, "holdingsRecordId": "8190cb08-70bf-4ac8-bc7e-b509dce930d4", "notes": [], "circulationNotes": [ { "id": "1b92bf47-3aeb-463f-93ee-4980e764d10e", "noteType": "Check in", "note": null, "staffOnly": true, "source":

{ "id": null, "personal": null }

, "date": "2023-08-28T15:52:03.120+0000" }, { "id": "38a4da42-2b0f-4b4a-b1ce-4c43a04c6ae2", "noteType": "Check out", "note": null, "staffOnly": true, "source":

{ "id": null, "personal": null }

, "date": "2023-08-28T15:52:03.120+0000" } ], "tags":

{ "tagList": [] }

, "yearCaption": [], "electronicAccess": [], "statisticalCodeIds": [], "purchaseOrderLineIdentifier": null, "materialType":

{ "id": "1a54b431-2e4f-452d-9cae-9cee66c9a892", "name": "book" }

, "permanentLoanType":

{ "id": "2b94c631-fca9-4892-a730-03ee529ffe27", "name": "Can circulate" }

, "metadata":

{ "createdDate": "2023-08-28T15:52:03.154+00:00", "createdByUserId": "526f3066-8b26-579b-8cdf-3237fd477164", "updatedDate": "2023-08-28T15:52:03.154+00:00", "updatedByUserId": "526f3066-8b26-579b-8cdf-3237fd477164" }

, "effectiveCallNumberComponents":

{ "callNumber": null, "prefix": null, "suffix": null, "typeId": null }

, "isBoundWith": false, "effectiveLocation":

{ "id": "fcd64ce1-6995-48f0-840e-89ffa2288371", "name": "Main Library" }

}

 

Interestingly, the id values associated with the source of the notes are still null, but this does not seem to prevent save of the records when editing in the UI.

Comment by Molly Driscoll [ 28/Aug/23 ]

Performed a second test mapping note type from the MARC like this: 

circnotetesting2.mrc

Comment by Molly Driscoll [ 28/Aug/23 ]

Results:

 

Comment by Ann-Marie Breaux (Inactive) [ 01/Sep/23 ]

Maksym Ishchenko Please make sure we have TestRails that match the testing descriptions laid out in the Jira description. Thank you!

Comment by Kateryna Senchenko [ 06/Sep/23 ]

For records that don't have the value specified in the mapping profile notes (with empty values) will not be created. It was fixed in scope of UIDATIMP-1384 Closed

Comment by Maksym Ishchenko [ 06/Sep/23 ]

Return ticket to "In QA" column because it's needed Testrail update

Comment by Maksym Ishchenko [ 26/Sep/23 ]

Hello Ann-Marie Breaux 

Ticket verified on Orchid Bugfest env. Circulation notes remains empty when there is no field in incoming file. Result jobs here: https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/bb54b8fa-ab3d-4d2c-b5fe-4b0cad80e364 and  https://bugfest-orchid.int.aws.folio.org/data-import/job-summary/8841f622-4731-4997-b2c6-d7c22377aba0

 

Please doublecheck and if all looks good to you close the ticket

Generated at Thu Feb 08 22:23:19 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.