Batch Importer (Bib/Acq) (UXPROD-47)

[MODDICONV-300] Match and action profiles cannot be re-used in an import job profile - short term fix (Poppy) Created: 07/Mar/23  Updated: 12/Oct/23  Resolved: 19/Jun/23

Status: Closed
Project: mod-di-converter-storage
Components: None
Affects versions: None
Fix versions: 2.1.0
Parent: Batch Importer (Bib/Acq)

Type: Bug Priority: P2
Reporter: Christie Thomas Assignee: Volodymyr Rohach
Resolution: Done Votes: 0
Labels: architectural, arlef-di, data-import, epam-folijet, folidemo166-167, folijet-olamide, has-testrail, support
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Snapshot_NewProfileInEditor.png     PNG File Snapshot_SavedProfile_1.png     PNG File Snapshot_SavedProfile_2.png    
Issue links:
Cloners
is cloned by MODDICONV-317 Match and action profiles cannot be r... Closed
is cloned by MODDICONV-318 Match and action profiles cannot be r... Closed
Defines
defines UXPROD-3838 NFR: Data Import & Inventory Support ... In Review
Relates
relates to MODDICONV-310 Add wrappers around profiles to build... Closed
relates to MODDICONV-311 Migrate existing profiles to have wra... Closed
relates to FAT-6310 Review of C385654 Closed
relates to FAT-6311 Review of C385653 Closed
relates to FAT-6312 Review of C385629 Closed
relates to MODDICONV-302 Cannot view details of a job profile ... Closed
relates to MODDICONV-326 Cannot view details of a job profile ... Closed
Sprint: Folijet Sprint 166, Folijet Sprint 167
Story Points: 1
Development Team: Folijet
Release: Poppy (R2 2023)
CSP Approved: Yes
CSP Request Details: Nolana/Orchid CSP requested 21 June 2023
Approved 22 June 2023 by Khalilah, Mike G, Kristin M, Mark V, Debra H, Harry K
Potential Workaround: Work includes creating new match profiles with the same match point but a different name. This is the work around that worked until Nolana when only the primary match point had to be unique, but after Nolana even sub-matches must be unique profiles.
Affected Institution:
!!!ALL!!!, University of Chicago
Epic Link: Batch Importer (Bib/Acq)
RCA Group: Incomplete/missing requirements
Affected releases:
Orchid (R1 2023), Nolana (R3 2022)

 Description   

Overview: Data import job profiles can be created using the same match profiles more than once, but once the job profile is saved the reused components are repeated under each section. For instance, Chicago needs to create a job profile that matches the 035$z of the incoming record to the OCLC number in the instance, then submatch to the instance status is equal to a constant value (e.g., Batch Loaded), and then submatch to the holdings type (e.g., Electronic). If there is not match on the 035$z to the OCLC number, then the import should match the 035$a of the incoming record to the OCLC number in the instance record with an instance submatch where the instance status is equal to a constant value and the holdings type is equal to a constant value. The submatches are the same for each of the components. This is not currently possible without creating unique match profile for each time a match is needed to be used in an import job profile.

This bug does not require importing anything, only building the various profiles and assembling the job profile

Steps to Reproduce:

  1. Log into Folio Snapshot
  2. Go to Settings > Data Import.
  3. Create a new field mapping profile for an instance and save it
    1. Name: Update instance
    2. Cataloged date: Select Today from dropdown list
    3. Instance status: select any value from the dropdown list
    4. Administrative notes
      1. Add these to existing
      2. "Updated via OCLC match"
  4. Create a new field mapping profile for a holdings and save it
    1. Name: Update holdings temp location
    2. Administrative notes
      1. Add these to existing
      2. "Updated via OCLC number match"
  5. Create a new action profile with the following options:
    1. Name: Update instance via OCLC number match
    2. Action: Update
    3. FOLIO record type: Instance
    4. Link the instance field mapping profile that you just created. 
  6. Create a new action profile with the following options:
    1. Name: Update holdings via OCLC number match
    2. Action: Update
    3. FOLIO record type: Holdings
    4. Link the holdings field mapping profile that you just created. 
  7. Create a new match profile named 035$a to OCLC with the following options:
    1. Select MARC Bib-to-Instance under existing records
    2. Field: 035
    3. In. 1: [space]
    4. In 2: [space]
    5. Subfield: a
    6. Match criterion: Exactly matches
    7. Existing instance record field: Identifier: OCLC
    8. Save and close
  8. Create a new match profile named 035$z to OCLC with the following options:
    1. Select MARC Bib-to-Instance under existing records
    2. Field: 035
    3. In. 1: [space]
    4. In 2: [space]
    5. Subfield: z
    6. Match criterion: Exactly matches
    7. Existing instance record field: Identifier: OCLC
    8. Save and close
  9. Create a new match profile named instance Status Batch Loaded
    1. Change MARC Bibliographic to Static value (submatch only)
    2. Select Instance under existing records
    3. Incoming Static value (submatch only) record - Text: Batch Loaded
    4. Match criterion: Exactly matches
    5. Existing instance record field: Admin data: Instance status term
    6. Save and close
  10. Create a new match profile named Holdings type electronic
    1. Change MARC Bibliographic to Static value (submatch only)
    2. Select holdings under existing records.
    3. Incoming Static value (submatch only) record - Text: Electronic
    4. match criterion: Exactly matches
    5. Existing holdings record field: Admin data: holdings type
    6. Save and close
  11. Create a new job import profile with any name and accepted date type = MARC Bibliographic. (See screenshot in edit mode for what the profile should look like before saving.)
  12. Click the plus sign and add a new match: 035$z to OCLC. 
    1. Under For matches add a new match profile: Instance Status Batch Loaded
    2. Under For matches add a new match profile: Holdings type Electronic.
    3. Under For matches add a new action profile: [the update instance profile created]
    4. Under For matches add a new action profiles: [the update instance and update holdings profiles created].
  13. Under the third For non-matches heading, click the plus sign and add a new match profile: 035$a to OCLC then add the same sub-match profiles and actions as under the 035$z to OCLC match above.
    1. Under For matches add a new match profile: Instance Status Batch Loaded
    2. Under For matches add a new match profile: Holdings type Electronic.
    3. Under For matches add a new action profile: [the update instance profile created]
    4. Under For matches add a new action profiles: [the update instance and update holdings profiles created].
  14. review the screen to see that match profile holdings type is present only once under each OCLC match and the update instance and update holdings actions are listed only once under each OCLC match. 
  15. Save the job import profile.

 Expected Results: I expect to see a job profile that will match the 035$z to Instance OCLC number, instance status, and holdings type then update the instance and and the holdings if it is a match. If there is no match I expect to see a match on the 035$a to the instance OCLC number, then submatches to the instance status and holdings type with update actions taken if there is a match. I expect to be able to build complex matches that re-use match profiles within a single job import profile.

Actual Results: Each primary match 035$z to OCLC and 035$a to OCLC  each contain two matches to holdings type and each holding type electronic match contains two update instance and two update holdings actions.

Additional Information: This behavior is present in Nolana and FOLIO snapshot.

Interested parties: University of Chicago

 

BE Notes:

  1. Existing workaround - create job profile step by step - save one branch (building block) with links set up, and then edit it again adding other branch using the same building block. It works, but required steps that user has to make are ugly
  2. Short-term, easy to implement and safe to back-port workaround in the code - add constraint (or BE validation) when creating associations to ignore creation of duplicate association in scope of one job profile. It will allow creating profiles reusing the same building blocks (match and action profile structure). Such fix will be incomplete though - described case will be covered, but in case different building blocks would be required using same associations, we still might run into a problem.{} This solution will be implemented in scope of the current ticket.
  3. Adequate long-term solution - association entity currently stores parent and detail profile ids, to ensure the association is unique, instead of actual profiles ids we need to store ids of profile wrappers (new entity) in parent and detail columns. Add table for profile wrappers - wrapper id, profile type, profile id. Changes are needed in operations over associations themselves and in functionality constructing the profile, and migration. See stories MODDICONV-310 Closed , MODDICONV-311 Closed . It is the most preferable way to go, but it will take something like 13 SP and will be too risky for back-port (will be done in Poppy). 

ORCHID Critical service patch details

  1. Describe issue impact on business: Work done in Nolana to stabilize job profiles inadvertently decreased the ability to use the same matches and same actions multiple times in the same job profile. This short-term fix allows for the same matches and actions to be reused in a job profile as long as they are applied in the same relationship to each other in all sections of the profile. Longer-term, the intent is to adjust the job profiles so that matches and actions can be applied without being in parallel relationships throughout the job profile. However, per the development thoughts above, that long-term solution is significantly more work and is too risky to backport.
  2. What institutions are affected? (field “Affected Institutions” in Jira to be populated): Any that use complex, branching job profiles
  3. What is the workaround if exists?: None that are simple. One used in the past is to create multiple duplicative match or action profiles with the same details, but different names, and then refrain from using the same match or action profile more than once in a job profile. However that involves significant work for the DI users, especially when profiles are updated, and duplicative match/action profiles must be reviewed to ensure they are kept synchronized. It also introduces possibility of error, if the duplicative profiles are not kept synchronized in terms of any manual edits.
  4. What areas will be impacted by fix (i.e. what areas need to be retested): Data import profile creation and import with the 3 TestRails associated with this Jira. Plus testing by SMEs at Cornell and Chicago to review/confirm.
  5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high). The work is complete and consumed 1 SP of effort. Significant effort went into teasing apart a longer-term solution and documenting on separate Jiras, plus creating TestRails and manually testing. Risk is minimal.
  6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO. Test cases already exist and level of effort is .5-1 day, spread across manual testers, PO, and SMEs
  7. What is the roll back plan in case the fix does not work? The patch can be rolled back if necessary, and re-evaluated

NOLANA Critical service patch details

  1. Describe issue impact on business: Work done in Nolana to stabilize job profiles inadvertently decreased the ability to use the same matches and same actions multiple times in the same job profile. This short-term fix allows for the same matches and actions to be reused in a job profile as long as they are applied in the same relationship to each other in all sections of the profile. Longer-term, the intent is to adjust the job profiles so that matches and actions can be applied without being in parallel relationships throughout the job profile. However, per the development thoughts above, that long-term solution is significantly more work and is too risky to backport.
  2. What institutions are affected? (field “Affected Institutions” in Jira to be populated): Any that use complex, branching job profiles
  3. What is the workaround if exists?: None that are simple. One used in the past is to create multiple duplicative match or action profiles with the same details, but different names, and then refrain from using the same match or action profile more than once in a job profile. However that involves significant work for the DI users, especially when profiles are updated, and duplicative match/action profiles must be reviewed to ensure they are kept synchronized. It also introduces possibility of error, if the duplicative profiles are not kept synchronized in terms of any manual edits.
  4. What areas will be impacted by fix (i.e. what areas need to be retested): Data import profile creation and import with the 3 TestRails associated with this Jira. Plus testing by SMEs at Cornell and Chicago to review/confirm.
  5. Brief explanation of technical implementation and the level of effort (in workdays) and technical risk (low/medium/high). The work is complete and consumed 1 SP of effort. Significant effort went into teasing apart a longer-term solution and documenting on separate Jiras, plus creating TestRails and manually testing. Risk is minimal.
  6. Brief explanation of testing required and level of effort (in workdays). Provide test plan agreed with by QA Manager and PO. Test cases already exist and level of effort is .5-1 day, spread across manual testers, PO, and SMEs
  7. What is the roll back plan in case the fix does not work? The patch can be rolled back if necessary, and re-evaluated


 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 08/Mar/23 ]

Hi Christie Thomas I was definitely able to reproduce this on Snapshot. Thank you for writing it up!

Mariia Aloshyna and Kateryna Senchenko please review. Per Christie, we've made the work-around more of a headache as of Nolana. Is there any simple way we can allow the same match and action profiles to be used more than once in a job profile, without having to duplicate the match/action profiles and give them different names?

Comment by Ann-Marie Breaux (Inactive) [ 30/Mar/23 ]

Ann-Marie Breaux Add something on the wiki that the same match/action/field mapping profile can be used more than once in a job profile (in tips and tricks or in Folijet dev reqs area)

Comment by Ann-Marie Breaux (Inactive) [ 05/Apr/23 ]

Folijet refinement: BE needs to review more and discuss. Also consider whether it could be released as a Nolana and/or Orchid patch, depending on the complexity and risk

Comment by Jenn Colt [ 10/Apr/23 ]

As we work more in Nolana I find we are encountering more and more of the same problems Christie has. I used to be able to create two checks for matches and have each branch end in a create if there is no match and now I can't. I didn't realize that was what the problem was because the profile saved with no problem it was only after I went back and looked at it that I realized it had removed the duplicated actions with no warning at all. This will mean increasing even further the number of profiles that we have.

Comment by Ann-Marie Breaux (Inactive) [ 11/Apr/23 ]

Hi Jenn Colt I'm really sorry to see that. We'll try to pull this bug into the current sprint and work on it ASAP. Has this caused profiles that worked for you in Morning Glory to no longer work in Nolana? If so, it sounds like we should also consider a patch for Nolana and Orchid.

On Orchid bugfest, would it be possible to create a sample profile that worked previously, but is no longer working, and attach a relevant file to this bug?

Thank you so much

Comment by Maksym Ishchenko [ 13/Apr/23 ]

Hello Ann-Marie Breaux 

Could you create scenarios for the 2nd and 3rd "Checks" in BE notes and for Edits and Duplicates scenarious?

Comment by Maksym Ishchenko [ 07/Jun/23 ]

Hi Ann-Marie Breaux 

This ticket was checked on snapshot env. No additional profiles are appears now when creating/duplicating/editing job profiles with repeatable match/action profiles. Moving issue to In review column 

Comment by Ann-Marie Breaux (Inactive) [ 19/Jun/23 ]

Thanks, Maksym Ishchenko Looks good to me also. Thank you for the work, Volodymyr Rohach

Christie Thomas and Jenn Colt Please give this a try on Snapshot when you have a chance. The current solution (which we'll aim to backport to Orchid and possibly Nolana) should resolve the type of profile that Christie has outlined. Per the developers, the described case will be covered, but in case different building blocks would be required using same associations, we still might run into a problem.

And in Poppy, we will implement a long-term solution that will resolve it despite different associations. However, the developers deem that fix as too risky to backport to Orchid or Nolana.

Generated at Thu Feb 08 23:30:19 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.