2021-12-14 Bulk Edit Working Group Meeting Notes

Housekeeping

• Please add your name to the attendees list
• Should we cancel our next meeting (December 28)?

• Magda: Please add your names to the attendee list in case I need to get back to you later.
• Magda: December 28th meeting is canceled. Most US libraries are closed this week.
• Our next meeting will be on January 11, 2022.

Announcements

• Product Council recommended including Bulk Edit Pilot in the Lotus release. 
• The technical council requested additional documentation - which is expected considering that the deadline for the complete platform modules is February 4th 

• Magda: Bulk Edit was presented to the Product Council a couple of weeks ago. They recommended including the Bulk Edit Pilot into the release and submitted, the JIRA for the Technical Council who requested additional documentation which was expected considering that we are still in the active development process. The development deadline for modules that are part of the complete platform, which Bulk Edit falls into is February 4th. I don't expect us to write the code until February 4th, but we definitely will be using time for checking if there are some missing requirements and things that need to be addressed before we start the integration testing.

Development updates

• Development status
• Development team scrum board
• Designated environment:
  • • RANCHER-63
    • Example of the test data

• Magda: There are not that many changes to the UI. You can now choose a file to upload and get a response to choose the file but then you will receive an error message that the file failed to upload. The reason for this is that there is a lot of backend work that is happening, and we can see this on the development team scram board. There are a lot of stories that have been completed but just a handful of the UI stores have been implemented because of back-end dependencies.
• Magda: If you want to see what is done on the backend, you can click on the user stories and look at the comments where developers update step by step what the requests and responses are on the backend. This is a good way for me to validate the story to make sure it is working as expected.
•  Magda: But for those who would like to venture into accessing the functionality using the backend directly using the API calls, this will be helpful as well.

• Magda: In addition, when the development progresses and we see changes in behavior we can go back to the original story to see how it was implemented.

• Magda: So, not much to see on the front-end in the UI. The progress was mostly on the backend. Hopefully, by the end of the week which is the end of the sprint, we will be able to see more functionality here in the UI.

• Magda: The Rancher Environment:

  • I requested a separate environment for us to do user acceptance testing. It will not be refreshed nightly, rather only on request, and we'll have a little bit more data than is currently in the snapshot environment.

  • The URL is https://concorde.ci.folio.org/settings (diku_admin/admin) for now. It may change.

  • There are still issues with this environment. It is missing some data. Only has a few modules. Obviously, we will need to have a Bulk Edit which is not currently there. But when you click on the users, there are 35,000 records right now.

  • Do you feel this is enough data?

  • Question from Erin: Is the record count at the top actually correct because that's not always?

    • Magda: It is still running Postgres, but I spoke with the developer and the number of user records is close to 35K and not less.
  • Erin: Yes, I think so (in response to Magda's question about the number of records being enough).
  • Magda: We can start with this and see how it goes. My concern was about performance. The development will be completed by next Friday and we can talk about this at our next meeting.

Mitigation of Bulk Edit UX issues 

Slide 16:

• Long term -  we will support both in app and local editing
• Short term  - local editing for the pilot program

Slide 17:

• Provide a more transparent way of understanding what FOLIO did when batch processing
• Better understanding of what a failure means -- did it revert the record, delete the record, invalidate the record?
• Itemize failures so it’s easy to identify affected records, e.g. “record #XXXXXX failed,” not “7 of 10,000 changed failed”
• Pay particular attention to global failure where feedback is especially spare
• Give some indication, if possible, of how failure can be avoided
• Provide some indication on how to re-try or follow-up from a failure

Slide 19

• Optimize .csv files for understandability
• Make it very clear which fields can or should be edited and which should be avoided  (in app or local edits?)
• Provide some degree of instruction on how to make changes to commonly edited fields like status and location (in app?)
• Give users the power to rollback changes upon upload if files are corrupted or mistakes were made (pre-update records stored in S3)
• Anticipate potentials error and provide warnings, e.g. “Is this correct?,” “Are you sure?”

Mockup- file upload | Mockup - are you sure | Mockup - delete modal | Mockup - delete errors

Mockup 7 - download matched records

• Magda: This is the example of saving the file with the records that we marched with the standard naming convention displayed at the bottom.

Mockup 8 - actions dropdown start bulk edit

• Magda: This is an example of starting the bulk edit process.

Mockup 9 - bulk edit modal

• Magda: Example of uploading the file from your local machine with changes you made locally.

Mockup 10 - drag and drop

• Magda: This shows the process of dragging and dropping the file to upload. You can also browse to the file via your local computer's file system.

Mockup 11 - bulk edit confirm

• Magda: You uploaded the file with your changes. This message is letting the use know that once you click save & close, 300 records will be changed.
• Question from Dan: Are the records you are working on locally locked in the system while you are doing this?
  • Magda: No (I did not hear any elaboration on this point that Dan brought up)

Mockup 12 - progress bar

• Magda: The uploaded file is being processed on the backend.
• Magda: The progress bar is provided to monitor the progress.

Mockup 13 - success

• Magda: Process completes with a message to the user that 297 records were changed the way you specified.

Mockup 13 - success2

• Magda: However, there were some errors.

  • One of the barcodes that you provided was not unique. So the record was not processed, and the record in the FOLIO was unchanged from what it was before you began the process.

  • You provided invalid data. You wanted to change some of the users to be active, but you provided populated it with the word "Yes" when the value should be true or false.

  • Preferred contact data was also invalid. You provided the word "Home" when valid values can only be "email" "mail" or "text."
• Question from Thomas: With the data upload, do you have to return the full CSV file, or can you just return the fields you want to be updated? By only sending the data you want to be updated lowers the chance that you would change another field that may have been changed in the system by another user in the interim when you were working on the file locally.
  
  • Magda: This is a very good point and I will bring it up with the developers.
• Question from Erin: If you are not sending back all the fields you have to be very careful that your mappings are absolutely correct. Example: sending 500 email addresses mapped to the first name field.
• Comment via chat from Christie: There's also the chance of data corruption introduced by the tool that you are editing the data with outside of FOLIO. So I think the safest option is to send just the data that has changed.
• Comment from Bob: One system that I use has a mapping confirmation process built into the interface. You can see it here: https://help.memberclicks.com/hc/en-us/articles/230535007-Profile-Import.
• Comment from Sara: It's not even that we have to introduce corruption. It can also be even within a very short amount of time, not days of having the file out and then putting it back. Somebody finds a typo in a person's name and fixes it. And then if I overlay the record completely, then I just put the error back. I think that's also something we have to keep in mind that going forward is that we can have the option to either put or patch and we should be able to distinguish between the two. Sometimes you want to totally overlay a record. Then also very often you just want to correct a very specific part. And I think we do need to have that capability of being able to do that safely. To add a line without saying that we're going to overlay it all again. I can do that with my current system.
  
  • Magda: When you say add a line do you mean add a new record?
  • Sara: No I mean adding for example emails to records where that data element is missing in some of the records.
  • Magda: So when you get the file from the system, you will you will have the email column. In your example, the email column will be empty. So then you would populate that.
  • Sara: Are you were saying that all columns are always present, whether they have value, values associated with them?
  • Magda: Yes.
• Magda: I would like to make a comment about the patch and put in post methods. To be honest, I'm not sure in FOLIO if we support patch. Maybe we do, but I know most APIs that I worked with support the post method the new records and put for modifications.

• Magda: I see the problem with what Sarah mentioned, someone changed or corrected the name of the user. And we are overwriting the data. And to be honest, at this point, I don't have a quick and easy solution for that. This may happen. Definitely.

Survey review:

• Bulk Edit Limit Size
• Bulk Edit Permissions

Future discussion topic (time permitting):

• Bulk Edit of the linked records
• Scheduling edits