2021-11-16 Bulk Edit Working Group Meeting Notes

Housekeeping:

• Please add your name to the list of attendees
• Make sure Transcript is turned on during the meeting

• Ryan: The goal of the research was to figure out how we move forward with bulk edit functionality in FOLIO by understanding some of the challenges that librarians are facing when making bulk edits, not just in FOLIO, but in all the applications that they are using, which are quite varied. I sit within the UX research team at Ebsco and I work on all kinds of products, so FOLIO is less familiar to me.
• Ryan: I wanted to understand what some of the biggest challenges were in the bulk edit process so that we can account for those challenges in Future FOLIO updates. I wanted to understand specifically what you all saw as the drawbacks and advantages to editing local files, especially CSV files, as opposed to using an in-app editor. I wanted to understand the drawbacks and advantages of both, what would tip the balance for individuals and institutions to accept and adopt a CSV bulk edit workflow?
• Ryan: The goal was not to dig into the specific problems with any particular UI but to understand the challenges that you all face generally in performing bulk edits.
• Ryan: The research was conducted over a series of about two weeks. I did moderated interviews, which were open-ended questions designed to identify the bulk edit workflow and understand the toolset that is used for making those edits.
  
  • Were there student workers involved and how does that add an additional layer of challenge to the process?
  • How they reacted to being able to do local edits in FOLIO if that were an option and what the requirements would be for this option.
• Ryan: People interviewed were librarians involved with metadata management and bulk editing. Spoke with one K-12 library assistant. So it was a varied group considering how specific this audience is. This group was using a number of different platforms.
• Ryan: What are the example use cases for bulk editing uncovered via the interviews?
  • All of these cases would require bulk editing of statuses or locations:
    1. Graphic novels spread out throughout a large academic library combined into a single special collection.
    2. An academic library is moving upwards of 180,000 physical holdings to an offsite annex.
    3. Volumes are being shifted from within the same building to a different floor.
    4. Items sent to the bindery
    5. Items being withdrawn
• Ryan: How does bulk edit work?
  • Records need to be identified which can happen using one of two methods:
    • Queries - platform is used to query the DB based on some criteria, e.g., location.
    • Itemized - records in the DB are identified using an existing list of records, e.g., barcodes or another identifier.
    • In reality, it is often a combination of these two methods.
  • Once identified the changes can be made in one of two ways:
    • Native editing - the platform has the ability to make changes to records directly within the app.
    • Local editing - records are harvested, exported, edited locally in an external editor, and then re-uploaded back into the DB.
    • There was a lot of feedback on these two options.
• Ryan: Advantages of in-app native editing: data is normalized in the system and hides data that does not need to be changed for the specific edit being done.
• Ryan: The disadvantage of native editing: confined to what can be done on the platform, maybe only one edit at a time. You are not given access to the raw data to edit.
• Ryan: The disadvantage of local editing: dealing with raw data files exposes a lot of meaningless data presented in a file that is chaotic and difficult to read. And it is not always clear what fields need to be edited to enact the necessary changes, all of which can lead to errors. And there is little or no feedback regarding the errors from changes made and no opportunity to roll back changes if errors were made. There is a lot of room for error.
• Ryan: Feedback for example from UMASS - working directly with CSV files feels dangerous. So much can go wrong. When you bring data back in, it's hard to know what was changed. And if there is an error or mistake, it's difficult to track down.
• Ryan: As a group, across the board, there was enthusiasm for having some sort of in-app solution that can streamline workflows, reduce risks of bulk editing, normalize some of the behind-the-scenes processes, and make everything easier.
• Ryan: When speaking with the Library Assistant from the K-12 school, she said doing local edits makes it easier because I don't have to give admin rights to all the students I'm working with. They can save and hand off the file. I'm the only one who does the actual uploading. This scenario is where there are mostly volunteers doing the editing and the scope is small, maybe 10-100 records to edit at one time. In comparison, none of the academic libraries interviewed employ students to do bulk edits because the work is too complex and the possibility for error is just far too great to have students involved.
• Ryan:  I think based on what I heard from you all, having some sort of in-app editor that cuts through all that data level noise is going to be the best long-term option for user workflows and customer satisfaction. Of course, this option requires a lot more development.
• Ryan: In the short term, what I heard from folks is that they could accept a local bulk edit option if it was explicitly positioned as a stop-gap measure on the way to getting an in-app solution. And if a few features were put in place to reduce the risk of errors and increase feedback when there are errors in that process.
• Ryans: So one thing we need to focus on is providing really clear error reporting and a transparent way people can understand what's happening. And we also need to know what failure means, e.g., did it revert, invalidate, corrupt, or delete the record? What is the status of the records that failed to process? And when there are failed records, there is a way to identify the bad records from within the potentially very large set of processed records. Also, some feedback, if possible, on how to prevent this from happening again as well as some guard rails from the onset so it does not feel so dangerous.
• Ryan: We also need to make sure that we're optimizing the CSV files for understandability, e.g., avoiding any kind of Ebsco-centric jargon or codes, and label the data in a way that correlates to the records we are changing.
• Ryan: We want to make it very clear which fields should be edited and which should be avoided. And that might mean providing some degree of instruction on some of the more common edits like status and location fields; for example, indicating that if you want to change the status or location of an item, these are the fields you have to change, and ignore everything else. And maybe even by making some of those fields unavailable for editing.
• Ryan: And if you want to undo the effects of a bulk edit there should be an undo feature to restore the data to a previous state.
• Ryan: Provide some warnings if the data uploaded is problematic before the process is run, i.e., "are you sure?," "is this correct?" Having some guardrails around the process, so it doesn't feel so obscure and opaque and there is some feedback happening.
• Ryan: In the longer term for the in-app development we can return to the group for concept testing, usability testing, so we can gauge reactions and get a sense of what people like and what they don't like.

• Ryan: And for the CSV local editor option in the interim, we just make it really, really clear that this is short-term and what our roadmap looks like. We want to make sure that in the local editing process, we have crystal clear error reporting and feedback and reassurance to really help understand what's happening. Make sure there are instructions on exactly what fields need to be amended to make common changes. And we want to have some improvements to be able to roll back when there are human or computer errors. And we want to provide the user tools to be able to retrace their steps and amend any corrupted records, just to try to make this process a little bit easier, a little bit safer, and a little bit less difficult.

• Ryan: And I would just try to communicate that being presented with a local editing option now does not mean this will be the only thing you will ever have. And that there is more on the roadmap to come and it is being worked on.

• Amanda: Is there a timeline for what you mean by short-term and long-term?
• Ryan: I cannot in good faith say what that would look like. I'm not sure if anyone else on the call can speak to that?

• Magda: In-app will not be possible for the pilot project. I hear the feedback loud and clear that the local edit option is not acceptable long-term and we will begin looking at that.

• Scott Perry: I see the value of both options. I just want to make sure there is a way to go broader with all sorts of data, not just metadata but also for example financial and acquisitions data. I would also encourage that permissions to be area or module based not just bulk-edit-based.

• Magda: A couple of things. This group is metadata heavy only because that was who responded to participate in the research.

• Magda: I did hear from other SIGs that the local editing option was useful and helpful, so we will keep both options in the end. The CSV option is not going away, but we may limit access.

• Ryan: Scott that is good context. The one person I spoke to that wasn't an academic metadata librarian was pretty gung-ho on the local editing option. And mostly because like you said, most of the work she was doing was with patron records. And it was much simpler to deal with this data when it is not cataloging metadata.
• Bob: What is the difference in functionality between the FOLIO import-export app and the proposed CSV bulk edit option. In our current Sierra system we do not use bulk edit to export, edit, and re-upload records. We use the Sierra load profile to do this.
• Magda: FOLIO Import-Export only handles inventory metadata, not users, acquisitions, and other data. That is one reason we are starting with user data as we assume metadata options to be available via the Import-Export app.
• Bob: But will Import-Export be able to handle user data in the future?
• Magda: There are perhaps plans for exporting users to CSV. But I do not think there are any plans short-term with the Import app to bring in user data.
• Erin: Data Import has been completely focused on Inventory and SRS and the associated pieces. No discussion on user data import.
• Magda: We will table this for now and come back to this next time when we see a presentation with mockups.

• Sarah: So one thing, to Bob's point, I think we have to consider is that not everything is going to be MARC-based. We are not going to have MARC Holdings in FOLIO. So, bulk editing is happening outside of MarcEdit which is only usable if dealing with MARC data.

• Sarah: I've been wondering, and I see that Scott in chat has also mentioned that in Orders, Dennis has been talking about editing and export, and changing records. This has also come up in ERM, around agreements. And I'm always wondering how much communication is going on across the apps to make the experience more similar. Or least help each other with being able to edit records in batches. While this has not come up so far for me in my current system, because I don't have agreements in my current system, that is definitely a thing now in FOLIO where I am actually already in production. It would be great if I could identify a set of records and then batch change them just like I would with other records. I'm just curious to hear a little bit about how cross-app communication is happening around all of this.

• Magda: I am not sure I know what you mean by communication, Sarah.
• Sarah: I go to the Metadata Management, Acquisitions, and the ERM SIG every week, and it seems like certain things are being talked about simultaneously, but there is little indication that they're being talked about together.
• Erin: In my experience, the most important thing you can do is tell the PO. An email or Slack can help those POs make the connection. In practice, the POs check to see if anyone else has done work on a function because you want to be able to reuse functionality and keep the UI consistent. So the best thing you can do is say something. Be persistent.
• Sarah: I agree that we have to start with export, edit, reimport. We should also not just think about MARC.
• Magda: Ryan, I think you are hearing opinions outside of what you heard in your research. Not everyone is against the local editing option. What you are bringing up Sarah is the crux of communication, and it is my understanding that this is the role of this group. Also please share feedback with me in these meetings, Slack, or directly regarding cross-app issues. I may not be aware of it.
• Magda: Let me take a quick look at chat to see if I missed any comments. Erin, you mentioned being burned by FOLIO before. What do you mean?
• Erin: That was mostly a joke Magda, but we have collectively seen a ton of FOLIO development that is kept really basic and simple at first, not really doing what we want, with the idea that functionality will be built out later, but then it never happens because other priorities come up. So I think that some of the concerns that might come out of discussions about the local editing option will be a feeling that it needs to be explicit so that we don't lose track of our long-term goals and that hopefully, we have an idea of when those long-term goals will happen. I just wouldn't want to see a local editing solution that gets developed and then all of a sudden we are at Zinnia and it's time to build the UI. I think that would really be a bummer.
• Magda: I agree it would be a bummer. And I definitely don't want us to wait until Zinnia. I don't think I will live that long, to be honest. Let's come back to this conversation next meeting. And we will look at the mockups and the two options, local editing and in-app.
• Magda: Please look at the Wiki page links to the mockups on the top right corner and provide comments. We will revisit this at the next meeting in two weeks.
  • Note: The Google Folder where mockups live.
• Magda: The next meeting is on November 30th.
• Magda: Respond to surveys in the Homework section below by Thursday, November 18th.

00:04:10    Magda Zacharska:    https://folio-org.atlassian.net/wiki/display/BULKEDIT/2021-11-16+Bulk+Edit+Working+Group+Meeting+Notes

00:20:19    Erin Nettifee:    Christie! Christie's awesome!

00:20:46    Leeda Adkins:    +1 Erin :-)

00:25:03    Erin Nettifee:    I love all of this but I've been burned by FOLIO before ;-)

00:25:28    Thomas Trutt:    Agreed.. The lest do this now with a full solution late.

00:25:50    Amanda Ros (Texas A&M):    is there a timeline for short and long term?

00:28:16    Jackie Magagnosc:    There are functions, for example deleting item records, that wouldn't make sense for a CSV option unless there is something about this I don't understand

00:28:32    Erin Nettifee:    Bulk Edit is not deletion, @jackie -

00:29:24    Erin Nettifee:    we discussed that some on the slack channel earlier this week or last week i think. do you have a bulk edit tool right now that supports bulk deletion?

00:29:38    Jackie Magagnosc:    Right, but it's the kind of use case I'm particularly interested in given a current local project, so I'm trying to understand

00:29:48    Jackie Magagnosc:    And I don't have access to a tool

00:29:55    Erin Nettifee:    Sorry Scott - that was directed towards Jackie :-)

00:30:29    Erin Nettifee:    Agreed Scott --- needs to be about the type of data involved

00:31:07    Christie Thomas:    Batch delete is somewhat included in the future of data import, but I have no idea when that will happen.

00:32:07    Scott Perry (he/him):    I think it partially depends on the complexity and quantity

00:36:22    Scott Perry (he/him):    There are already several apps in Orders and Finance that support CSV export or will soon.

00:38:40    Jennifer Eustis (she/her):    brb

00:40:29    Jennifer Eustis (she/her):    back

00:40:44    Erin Nettifee:    POs and Devs share code and design stuff pretty frequently - but ERM is a special case because they were sort of developed on their own with separate funding

00:41:29    Erin Nettifee:    We try to not re-invent stuff - what happens is that usually that means the first app to do a thing ends up defining the pattern for other apps going forward. Not always a great fit, but that's been the typical approach.

00:43:43    Scott Perry (he/him):    And the need is soon which is why I think it is percolating through so many different SIGs.

00:47:44    Sara Colglazier (MHC/5C):    Just also to be clear: in-app batch or bulk editing is my ultimate hope, but in the short term I cannot imagine being without any options

Development updates:

• • Development status
  • Development team scrum board

Bulk Edit Pilot - expectations - continue discussion 

Homework:

• Bulk Edit Limit Size
• Bulk Edit Permissions

Please respond to the surveys before Thursday, November 18th.

Future discussion topic (time permitting):

• Mitigation of reported UX issues
• Bulk Edit Permissions
• Bulk Edit of the linked records
• Max number of records to be edited via User Interface
• Scheduling edits