2022-11-1 Bulk Edit Working Group Meeting Notes

Attendees (please add your name):

Magda Zacharska (OLD ACCOUNT) Erin Nettifee Jennifer Eustis Amanda Ros Timothy Dannay Thomas Trutt Jackie Magagnosc Monica Arnold Christine Tobias Erin Weller Scott Perry Kimie Kester Donald Depoorter

Note Taker:

Robert Scheier

Meeting Recording:

Recording and chat: https://recordings.openlibraryfoundation.org/folio/bulk-edit-working-group

Discussion:

Topic	Details	Notes
Housekeeping Attendees - please add your name to the list of attendees Meeting host - please turn on Transcript option for the meeting - do we still need that? Updates on bulk edit rancher (aka bulk-edit-perf environment) We still need someone to take on upkeep / edits of the Bulk Edit app documentation ((OLD ACCOUNT) Erin Nettifee ) - can you ask at your institutions / SIGs if you are not able to do so?	[Magda Zacharska] 10:03:13 Big. Thank you, Bob, for being our note taker for more than a year Now we've been meeting for more than a year in October. October was the one-year anniversary we really appreciate your detail. I really appreciate your detailed notes. They are very helpful if I need to go back to our prior meetings. So thank you very much for your work, and if you get to the point that you feel you need someone else to take it over, please speak up. [Magda Zacharska] 10:03:51 I added a column on our wiki space notes pages table to allow a place to add any notes because I started to add a little bit more details for each topic that we discuss to provide some information to help you plan your week if you will be attending the meeting or not. The update on Bulk Edit Rancher--we talk about that during our last meeting. It used to be the Bulk Edit performance environment that we shared for user acceptance testing and performance testing that the team was doing. Then we run into the problems with updating the environment. I promised to share the link. I did not share the link, and this was not by omission. This was on purpose. We are competing for the same resources as much as the UAT is important, performance testing is also important. In Morning Glory, the UAT took press precedence, and we spent a significant amount of the last weeks before the release improving performance. I did not want. This happened in Nolana as well. So I suggest we either do UAT using the bugfest or snapshot environment. Other teams have done it. We will get to this conversation later. So you don't need to speak up right now. But this is what I will propose. In the future, and by that, I mean the Orchid release, we will have the option to spin up an environment only for Uat purposes. But this is not in place just yet. so that's why I suggest we use either of the existing environments. Any comments on that concerns? [Erin Nettifee] 10:06:30 No, I mean, I think snapshot would be challenging to do just because there aren't enough records. But I imagine we'll get into that later. on. [Magda Zacharska] 10:06:35 Okay, that makes sense as well. [Erin Nettifee] 10:06:42 We are still looking for somebody to take over the upkeep and edits of the Bulk Edit app documentation. And my request at this point, because no one has said they would do it, is, can you please ask around your institutions or SIGS if you don't have the capacity to do this? I think it's a good opportunity to learn the app and to get involved more deeply with FOLIO, and there is definitely onboarding that happens with that process. You're not just thrown into the deep end, but you would be taught how to maintain documentation and how that process works and stuff like that. So we really still need to find somebody who can do that. So I'm just gonna reiterate that request. [Magda Zacharska] 10:07:43 How much time do you think it takes? How much time would that require? [Erin Nettifee] 10:07:56 It is hard to forecast because we are adding functionality. So that's a lot of updates and creating new content instead of editing or fixing existing content. I would say, I think it's probably gonna be maybe at least 5 hours a release right now, and maybe more, and regular meetings. So it's not a trivial amount of time, but it's also not like asking somebody to be a product owner. It's just making a commitment and being willing to take on that ownership. [Magda Zacharska] 10:08:34 Great. Thank you, Erin, for bringing it up. We will be announcing this until someone finds time to take this on.
Development updates Development status Development team scrum board Recordings of implemented Nolana functionality can be found here	[Magda Zacharska] 10:08:47 Regarding the development updates. The links to the Snapshot environments, and these scrum boards are here, as always. [Magda Zacharska] 10:09:29 You can check them any time you want. but I would like to bring to your attention that we have a folder with recordings for functionality added for the release. [Magda Zacharska] 10:09:40 So here is the folder. There is a folder called Bulk Edit app recordings, and for Nolana, I created a folder with the name Snapshot. [Magda Zacharska] 10:10:03 The folder has recordings for each functionality that we have delivered in Nolana. And this covers user records in-app approach, item records in-app approach, and holdings records in-app approach. [Magda Zacharska] 10:10:22 I also attached the CSV files that I am using in my demo to trigger Bulk Edits in case you want to play on your own, you can use those files. [Magda Zacharska] 10:10:39 The recordings don't have a sound it's only the recording you can stop and rewind if something is unclear. Any questions? [Magda Zacharska] 10:10:53 I do believe this will be helpful. when you start looking into functionality that was in the scope of Nolana. [Erin Nettifee 10:14:53 I think it's especially helpful to have the files and the videos in the same location.
Review of the functionality delivered in Nolana Modifications to the UI: Selected columns persists through out the bulk update screens Adjustments to the left pane so that the record types are listed on top before identifier selection User records - in app approach: Patron group can be updated Expiration date can be changed Part of the email address can be updated (host name changes) Holdings records: Permanent location can be updated Temporary location can be removed or updated Item records: Permanent loan type can be updated Temporary loan type can be removed and updated	Magda Zacharska] 10:11:10 The next topic is the review of functionality. [Magda Zacharska] 10:11:20 I have a Powerpoint presentation that I would like to walk through and then do a quick demo of the functionality we have done. [Magda Zacharska] 10:11:37 Please feel free to interrupt me while I go through the slides. [Magda Zacharska] 10:11:52 So in the Nolana there are 3 main functional features, one to support user data updates using in app approach. For item records we implemented locations and statuses. We also added loan types. [Magda Zacharska] 10:12:18 Records we added to already a implemented holdings, locations, and statuses. [Magda Zacharska] 10:12:25 And for holdings, we started with the locations as well. [Magda Zacharska] 10:12:34 Those 3 functional areas are completed. We are still testing. Possibly some bugs will be discovered. But the main development has been completed. [Magda Zacharska] 10:12:47 In addition to that, we have a series of NFS testing which includes automated tests for the backend. [Magda Zacharska] 10:12:57 This is Karate for front and releasing the modules and updating that technology, especially for the spring application we needed. [Magda Zacharska] 10:13:18 So what have we done while working on those features and some changes to the Ui? [Magda Zacharska] 10:13:31 Some of them were mentioned during user acceptance testing at the end of Morning Glory. [Magda Zacharska] 10:13:39 One of them was to change the order of components in the left pane, where the user selects the record type first, and then selects the identifiers. [Magda Zacharska] 10:14:01 The next suggestion from user acceptance testing was preserving the selection of the columns. [Magda Zacharska] 10:14:10 Now the user can select the columns that are displayed on the landing page when the matching records are displayed, and the selection of those columns is preserved on the are you sure form where we preview the changes that will be made and also on the confirmation screen once the changes are committed. [Magda Zacharska] 10:14:39 And the last Ui improvement is on the export manager. Magda Zacharska] 10:14:50 We can now filter out Bulk Edit jobs and also by statuses: successful, in progress, or failed. [Erin Nettifee] 10:15:05 Bulk Edit does not use scheduled right? [Magda Zacharska] 10:15:28 Correct. In this case it means that something went wrong with the uploading the identifiers and the job never progress further. [Magda Zacharska] 10:15:38 So this is something that is in a not expected behavior. When you click on the export manager on the row, it will tell you the details of what went wrong. But you are correct. The scheduling of the Bulk Edite Jobs is not implemented yet. It is planned for the later releases. [Magda Zacharska] 10:16:01 And the other questions, the comments. [Magda Zacharska] 10:16:08 The nice thing I will move to that user data in the in-app approach. We will be supporting, in addition to the CSV approach that was delivered in the scope of Morning Gloria, on the request of community, we also added support for the in-app approach, starting with the most common changes: change in email hosting, expiration date update, and patron group. [Magda Zacharska] 10:16:55 I will demo this later, and this is just a short preview of of the of the changes we made. [Magda Zacharska] 10:17:12 On the item records Bulk Edit, we can also do the changes to temporary loan type and permanent loan type. [Magda Zacharska] 10:17:17 In addition to already implemented support for a temporary item location, permanent item location, and item status. [Magda Zacharska] 10:17:35 For the holdings records. we added support of temporary and permanent holdings locations. [Erin Nettifee] 10:17:50 And this is just the inventory records this is that's that is correct? Not SRS? [Magda Zacharska] 10:18:02 That is correct. This is only for records that have a source folio. [Magda Zacharska] 10:18:04 Okay, So let's do a quick demo Note: see recording for demo. Notes below are just regarding questions and comments that arose. [Erin Nettifee] 10:19:34 Is the the fact that the actions are great out here mean that is the only choice? Or could you choose a different action? So if I wanted to, for example, erase remove the expiration date from a set of user accounts, I would have to use the CSV approach? [Magda Zacharska] 10:20:19 We don't support setting it to empty Also the same with Patron Group. Patron group which is a required field, and it needs to be populated. [Erin Nettifee] 10:20:30 Expiration data is not required on the user record. [Magda Zacharska] 10:20:34 So for now we can only support replace with something. [Erin Nettifee] 10:20:45 We will need to add this functionality separately. [Magda Zacharska] 10:21:47 So you can still download the CSV file and see all the records. And you see the records in its the entire record, not only the columns you have selected. [Magda Zacharska] 10:22:13 And we are committing the changes, and here is the confirmation of the of the changes we have a committed It's also the the option of downloading the the change records. [Magda Zacharska] 10:22:33 Any questions on on this functionality 10:28:10 Note: There was an error when Magda tried to demo Bulk Edit for holdings records. She said that this was due to a code change on the back end that the UI did not handle correctly. This will be address in bugfest. the issue.
Nolana UAT Scenarios Decide which environment to use	[Magda Zacharska] 10:29:02 So in I would like now to move to the user acceptance testing and the scenarios. [Magda Zacharska] 10:29:19 I change a little bit. approach this time. Instead of sending the survey and awaiting for your responses, I would like to use the Excel spreadsheet approach. Magda Zacharska] 10:29:38 This is with a spreadsheet that contains the steps for each record type and each step that needs to happen. [Magda Zacharska] 10:30:12 Start, and all the steps that we that we have that I showed you today. [Magda Zacharska] 10:30:19 I added column for each institution that is on the Bulk Edit Working Group list. If your institution has more than one representative, if you want to add your comments, please clone the column and and add your comments in this empty fields. Type Pass, Not Pass, or any other comments that are applicable. [Magda Zacharska] 10:30:59 Since we agreed that we are going to use the bugfest environment and bugfest starts next week, I will re-post the link to this document in our slack channel with the logging information to the bugfest environment and we will have 2 weeks to complete the the user acceptance testing. [Erin Nettifee] 10:31:50 Will we all be doing the same editing on the same records? [Magda Zacharska] 10:32:16 It's up to you because if we I can provide you with the list of records identifiers, then you will be using the same files, and you will be most likely stepping on each other toes, if you do those at the same time. [Magda Zacharska] 10:32:25 If you can use your own for identifiers or I can provide you with the multiple list of identifiers, and then you will be marking which one you use. If you use your own file please drop those on the Google Drive in the folder. [Jennifer Eustis (she/her)] 10:34:26 Magda, I just have a quick question. Is the UAT testing in addition to the tests we are doing TestRails? [Magda Zacharska] 10:34:34 That's correct. [Magda Zacharska] 10:34:43 We do have a lot of added tests in in TestRails, but this is a slightly different point of view. You can use your
UXPROD-3842 Bulk edit - architectural improvements The changes are needed because the current implementation is not scalable enough to support larger data. Results of current performance tests conducted by the team Proposed redesign	[Magda Zacharska] 10:36:18 We have less than 30 min left and I would like to spend this time talking about that and getting your feedback regarding architectural improvements. This is a a big endeavor to make architectural changes, we need to start them early to make sure we have enough time to test it properly. [Magda Zacharska] 10:36:50 And this will definitely affect the Bulk Edit road map I don't think we will be able to talk about this today. [Magda Zacharska] 10:36:59 I'm also in the process of updating a Bulk Edit road map, so we will return to to this next time we meet. [Magda Zacharska] 10:37:10 So why did we decided to make architectural improvements? In Morning Glory when we did performance testing, the goal was to support editing at least 100,000 records? [Magda Zacharska] 10:37:31 We realize that the architecture does not support that, and the performance is even worse if there are other application using export manager: exporting EDIFACT records, Bursar exprot, or circulation log exports. [Magda Zacharska] 10:37:59 All of those when they are happening they have an impact on bulk edit. [Magda Zacharska] 10:38:03 So, after further analysis of the code our Solution Architect proposed we redesign and separate Bulk Edit from Export Manager so that it is not affected by all the other things that are happening while other exports export the data. I added a link to the performance results from Morning Glory, and you can take a look at them when you have a chance. [Magda Zacharska] 10:38:50 I added the link to the proposed proposed design, and I would like to spend a couple of minutes talking about it. I will not be going into the technical details, but there is one part I would like to spend a couple of minutes on. It's a bulk operation states. Because this describes how the data is being processed. [Magda Zacharska] 10:39:17 How will be processed? They will be several states. Not all of them will be implemented in scope of the Orchid release, but those that will include: New - This is the the moment when the user uploads the identifiers or runs the query. Retrieving records - This is when the user retrieves the records.So we submitted the the way we want to identify the records, either identifiers or the query, and the records are being retrieved from the storage storage modules. This is the second state. Saving records - This is the third state. This is the snapshot of the records that you get when you run the query. Those files are written to a CSV file, but this is basically a file that is created in export manager. But we will move this file to the Bulk Edit shared storage. This will be not model internal storage. This will be a storage in S3 Armenia for those who are not using Amazon Web Services. This is the file where it will be stored. This is not the CSV file you are downloading to your local machine. This is something that is still happening in the system. Data Modification - This is when the user specifies what changes will happen either in the in-app approach or in the CSV approach for user records. Review Changes - This is where the records are being displayed on the are you sure form. Apply Changes - This is when the changes are being committed to the records stored in the storage modules. So this is saving the data. Suspended - This is one of the statuses that we will not support in Orchid. This will be implemented later, when the user will be able to suspend the bulk edit. This will obviously require further discussion. Completed - This is self self-explanatory. It is when the job that was completed with no errors. Completed with errors - This is when the job is completed with errors and you will still see the errors in the error accordion. Cancelled - We are not going to support canceling jobs in Orchid. This is something to discuss later. Scheduled - We are not going to support canceling jobs in Orchid. This is something to discuss later. Failed - This is when something went wrong and the operation did not complete. [Erin Nettifee] 10:43:08 So this list of statuses is this something that is part of the new design? And is this something that would end up being exposed in the Bulk Edit app or both? [Magda Zacharska] 10:43:25 Yes. This is a part of the new design, and it will be something that will be exposed in Bulk Edit. We are going to do this by an adding additional tab to Bulk Edit. In addition to the identifier and query tabs there will be the logs tab which will again be driven by permissions. [Magda Zacharska] 10:44:10 But if you have permission, you can see the logs and the standard filtering by status, by record type, and by type of Bulk Edit operation--if it was Bulk Edit edit or delete. And also when the job started when the job ended. [Erin Nettifee] Okay, So this is kind of like deciding to build our own version of the export manager just for Bulk Edit so that it can be customized to Bulk Edit's particular needs. [Magda Zacharska] 10:44:53 Exactly. It has a more information. You get more data here. [Magda Zacharska] 10:45:12 So what you have Bulk Edit Operation Type, Record Type, Status, those are the status that we mentioned, who ran it, when it started running, when it ended running , the number of records requested, how many records were were processed., was it the editing in app or Manual/CSV, and Actions. If Actions is empty it means no files were created and no files are available. When you click on the three dots you will have the option to download the files, and you can download each of the files that are being created in the process of of Bulk Editing. [Erin Nettifee] 10:47:02 Thomas is asking about in the chat about whether the information that we're seeing here would be useful in the export manager. But I am guessing that when this is implemented, there will no longer be Bulk Edit data in Export Manager. It will just be all be encompassed in its own interface. [Magda Zacharska] 10:47:31 So when we go to the Bulk Edit Operational Statuses there's one part that still happens in the Export Manager. When the records are being retrieved from the back end this is happening in the Export Manager, but the file that is being created is being moved to Bulk Edit shared storage, so the file will not be accessible. You will be able to see that the records are being retrieved, but we will not have those files at all in Export Manager. [Erin Nettifee] 10:48:11 Okay, that feels a little disjointed to me but I don't know if others have thoughts or on it. [Erin Nettifee] 10:48:18 I think I would want to be able to see all the stuff just in one app. [Magda Zacharska] 10:48:25 So you would see all the files in bulk edit. You don't need to go to Export Manager. We can hide whatever is in Export Manager. So it's. not confusing for anyone. [Erin Nettifee] 10:48:35 I mean this is all kind of abstract, so you I'm not advocating for a particular decision. [Thomas Trutt] 10:48:56 My big worry about this is are we just moving the problem to another app? [Magda Zacharska] 10:49:37 First of all, we are not competing for resources in with another applications.That was one of the issues we are facing. The most resource greedy part of Bulk Edit is actually saving the changes to the database and handling those changes accurately. Until Nolana, this has been done by Export Manager, which is not the place where those things should happen. This should be a Bulk Edit responsibility. Does it answer your question? [Thomas Trutt] 10:50:43 A little bit. I still have a concern. Yes, it's getting more access to the Bulk Edit back end process. Because now it's his own app and it's able to do all this processing on its own. My worries is that it's still going to be competing with Export Manager and other processes, because it's still going to be hitting the same API endpoints on the other internal apps. So it might even compound it, because you now have 2 large apps hitting the same APIs at the same time. [Thomas Trutt] 10:51:12 I guess I could see where this might make a little bit more sense, because the Data Export App was doing the updates moving that component out of it. And that does make sense. Because why would you have updating data in the actually export App and I see your point that we are hitting the the same like in case of hi, Thomason, holding. [Magda Zacharska] 10:51:42 So we are. We are hitting the same environment with different apis [Magda Zacharska] 10:51:51 But this will happen if somebody also exports, or does the oapm H. [Magda Zacharska] 10:51:58 Or make any other inventory changes. this is not what we can prevent. [Magda Zacharska] 10:52:06 What we know, however, that if something goes wrong we can identify the module, and we can handle this within a module. [Magda Zacharska] 10:52:16 One module. and this is a good point. We probably will be coming to this once. [Magda Zacharska] 10:52:22 We have a a performance test for the in place for the new for the new design. [Magda Zacharska] 10:52:31 I was told this will resolve our problem with the limit of 10,000 records. [Magda Zacharska] 10:52:42 At this time we should be able to go up in the number of records we can update through the bulk operation, but this definitely will not be a superb blood for inventory or circulation modules performance. [Magda Zacharska] 10:53:06 I would like we have some of the minutes left. There is one more mockup that I would like to show. [Magda Zacharska] 10:53:17 Is those [Magda Zacharska] 10:53:22 Hi Tim, that have expired because we will be putting the record. [Magda Zacharska] 10:53:28 The the files, a large number of files in the external in the external storage. [Magda Zacharska] 10:53:37 We will keep them for a month. So for that bug edits that happen in the previous month, you will be able to access them. [Magda Zacharska] 10:53:47 But then we will remove them, and the files that were removed will be marked with this information, as you see on the screen, plus unavailable for download, because 30 days have elapsed since the job was run [Magda Zacharska] 10:54:04 in export manager currently you are getting the Xml error that tells you that the exploration talking elapsed, which is not very user friendly. [Magda Zacharska] 10:54:15 That's why we decided to go this route to to notify the user that defaults and no longer, and [Erin Nettifee] 10:54:31 So the 30 days then is hard coded. we can make it. [Magda Zacharska] 10:54:41 We can make it configurable through the api because if it turns out it's a preferred approach. [Magda Zacharska] 10:54:52 We will start with hardcoded value of of 30 records 30 days. [Magda Zacharska] 10:54:58 This was the the value that was at some point proposed for the files that are being generated by data export as well. [Erin Nettifee] 10:55:06 Does anyone have any comments? Sure, I mean that that and 30 days makes as much sense as anything to me? [Erin Nettifee] 10:55:12 It might be worth a question to Sisops or to, you know, ebsco hosting or index data, just to get a sense. [Erin Nettifee] 10:55:21 This, too, what they think of that number and configuration and stuff like that. [Erin Nettifee] 10:55:27 But here I mean the the stuff that we would have to retain would be things like financial records, and we're just not. [Erin Nettifee] 10:55:34 That's not what's happening here. so I I think 30 days, I think would would be okay. [Erin Nettifee] 10:55:40 But it I it's probably worth just asking around [Magda Zacharska] 10:55:50 So I run this obviously by obscure hosting team. [Magda Zacharska] 10:55:55 And I do also believe that that may depends on size of the institution. [Erin Nettifee] 10:56:04 Sure, the large institution that have larger about edits, and they have a larger file size same way. [Magda Zacharska] 10:56:13 Also want to cop on calls of storing those files because they will be eating much more space than for smaller institution when they have a smaller and smaller file size, they will not be eating that much of this storage space and [Thomas Trutt] 10:56:35 30 days may be not even required for them. They could go longer without running out of space or incurring costs, for it sounds like this is almost a from what you even just described. [Thomas Trutt] 10:56:56 Now as this might move around based on the institution, and that would make more sense to have. This is a tenant level setting, or have it as an api endpoint that could be hit. [Magda Zacharska] 10:57:05 You could say delete all files after the State that could be set as a cron job or something. [Magda Zacharska] 10:57:11 I, Rather let the user the tenant to specify their grace period and yeah, that would be. [Thomas Trutt] 10:57:23 That would be my preference, as well but if that's not possible. [Magda Zacharska] 10:57:27 The second one would be having something that a host hit an Api and say, Remove these files up after this date sounds good. [Magda Zacharska] 10:57:36 I will bring it up to the to the development team [Magda Zacharska] 10:57:45 So this is it what I had for today? [Magda Zacharska] 10:57:49 Do you have any comments, questions. [Erin Nettifee] 10:57:59 I know how many i've [Magda Zacharska] 10:58:12 Thank you all, and i'll see you in in 2 weeks and We will start with the roadmap updates. [Erin Nettifee] 10:58:20 Thank you. Hey, Erin, Can you save that file? [Erin Nettifee] 10:58:25 Yes, I will save it for you. Thank you. have a good one.
Changes to the Bulk edit roadmap TBD
Chat Discussion	00:03:36 Jackie Magagnosc: Boy log in in the middle and wonder what is going on 00:04:01 Jackie Magagnosc: LOL 00:05:36 Robert rscheier@nelib.org: I am having issues. Can you turn on transcription 00:06:29 Robert rscheier@nelib.org: No worries. If you still want me. LOL 00:06:54 Robert rscheier@nelib.org: will do 00:07:08 Jennifer Eustis (she/her): Thanks Bob for being notetaker :) 00:13:55 Jennifer Eustis (she/her): This is great Magda. Thank you for these videos and lists of ids 00:14:08 Christine L Tobias: Thank you, Magda! 00:48:36 Thomas Trutt: But wouldn't the same statuses and information be useful in the export tool? 00:49:25 Erin Nettifee: i think the necessary backend changes to support bulk edit probably make building another interface an easier path... 00:56:13 Ros, Amanda L: I think it has potential. In the long run I will like it better as it's own app 00:56:30 Ros, Amanda L: (I hope) 00:58:38 Thomas Trutt: Agreed, id run the 30-days past a few other groups.