2022-5-3 Bulk Edit Working Group Meeting Notes

• Magda:  Let's start with the standard development status. This is the FOLIO snapshot. We did some progress on the in-app approach. I am logged as a user with both permissions for the CSV approach and for the in-app approach. The user behavior has not changed. So let's concentrate on the in-app approach and the changes we have made so far.

• Magda: The user selects in-app (by selecting the "Inventory - Items" radio button).

•  Magda: The dropdown menu is populated with the items-related values. So let's select the item barcodes.

• Magda: Choose the file to upload. And I will take item barcodes from my list of files.

• Magda: It populated the screen with the items that meet the search criteria.

• Magda: We can add or remove columns. And we can start to bulk edit.

• Magda: As I mentioned, because I have two permissions, both CSV and in-app, I see these two options.

• Magda: So I am starting bulk edit in-app.
• Magda: Here are on the screen where we choose the options., one option, is permanent location, we have two action options, replace with and clear field.

• Magda: If we select replace with, we are picking the location using the location lookup and then save and close.

• Magda: We can also add another option, which will be a temporary location, and we can clear the field. So whatever is in the temporary location will be cleared.

• Magda: We, do you have any questions so far comments?
• Erin: It would be nice if the location could also have the inventory type of ahead modal in addition to the drop-down modal. People may find that that is easier, especially if they have just a few locations, so they don't need to like go hunting and, or maybe they know exactly what the location is called and they can just type it.
• Sara: The only problem is, fo us at the five colleges that is not, you know, if we have a dropdown, then that dropped down becomes ...
• Erin: I'm not saying remove it. I'm saying have both options.
• Sara: Okay, good. That would be very helpful.
• Magda: I will write a story for this. I'm not sure we can deliver it for Morning Glory. I will need to talk to developers about how much work it would take.
• Magda: Um, I have another question related to this, but I want to go through the whole process and then we will return to this screen.
• Magda: Okay. So we are confirming the changes and we get the response that 2 records will be changed if you saved them. Before we do the save, we do the download preview.
  • Erin: So the previous says, there are no items. Is that a bug?
  • Magda: This is a bug. You will see a list here. Actually this story, the form "are you sure?" is still in progress, so it is not committed. So it's a work in progress.

• Magda: So when we look at the preview, this is the preview of the changes. This will show you how the records will look like after we make the changes, the changes are not committed yet.

• So as you see here, effective location and temporary location, this is before the changes are committed in the database. If you click cancel at this moment, you don't want to proceed, no changes will happen. But if you continue, you see that all the records will have main library and they will have no temporary location. And it will obviously affect the effective location. If I click keep editing, it will take me back to the screen and no changes were commuted.

• It shows that 2 records have been successfully changed. The error and reason for are displayed. I will need to investigate why we got this error, but at least we know we were not able to update this record. And also on the action menu, you can download the errors.

• Sara: I was just wondering would something not being able to be changed constitute an error? And how will that be counted? Because you have this interesting count thing going on. Let's say I have three records, in the meantime, somebody went in and manually already changed one to not temporary. And, so I'm asking all three to become temporary, to have everything be changed to their permanent location. But it can't actually affect the change on the one that was changed in the interim.
• Magda: It will overwrite the change. So let's say you have your list of identifiers, your scan the records and you are doing the bulk edit and you got distracted and do not proceed. In the meantime, someone goes and changes the records. Then you come back and you continue with your bulk edit. You're overwriting the change. Please note that this is the in-app approach. So it is not that you can upload the CSV file and you can start over. Those are the changes you are specifying in the UI.
• Sara: So I guess it still seemed though that I had this file that had three things.
• Magda: This is, this is just for your preview. In the in-app approach, you don't upload the file to trigger the update. The preview is only so you can see how the records will look before you commit the changes.
• Okay. Is it clear, Sara. If it's not clear, please ask more questions.
• Erin: What you will get from an error is you will get like what an API would return, which would be like what you would get if a pop-up showed up or something like that, as opposed to something where you might have over overwritten something or something like that, you're going to get an API message.
• Lloyd: Sarah's question made me wonder how is this is interacting with optimistic locking?
• Magda: Inventory obviously uses the optimistic locking. But what you have to keep in mind is you will be updating the latest version of the record. The bulk editor at this point, in the in-app application, does not have a snapshot of the data that was taken when you started the bulk edit process. So for this I would like you to forget about the CSV approach. There is no CSV approach at this point for item records. What is happening? Um, let's go, let's start again. And, um,

so the, the, you know, the polar records and show me the preview and stuff like that. You know, it may have gotten record fields, but it's, it's just pulled them down and you're not in any sort of edit function. It's just giving you the fields. And then when you do the actual bulk edit, it's gonna Repole the record apply the changes and put it there again.

Sorry. I think, uh, we, we hit another bug. Uh, let me try again. Uh let's um, I said this in China, and I guess this would be, I understand what the in the app want. It's a lot quicker turnaround times, so there's not that gap. So you don't have to worry about it, this CSV version, it makes sense. But in both versions, couldn't you store the version number of the record that was, is supposed to be edited and check that before you do the bulk update

is there should be a, there's probably a lock of magnetism is a version number. So the version of number incremented by one, it would mean that there was a change between. Yes. So here we go. Uh, let, let me walk again through the process and maybe this will clarify a user bucket. Choose the file.

I'm not sure what's going on right now. Uh, I mean, it's snapshot, so it may, maybe the inventory is having problems. The file you're choosing there is the results from your query that you saved. This was the file that I use a couple of minutes ago. Um, so let me go back to the snapshots to chose users in that.

Oh, did I choose the user? Oh yeah, of course. Of course. Uh, yeah. Thank you. Sure. Um, yeah, that would make sense why it is not fighting the buckets. So item barcode, Amanda item, I'm selecting the file. So the behavior was as expected this item barcodes.

Okay. I have the list. This is, those are the changes. Uh, those are two, three records that we intend to change. Right. Um,

and this is something is definitely not right, because I should have the in, um,

Yeah, sorry. We'll investigate that.

This is, you have a file with the, uh, with the barcodes scanned or, uh, wherever you get the file of the, of the barcodes. Now you are starting the bulk edit. You know, this is the, those are the items that we'll need to move from one location to another, or they have been moved. So you're selecting that you're attending the permanent location.

You're changing it to, uh, let's say, um, uh, here we are going to next dude. You could this time

save and close and, uh, we will also remove all the temporary location and clear the field to make sure. We are confirming the changes, meaning we are going to, um, to check the, uh, to check the changes, forget about the records. Count in this. This is, as I said, work in progress, but let's say you can see the F you will see the preview top 10 right here.

You can download the files, but this is not a requirement. This is for you just to see, this is the download of the file before the changes are committed. So at this point, no tenders are in the database. You select continue to make the change. Let's say.

No, no love the records was updated. So, um, I will need to investigate what was happening, but at least, you know, that the records were not, uh, and not, uh, were not, um, updated. So if we go back to our original, uh, screen, and this will be again, the option of starting your bulk edit, that we will be discussing shortly.

But if I go, uh, to,

uh, the item, the, uh, no, sorry, the barcodes.

So those are the same file that we start, uh, started the process with. And if I add a permanent location and, and temporary location,

I can see they were not change, um, due to the errors. So if there is an error, there is no that the, the update will not happen. The question you had about storing diversion and updating the version at this point in up in up approach, you don't have a version of the, of the records. You don't know what you are changing.

This is what, what, uh, what we do. Uh, the, what we see this is the given moment. When you run the update. Why would you like to know what is diversion of the after of the record you are updating? What would this, what would it be the use case for you to now at this moment, I was thinking of a safety precaution.

I understand that nothing has been state, no changes have been staged yet. And till you hit that final screen where you have the option of downloading the Excel file, at that point, you have staged changes. They're not committed yet. They're staged. And the window between that, and when they're actually committed is very small.

Um, but there is still a chance of, and I think this is kind of brought up a little bit before someone else could go in and change a file between those steps, even though it was a very small time period. And my suggestion was is that when you do start those that when you do that staging process, if you store the version number of the current.

And then when you find like commit the record, you could match them up to ensure that there was no changes in between those two steps. It's not more for me, it's more as a safety net, so, right, exactly. And, and I realized that with the in-app changes, that window is very small. You're talking about maybe a minute or two at most.

So the likelihood of big changes going through. Um, but I think it would be a good approach for both this, as well as the CVS for file type, the catch changes into the inventory between when you make that page file. And that commit file in all fairness, too, we have not, we have not uploaded the bulk at it and tried this with 10,000 records, 15,000 records.

So I do think. Until we know more about performance. No, it's about to say the same thing, Aaron, that, um, the window might not be small. You may decide what you're going to change. And like you said, it's 10,000 records. And depending upon how that, how fast that runs and what it might affect running and circulation or cataloging or somewhere you might choose to wait and run that overnight or later in the day.

Yeah. So I think when we are talking about items, um, and thousands of items, um, probably this will not be going into ours, but we'll see, we don't have the permit. The performance test for items conducted just yet only for users. But, uh, I do, uh, think that. 10 of thousands or tens of thousand, shouldn't be that long, but if you go into hundreds of thousands, then we are adding the time.

And I make, I made a note, uh, for, uh, the version control. That definitely will be important once we start working on CSV approach for the, uh, for the items records. But, um, the CSV approach for item records will not be a part of morning glory, most likely. So, uh, um, I'm sure, uh, I will come back with more up questions and I do see that this is, uh, Well, yeah.

And Jen is pointing out in the chat that data import doesn't have a safety mechanism right now. So it's not like we'll get it is somehow not doing something that data import does the different sense that we are new applications. Right. So we, uh, we are more, um, we can be more agile and we can adjust accordingly.

But, uh, go ahead, sir. Sorry. So again, so I get the in-app part. So in app, in my system, at least right now in all of I can, I can do a query. It creates a file. That pile is an app. It sits on my computer. I haven't downloaded anything, blah, blah, blah. Right. I think Lita was saying, for example, if it's a especially large one, I might not run it right at that moment, but on my schedule, I can, I can prep it.

Now I'm going to run it at 7:00 PM when I know things are slower or something. Right. And so, so there's a window right there of time where things can change. And so I guess my question before was more about going back to how it was nicely saying, oh, I've updated two records, there's an error. And so the third one wasn't updated was I was just trying to get a handle on what constitutes an error.

What is tip was not counted, so that, that output can then be evaluated by me. Right. So I was asking if, if it finds that in the interim, the change that I was trying to do by book on an item. Had already been handled. Would it count that as an heir or would it just count? How does that count? Because it's not effecting a change on that then, because it's not finding anything too deep.

Sermonize you see what I mean? I think that's and because this, again, it's not so much about version control, but there is, um, so Sarah to make sure I understand your question. Um, uh, you run the bulk edit and then you encounter the error was, uh, recorded in the errors part, which means the record was not.

Updated then you are just at the record so that the error will not occur anymore. And you are running again. The bulk edit. No, no, this is so I'm sorry. Can you, let's say we have, uh, there's I do a query for everything that has permanent location poetry room, and I do this at 11:00 AM. Um, and so I get three things, let's say, for example, right?

So I want to run. I want to change it so that they're no longer a temporary location permit a poetry room, they should just go back to what they thing is. And, but sometime lapses for whatever reason. And so I don't run it. And in the meantime, somebody has come in and on one item has changed it manually taking the, the temporary off, which is fine.

Right. And so now two can be changed by the bulk edit. And the third one can't be changed because there's nothing to change. You can, they'll just, it will just override it and be able to just, it will just wipe out what's there, which is nothing. It will wipe out nothing with nothing. Okay. Currently an error.

Yeah, but I don't get errors then. No, you, you, you, you will just overwrite, but this behavior will be exactly the same. If someone moves the, um, from, let's say poetry room to a current events room, and then you, uh, let's say in the morning that the item was in poetry room, and now it's in the, in the, uh, current events room and you provide the barcode of the item after you run the bulk edit, it will be set to.

Yeah, no, no, this, I understand that. So I was just trying to ask whether nothing with nothing is an error or

it doesn't have any way of knowing what the state was before. It would only be ever be an error if, if a field was required and you tried to erase it, but that, that would return an API error and you would see that message. So I think if, if we had what Thomas was suggesting, then it would know that there had been a change and there would be an hour, but since it has not recording the state that it was in, it's just at the moment you click run, he was making the changes that you have in that you've asked for, regardless of what state was before or not.

It's not keeping track, right. Magna.

So did I answer all questions or, uh, I have couple of questions regarding the, um, the, uh, locations, because as you saw on the,

on the under screen bulk edit, I can set permanent location to clear field and I can also do a temporary location. Can you a field and nothing prevents me from doing it. Is this desired behavior, or would you like to have some limitation on. This is only changing the locations on the item record. So the holding records still would have, could still have a permanent and temporary location if, if it was populated before.

But if you, if, for example, you remove, if, if all the locations are being stored on the item level item, record level, and you don't have anything in the holdings, uh, location, uh, holding will always have a permanent location that is a required field. Okay, then. Perfect. Thank you. Then in that case, I don't have an issue with clearing the permanent temporary location off the item field.

Since the holding record will always have a permanent location, always a location there's always will be an effective location. I think there are, there are definite use cases for being able to clear both. Awesome. Okay, great. Thank you. Because this was my concern that we, uh, we may be a little bit too aggressive in this.

Uh, the other question that I have, um, I'm sorry, I'm just, uh, I keep missing, uh, going away from our agenda.

Let's just say, Rob has his hand up to, oh, go ahead. Sorry.

Sorry to interrupt your flow. I just a quick question. Not that I'm not a big question, but about the UI behavior, if you, um, choose, uh, to change the permanent location, and then you went down to the next line. When you went to choose temporary location, you could still see permanent location. Is that something that could be updated when you choose a low, uh, something from the dropdown.

The second line would not present that as an option in the temporary, in the left side. Uh, hold on. If we are here in replay independent location, we are selecting it to, um, we don't have luck with duke, so let's, uh, go to something else.

And from what you're saying, we select it here. Uh, it's even close and let's get back at, let's remove this. So we are here and then when we add the next one, You don't want this to be permanent, you would like, because the only option would be the temporary location here. Right? I think the assumption Bob is making, which I think is probably true, but people should jump in if it's not.

Is that when you're doing this kind of work, you would only make one change on each field. Well, unless it has multiple, you know, in certain cases, I guess. Yeah. I don't think I can think of a reason why you would have it twice. Right? I'm specifically talking about location. There may be because location is a pretty simplistic field.

Um, you know, we're not talking about something that's storing an array of values or things like that. I mean, it's not a huge thing, but it just jumped out to me. And I saw that again, you know, we've already chosen it. You are absolutely right. Uh, I'll add to the story for this as well.

Um, I'm taking notes. Um, would it make more sense or make it easier instead of having a dropdown box where you choose permanent item, location, and temporary type item location, they're just static. And then the action could be select action. And if dolphin is selected, then nothing has done. Can you repeat?

Because, uh, I think I missed it. Can you, instead of having the options as dropdowns static options, so have permanent locate item, location, temporary location, but then have the auction dropdowns have a default to select action. And if, if it stays on select action, it just doesn't do anything. So you don't have to worry about disabling an option, whichever one they choose.

It's just static cases across.

Okay. I don't know other people should chime in, but I, the thing that I like about this is that the UI looks like other UI options in folio. So I don't know that we have another field. I mean like, think about like an electronic URL on an inventory record. Like, it looks like this, like you choose the value, you choose the thing, right?

You just go drop down by dropdown. I do believe this is the reason that we went this route to make it consistent. The list will obviously grow as we are. So by the end of morning, glory for morning blood release, we will have permanent location, temporary location status, those three fields, and the list will grow, grow, grow.

As we, as the application matures.

Okay. Oh, sorry. I'll put my hand down. So, uh, let me, I have this control here and, uh, going down the agenda. So we covered the expected behavior from, uh, from, um, uh, wiping out permanent and temporary location. And you agree that this is, uh, supported and this is supported the functionality. The next question is we talk about that at some point about new bulk, edit a button, a button to add on the CSV, uh, approach.

This was when we talk about the users records. And I have a question about visibility of this button and also how does shoot work in for the in, in approach? So when we are on this screen on the bulk edit, we start, uh, here we, we have the preview populated. I would like to have this, uh, starting a new, the new bug, edit button, still visible here, because this would simplify providing the list of the new, um, uh, in your file to enable this will reset the screen very easily.

Do you have problem with this approach?

I see heads moving. I don't care anything. We'll say it again. Like I guess I do walk through it again. Okay. So, um, in this, uh, mock-up that we discussed, this was discussed for the CSV approach for you completed the, um, the bulk edit and, uh, of one group of records. And now you want to do another, um, bulk edit.

So to reset the form easily, to go back and start clean, you click the new bulk edit button and you go, you know, you, you, it reset. The record identifier, it resets the notification screens, it's resets errors and produce, et cetera. You just wipe, uh, the app. Um, clean. My question that I have is if this would be also expected behavior for the in-app approach.

So you would, you would see this on the beginning of the, uh, like right now you selected the records you decided, okay, this is not the data set. I would like to work. You click the button in your bucket and you have an option to upload another. I think someone said, no, I heard something. No. I said, I think that is good.

Um, I think what, making sure that you're starting from a clean slate is really exactly that you don't introduce any pre settings. So that's often in many of the other apps, even if you do reset all, some things are sticky.

And so I think for bulk edit, it's really important that the slate is clean and you don't do some kind of edit that you didn't intend. Okay, great. Thank you. This is what I wanted to hear. I just wanted to make sure we're all on the same page. I have a question. Um, I just a question. Um, would it be people be more expecting this?

Just go up to the actions and start over from there. No, no. Cause at least, well, at least on the RA side and session as its own button in various places. So I think we would expect a button like, like has been designed. Sounds good. Thank you. Um, the next part on our agenda is, uh, performance tests. Um, we conducted the performance tests for the users only at the, uh, the feature for bug edit performance.

And let me make this bigger as well.

Um, we'll include established performance for.

Item locations and item, item, statuses. However, this is not implemented yet. That's why it's blocked. Um, uh, we, we will be working on this, um, slightly later, but I would like to walk you through the, uh, quickly through the, uh, results of the performance test and, uh, how we approach that. It's a lot of technical information and I added the link to agenda.

If you want to spend some of your time looking at this, but it's rather a self exploratory, what we have done. So we, we run three, basically three tests, one, uh, we tested results of updating 10 records of users. So one user updates, 10 user records. And does. One, um, uh, per sec, one requests, a cent per second, and it's repeated 100 times.

So, um, those times that are, uh, recorded here are in milliseconds. Um, uh, I was surprised to see that the logging took as much time almost, uh, as the, uh, getting the link to download the much records.

So this keeping in mind, this is for the CSV approach. This is for the users records the next, uh, um, the next type of tests where we had again, one user updating 100 records of user data.

And then it was done during one second and repeat it 100 times.

And obviously the, the, the time, um, it's, um, growing up when we are, uh, more, but I think the performance is still pretty good. And the last one was that we tested the same, a one user updating a thousand records at once. And does it, uh, well, uh, one per second and does it 100 times.

So, um, For this amount of data, we did not encounter any problems when we were submitting the list of identifiers, the problems where once we, uh, around the query, this was the, this is the, uh, sorry, this is what we want here. Let's see.

And, um, so this is the query that was executed.

Oh, this type of query was executed. And then, um, Uh, we, we got the air due to the performance of the query for now. We just don't do anything. Um, we will be addressing it. Uh, sometime later if you scroll down, I think there should be a link to the, there was a link I will pass the link in our chat to the examples, uh, of the query.

And he a user identifiers that we use in our tests. Do you have any questions? Um, well they also provide some data on the overall system impact. That's what we see with data import. Like it may run quickly, but it uses a great deal of energy from the system. So I think with the users, the impact will not be that bad, but for the inventory.

That the impact of the bulk edit will definitely be, um, affecting, uh, other permissions other areas. I will keep it in mind. Well, once we start, once we start working on the,

once we start working on the performance for, for inventory, but I will ask the developers to double check the impact of the bulk of users

on the system. Yeah. I would mainly check also a check-in check-out because the wires, the user data quite a bit. So I could see if it was a lot of records being updated, can still those to them.

Micah also just, uh, with all this talk, I've just had another thought. So in our current system, when, um, Bulk editing is done in, let's say that's my primary experience. So, um, certain, um, so it could be that in our system, uh, kind of data import and bulk editing kind of happens within the same module. And, um, and that might be different in folio.

And you could just say that and then I'll stop. Um, but if not because records are being impacted, right? Like it affected it simultaneously potentially. Um, So, so we're going with this, sorry, is that like, I can set up a big task that, uh, for many, many, many records, and then there are other jobs, um, that have higher priority and they can interrupt mostly you end up with a queue, but can, it may interrupt.

They're allowed to interrupt ma to do their thing. And then once that is done, mine resumed. And so a, this again, could happen that there's a delay right. In starting something and finishing, but also overall system impact. Yeah. And that's what you're talking about here. We'll will we, will that be a possibility in folio two?

And do we have to somehow allow already for that or account for that? So you are asking if we can queue, if we can cue the jobs. So for example, if you have, uh, we, if you are importing the data, uh, at a given moment, and then someone else does the bulk edit and both of them are taking a lot of resources.

So you would like to the import complete first, before we start doing the bulk edit, right? This is for example, yes. Or it could be other types of things. Yes. Uh, what other type of things do you have in life? Well, because of it, because it's also impacting inventory. For example, you have pulse flips, um, that certainly run right to go and fetch things off the shelves, right there also querying this basically the same underlying data that they are, et cetera.

Right. So those, I think come into, I think they can run sooner. And then, and then I'm mostly just thinking, because because of the five colleges, you know, I might start something UMass might start something, start something, and we're basically all. Utilizing the same resources. And this is also an example of  something that we see, oh, APM H affects when you do the harvesting.

You're getting the data from inventory from SRS and it takes a lot of resources. So it impacts all the system. And especially if you do the full harvest, you should be doing it outside the regular, um, uh, hours because of its impact. And I, I see that what you are getting, uh, here, um, Sarah, I will need to run it by the developers and by the architects, uh, how this is being handled, because obviously we don't want to clog the system, um, with the, uh, Yeah, let, let me talk to them and I will get back to you on our next meeting, how we are going to handle that.

This is definitely a part of the performance, um, investigating and, uh, addressing performance issues. This was very good point. And also just going back to, uh, just now with locations, um, And then you, uh, brought up the harvest thing of the acronym that I can just never get straight when peeing, whatever it is.

I don't know why. I cannot say that one, lucky that you don't need to learn it by heart. Um, so I know it wasn't in our current system that, uh, so our, our discover layer, um, and I might be using the wrong words here. Right? So we call it our tack, how it displays, where something is located in the, in the discover layer.

And that also depending there can be like a slight delay there. So I'm also not quite sure what the, you know, what kind of resources are utilized there or the timing. W what the delay is or something again, you know, performance wise, something to, from what I know about archaic, Arctic has also a module on the, on the folio side, it's more Arctic or edge ARPAC I don't remember.

And I also am not fully familiar what this acronym stands for. I only know that the R stands for real time. So it means that it is actually queering the database for the location. So it, your, uh, discover layer since requests to, um, sense some request to the, uh, Inventory for your inventory and gets the information, uh, from inventory about the location of the item.

So once you do update the inventory, the, the location item, location, artwork, we'll get the updated information in real time. So if you request before the update happens, you will get the old location. And then after it will get the, the new one, we are running out of time. And I was hoping I will be able to do walk through of the, uh, um, user acceptance testing.

We most likely won't have time for this. I will put this in the slack channel. Um, this relates to the, um, Um, to the usability and the user or user acceptance testing for CSV approach, the environment was totally reset from what we had discussed on the beginning. And unfortunately, the setting, the users that we set up in the fall, uh, with the, the passwords are probably not there anymore.

I have created a user for, uh, for us to test and they will also provide a sample of the different IDs. So you don't need to, uh, look for the files. You are more than welcome to do them, but in case you don't want to spend time on selecting IDs, uh, you can use, uh, you will be able to use the files. Uh, as I mentioned, I will pass this in that slack channel.

And if possible, if you can do the, um, uh, eh, user acceptance testing for user records for CSV approach, uh, this week, that would be awesome. And if something in my, uh, instruction will be unclear, please feel free to reach out to me for clarification.

Thank you. Thank you all for your feedback and great conversation. And I'll see you in two weeks. Bye bye.