2022-8-23 Bulk Edit Working Group Meeting Notes

Development updates

• Development status
• Development team scrum board

• Buk edit  - Morning Glory Bugfest tickets:

• Bulk Edit Morning Glory Known issues (as of August 18, 2022):
  • UIBULKED-122 - Implement logic for record counts when items bulk edit is triggered by holdings id
  • MODEXPW-203 - "Fail to upload file" error with large amount of Users barcodes
  • MODEXPW-202 -Incorrect number of records changed on the confirmation banner  
• Performance tests results:
  • Item records: Bulk Edit Items App report [Morning Glory]
  • User records: Bulk Edit report for Users App [Morning Glory]
  •  Combined tests: Combined test for data export worker (Morning Glory)

Let's dive into the Morning Glory bugfest tickets. As you see, the list is significantly shorter than last time; we have three remaining stories. Two of them are in code review. One is in progress. All of them are related to making MOD data export horizontal scaling. And the issue is not only for Bulk Edit but also for other modules that use it for exporting reports, e.g., EDIFACT exports, eHoldings exports, Circulation log, and Bursar information. All of them are using this module. So the work we put in here will benefit others as well. The other good news is that we were able to resolve the issue where once the Bulk Edit completes, the confirmation banner needed to be manually refreshed to provide the number of records affected or updated. This is done. We are not able to recreate it anymore. So it's good news. The links to the performance tests are here, they have not changed since the last meeting, and we will definitely be doing more tests once those three stories regarding horizontal scaling are addressed, and we are sure how the system behaves when we have more than one module available. And the questions?

Erin  08:16
So the other two things that are listed as known issues, it looks like we're going to talk about one of them, the other is going to be worked in no Nolana?

Magda  08:25
So the first, 122, will be worked on in Nolana. 

• UIBULKED-122 - Implement logic for record counts when items bulk edit is triggered by holdings id

The second one, 203, is in progress now. The developer is looking into it. If ,we find a solution for this and we can squeeze it into Morning Glory then we will include it there. The issue occurs when you try to upload more than 3000 user records, which if this is the case and we we are not able to fix it in Morning Glory, I still feel it's okay if we move it to Nolana. I will know more about that at our next meeting.

• MODEXPW-203 - "Fail to upload file" error with large amount of Users barcodes

In terms of the development status, if we look at the SCRAM board, in addition to the bugs that we are addressing from bugfest, the team is also working on the backend functionality for Nolana so that UI work can start in the sprint that started yesterday. So we will be addressing issues in Nolana. The links are here if anyone wants to review them. if you have questions please let me know.

Expected behavior for: 

• UIBULKED-122 - Implement logic for record counts when items bulk edit is triggered by holdings id

Existing behavior: 

Magda  10:30
So I would like to move to the expected behavior for the bulk edits-122. I have slides here and I can also demonstrate the behavior if this is not clear. So what is happening right now, is that if we upload a file with 25 holding records, and out of those 25 Holdings, only 20 holdings find a match, there are 5 errors with no matches. So does this result screen make sense?

Erin  13:03

So what can we control here? Can we add a blue eye and show information? What are we able to do to try and provide clarity?

Magda  13:22
Just tell me what you think would make sense. And then, I can discuss this with the developers.

Erin  13:33
Lita chimed in the chat, and I definitely think more people should talk. But you know, that's valid, right? Y

Lita  13:43
This is a case we are used to seeing when you're working with holding IDs. And in this case, you want to edit all the items associated with those holding IDs. Some of those holdings are going to have more than one item.

Magda  14:02
Yes. And we will have the same behavior if we start, for example, editing holdings by providing instance IDs. There are many other use cases for this. So I just want us to devise a solution or approach that would make sense so that it will not confuse the user. Bob has his hand raised.

Bob 14:34
Yeah, I just want to be sure I understand the confusion. Is it because the user knows they loaded 20 and see the number 24?

Magda  14:45 - Magda clarified the issue with the displayed results again.

Erin  16:27
So, my first suggestion is, rather than using the word records, can we use the name of the identifier? Or say the name of the thing we're talking about, e.g., 24 item records, 24 or 24 item records matched, and five holdings UU IDs errored, or errors.

Lita  17:01
I don't really have a problem with this top screen, because I can see here in the column that says barcode if I have a barcode, I'm on an item. I would start to worry is if I had fewer numbers than I submitted.

Magda  17:25
This may happen as well, because if out of those 20 holdings, there were some holdings that did not have associated items, then you may end up say, 18 items, for example.

Erin  17:50
So I think adding the name of the record type in front of "record," and the type of the identifier in front of "error" would help.

Magda  18:01
Okay, so the top level would be 24 items records match. And on the errors, we will have 29 entries, and I need to investigate how we got to the number 29, to be honest because 29 is nowhere to be found here. But this is the bug on the technical side.

Erin 18:55
24 item records matched. And then you can say five UID errors. Or, yeah, I don't know the best way to handle plural versus singular there. I'm sure there's guidance somewhere about it.

Magda 19:15
Would it be okay as holdings identifiers errors?

Erin  19:21
I thought it would be easy just to use the name of the identifier, but I don't have a strong opinion about it. If anybody else does, please speak up.

Bob  19:34
On the left, you're using the term holdings UU IDs.

Magda  19:38
So you would like to preserve this naming? Correct?

Erin 19:44
I just think it would make sense in context. It looks like Christie has her hand up.

Christie  19:56
My question would be, are the errors always going to be related to the holdings UU IDs, because I can imagine that there is a holding with an item, and there was an error with the item?

Magda  20:13
You will see all the errors that occurred while matching the provided IDs.

Christie  20:28
So it will only be the holding UUIDs there?

Magda  20:35
In this case, the errors that are here will be IDS you have provided that don't match.

Chrisie  20:44
So my other question is, what happens if you have a holdings ID and it's got 10 items, but only nine items show up because one of them had an error? Would you even know about that?

Magda  21:09
This is a very good question. And why do you think would not? Why do you think that one of those would not show up?

Christie  21:25
That's a good question.

Erin  21:27
All that's happening here is you're giving a set of holdings record IDs, and then it should just return all of the item records that are on these holding UUIDs.

Magda 21:40
But this is a good question. So, for example, if, let's say, one of the item records, has some invalid data that prevents rendering it in the UI. Christy, is this the scenario you have in mind?

Christie 22:02
I didn't really have a specific scenario in mind. I was thinking about this from a quality control standpoint. How will I confirm that I'm changing everything I expected to change? And, looking at the errors, I would be really surprised if we'd never have that happen. And this scenario you gave is actually a really good one because we're finding that some of the data we migrated with is just not standing the test of time as we migrate from one version to another. So I think it's perfectly reasonable that we have items out there with data that will go stale and no longer be valid in the system or is corrupted for some reason. And you know, what happens in that scenario? And I would expect it to show in errors that, you know, there was one record that we couldn't pull for whatever reason.

Magda  23:16
So, in this case, let's say we have one item that did not show up. So we would add here a barcode and the comment, data corrupted. And then, you will need to know which is the value for holdings and which is the value.

Erin  23:44
In Christie's scenario, you might not even get a barcode back.

Magda  23:49
But you will have the UUID.

Erin  23:56
My question is, how does Bulk Edit know how many items it expects to get?

Magda  24:09
It doesn't know. It just gives you this is what it found for what it was given. We can track the errors that occur in the process. It will be kind of difficult to recreate this issue with the data. But let's come back to this at some point.

Bob  24:47
I just have a quick question. So what's happening down below under the errors is a simple match on the UID in the spreadsheet to the UUIDs in the system. So no items are really being checked. It's just a simple comparison. So, it doesn't come into play regarding this error message down here, right?

Chrisite 25:22
It may come into play up above when we're trying to display the items. But it won't even do that, because it's it didn't, I'm sorry, yeah, so it makes the match. And there are five items attached to the holdings. And he tries to display it in the upper section here; you might get an error, right? If it's corrupted or whatever way at all.

Magda  25:48
When the user saves all matches locally, if there is a problem with one of those records, that may cause a problem down the road. I would like to table this and come back to this conversation. Let me think about this. For now, obviously, we will not handle this case; the record will be excluded unless the problem is such that will prevent users from saving the matches found. I will come back to this. But this was a good point, Christie.

Bob  27:23
Can I just add one thing I thought of in terms of the messages about the record numbers in the result? It just occurred to me that a single box with multiple lines describing the results in one place might read better. Reporting the results of every type of record that returned and the errors, all in one place so you're not scanning in different places to make sense of the numbers. Just one idea.

Magda  28:36
Okay, we can talk about this too. My understanding is that if everything goes fine, if there are no errors, and you see this top 24, this is all you really need. And when there are errors, you have them in one place, followed by what went wrong. But I put this also on the list of things to discuss how to make it a little bit more prominent or the reporting more prominent. Are we any comments in the chat that I need to discuss before we move to the next part?

Lita in Chat: When we download a CSV of the preview file, do we get only the columns chosen or all the columns?

Magda 29:36
We get everything in CSV format, whatever is in the item record.

Magda
I would like to start with a few questions that were reoccurring. Several responses to the question about call number display in the preview and confirmation screen indicated a preference to see each call number prefixes and suffixes elements in separate columns. And I would like to understand what the use cases are. Why do you think it should be separate? I just want to make sure I understand.

Jackie
Well, one thing is that that's how you can see if for example somebody put the prefix in the call number field instead of a prefix field. And if you have them all together, you can't actually see

Magda
Yeah, it makes sense.

Erin
That's an excellent example.

Magda
This is a good point. If you combine everything, you will never know what is the part of the call number and what is the part of the prefix.

Sara
I think the examples are exactly right. If I need to do a cleanup job, then the call number prefix, etc., need to be separated.

Bob
So are we talking about just this preview screen of 10 records? Or are we talking about the download file?

Magda 
The download file will have everything and it will be structured like the record is structured. So you will have everything separately, the prefix call number and suffix will be done separately. But we are talking about preview we are talking about the are you sure form and we are we are talking about their confirmation, and all those places when those records are being displayed.

Magda
The other question about the preview, are you sure form and the configuration screen is someone mentioned (it was Duke) the need to see the holding statements and holdings notes.

Leeda
So I checked in with my leader who helped me fill out this survey. And she said both types of data would be useful for analyzing the scope, ensuring accuracy and troubleshooting when needed. But again, since we're just talking about a 10 to 20 item preview screen. I think as long as we can get to it on that CSV, we're fine.

Magda
My concern is that if we are holding statements and notes, I don't know how lengthy those notes can be.

Leeda
It was alright for concatenating the summary statement and then for the holding notes I think just the presence or absence of a note would be fine. Again, it's just a case of when we get ready to run a bulk edit inevitably there can be records in there that we didn't really mean to hit.

Magda
Leeda, do you want those to be columns in the preview?

Leeda 
The feedback I got was yes, but I'm not sure certain that the that they understand that you can capture that in the CSV. I will get back to you on that one. If I had to make an executive decision, I would say it's not absolutely necessary. But I do like the idea if we can have a presence or absence. Kind of like a yes, no, and then download the whole thing to find out what's there.

Magda 
What about electronic access? This was also mentioned. Electronic access has several fields and are long. It would make the display pretty difficult. So how do we concatenate, or do we leave them out?

Leeda 
I could see again that it would be more for ensuring that you're touching the right records. Because I know at Duke, we have a lot of titles where we hold the print and the electronic. And we don't want to touch the wrong title in that case.

Sara
I would not rely on the Preview or 10 records. I would have to download them to make sure that I wasn't changing something that I did not intend to change. If I'm understanding you correctly everything will be the download file. Am I understanding that correctly?

Unknown Speaker  14:12  
That's correct, that all Holdings Record will be in CSV file.

Sara
So, in this case of only being able to see 10, it's unclear how this will help. Maybe then it's not worth putting so much time and effort into getting all these fields exactly right or including too many or too few.

Magda
And my thinking runs along the same lines as well as Sarah. To make you feel a little bit better, we plan to change the preview to allow pagination to the results, etc. But this is not happening in the Nolana. It will be coming later once we start building the query tool. Because when you submit the list of identifiers, we can assume that those identifiers were based on a query or you have the items in front of you, and you scanned the barcode. So, you know what dataset you are acting on. Once we get to running queries, then the top 10 is not going to do anything for you because you will need to see the results of the query. This is when we will be making changes to the preview. But this is not in Nolana. We may start talking about this in Orchid and then follow up in Poppy. So if is something in our pipeline, it is just not available here. So going back, do I understand what you're saying is that since the preview is only an abbreviated preview, you will need to go to CSV anyway. So we can be a little bit less verbose in defining what fields need the need in the preview screen. Is this correct?

Leeda
Yeah, I think so.

Sara
I just wanted to follow up briefly on something that you just think it's worth noting. Just because I have a stack of 20 books, and I can scan their barcodes into a file, does not necessarily mean that I know what is in their records. I think this is worth noting because this happens to me all the time. For example, I get a list of barcodes that need to go on display. I'm changing the temporary location. A pop-up note for the circulation people is supposed to be added. But sometimes it turns out there's already information there. And so then, if I do a bulk edit without checking first whether or not that field already has information, which I cannot tell from just having the physical book in my hand, I will override that information. So we should not assume that just because we have a stack of physical items that I know what is in the records. efforts.

Magda
That's a very good point because I didn't think about that.

Magda
So, if I still would like to take a look at the specific questions that I have from the responses to the survey. So again, we are talking about identifiers here. My question is about the OCLC number. Is there an OCLC number on the holdings or just on the instance.

Sara
No, it is only the instance level from the MARC source.

Magda
Okay. So I think someone who suggested OCLC understood that when we submit the list of OCLC numbers to identify instances and then from instances go to holdings, this is something that we will be addressing through the query. So I will table this for now, as well.

Magda
The next question was about the columns. I see that we need to put a call number with all the elements in separate columns, including their call number type. So we already talked about the holding statement, and electronic access. My concern is if we add electronic access to the preview, the rendering will be difficult at this point. And then the following questions are similar. One of the respondents suggested let the user manipulate the order. This is the long term goal, but it will not be happening for Nolana. So for now we need to have a predefined list of columns.

Magda
And the last question was about the columns in the action menu that are not selected.

Unknown Speaker  23:18  
For statistical codes - how they should be separate because this is a multiple entry field? Should we use a commas separated list or a new line? I

Jennifer
It looks like for notes they're using pipes. So that's probably fine for that two.

Magda
And call numbers we actually covered. We also have a response suggesting staff suppress. There is no staff suppress on the holdings. So I do believe the user or the respondent meant at the instance level and this will again be drilling down to the housings record that will be covered by the query. Electronic access as I mentioned, worries me about its rendering of it. Is the URL only valid or do you need the type of electronic access as well?

Christie
I think that in our use case, we would want to see everything but the URL because the URL is unique. And maybe the CSV is enough for this. We frequently put our platform names as the link text in the URL. And we're frequently normalizing the material specified statements. So for us it would be materials specified statements, link text, the notes, and the type.

Unknown Speaker  25:53  
And I understand that you may want to change that. In Nolana, we will only  support changes to be located holdings location, temporary and permanent locations. The other fields will come later. But do you want to see this information rendered in column on the preview screen?

Christie
I don't remember if we asked for this or not in the survey. I don't know that this is a high priority for us. I do know that we have a use case for changing that, which is why I spoke up.

Magda
And I understand that these fields that will be supported in the in-app approach for Holdings Record edits, but I'm not sure we need to display this data in the columns.

Sara
If this is just about the confirmation screen, I think it was Leeda who earlier said that it would be worthwhile having just kind of something indicating if it is present or not present just to have that added knowledge that you didn't get the ebook rather than the print book. Otherwise, I don't think the confirmation screen needs all this detail. In the CSV, it does need to be there, including the URL string, which I have had to do bulk changes too frequently. There are definitely use cases for needing to change the URL in batch, but that can be in the CSV, obviously.

Magda
There was one mention of including associated order and purchase order lines. Is there any feedback on that?

Magda
That would be something more at the query level. I think the usefulness of the preview screen would be more to preview the columns that I was actually taking an action on. I'm starting to think about it more in that respect.

Magda
In chat, I see Christie is asking about the ability to change part of a string. Yes, in Nolana, this is available to cover common use cases, for example, changes in email domain. Find and replace functionality will be added. It will be available for other fields down the road. Some fields will not be supported in Find/Replace like location and date as it is not applicable. Is that a limitation?

Christie
That is a limitation but there is a process.

Magda

We are at the top of the hour. See you all next time.