Page Comparison

Bulk edit documentation for Morning Glory 

Thank you (OLD ACCOUNT) Erin Nettifee and  @Bob Scheier, @Emma Gollner, @Christine Tobias and @Amanda Ros!

Morning Glory release notes

Bulk edit: 

• required configuration
• known limitations
• known issues

Bulk edit documentation for Morning Glory 

Thank you (OLD ACCOUNT) Erin Nettifee and  @Bob Scheier, @Emma Gollner, @Christine Tobias and @Amanda Ros!

Morning Glory release notes

Bulk edit: 

• required configuration
• known limitations
• known issues

Magda 16:18  
So let's go to our agenda again. And let's move to the next part, which is the Morning Glory release notes. As you know, Morning Glory was released on September 16th, and I would like to walk you through some points on the release notes. So first, the new app. There is only one a new app which is Bulk Edit in Morning Glory. And a quick overview of what we support. In the in-app approach, inventory items temporary and permanent locations and item statuses. In the CSV approach, user records. There are two sets of permissions that are required for this application, in-app edit and view , and CSV edit and view. Depending on those permissions, a user can access those applications. There are known there are Known limitations. As our performance tests discovered, in Morning Glory, you can only do a Bulk Edit of 2500 user records at the time. And for items, the limit is 10,000. This is a lot less than we were hoping for. Unfortunately, this is what it is for Morning Glory. We are looking at architectural changes to support a larger data set in the future.

Magda 18:54  
The last part here, module configuration, is for those who are hosting themselves. For everybody else, this is being set by the hoster. Those are the memory requirements for the container that runs the module data export worker, which is being used in Bulk Edit

Erin  19:01  
The performance test, the performance work that you were talking about, is tracked in a particular Jira?

Magda  19:09  
JIRA and reports. I will link the list. There is a new performance testing Jira that we started in the Morning Glory. Some of the work will spill over into Nolana. Starting next week, we will be working on bug fixes for the functionality that was already delivered and testing, performance testing, adding API tests, and adding automated tests for the UI. So, most of the tests can be run automatically.

Regarding the known issues, there are two Jiras that are linked to Bulk Edit. One which we discussed a couple of meetings ago, has to do with Bulk Edit results triggered by holdings identifiers. When the mapping between holdings and items is not one-to-one, it may happen that the list of identifiers is shorter than the list of matched items. This will need to be addressed in Nolana. We will be working on this functionality.

Then the next issue is a little bit more serious. When you update datasets larger or larger than 10,000 records, there is a significant delay of up to several minutes before the display of the preview screen, i.e., the "are you sure" form. We are working on improvements.

Unknown Speaker  22Erin  22:59  
did Did you ever see it timeout, or would it just it would always come?

Unknown Speaker  23Magda  23:05  
So, there are instances that we saw in our tests and this is also linked to this bulk edit it is timing out not often we saw this in we saw this rarelyWe saw this, but it rarely happened. So I cannot tell you know it never happened. Yes, we saw that happened. But if this is the case, you just need to submit the the bucket you need to start it all over. Okay, the must submit the Bulk Edit again. The data is not committed at this point, the data is not committed. So this is because you have to wait and then hit the button. Exactly commit the changes. And the additional questions.Unknown Speaker  23:55  . Any additional questions?

Erin  23:55
Jennifer has a question in the chat says . "Does it affect any other operations happening at that time?"Unknown Speaker  24:02

Magda 24:34  
You mean on the other operations? If somebody works in inventory, for example on make changes? Denise, is this your question?

Unknown Speaker  24:13  
So for example, what happens if you have this like a theme to import, single record import and then check ins and checkouts like everything happened? You know, we would say to import that, you know it can't disrupt other operations and it should record so I was just wondering what happened here.

Unknown Speaker  24:34  
So the problems we saw the problems that are happening and the slowness that you see is caused by a mod data export worker that is being used. So separate from the inventory module or a data import module there should be no interference. However, if somebody exports a defect at this time or a circulation logs are being exported. Or Bursar is being exported at the same time, it will have an effect effect. However, in our tests, when we did the tests we ran, we ran all of those three export at the same time, so EDIFACT, Bursar circulation and bulk edit. And that's why we got down to 10,000 records for items for example, because if we increase the number, it will definitely affect other export that may happen at the same time. Thank you. Any other questions? So the problems we saw with slowness are is caused by the mod data export worker that is being used. So separate from the inventory module or a data import module, there should be no interference. However, if somebody exports EDIFACT or circulation logs are being exported, or the bursar is being exported at the same time, it will have an effect.  However, in our tests, when we ran all of those three exports at the same time, EDIFACT, bursar, and circulation with a Bulk Edit of more than 10,000 records, that is when we say it affects other exports happening at the same time. Any other questions?

Magda 34:27
But I would like to start a conversation about the query tool that is needed. We actually started the last meeting. I have some high-level Balsamiq mockups that I would like to walk through and get your feedback on. Please feel free to interrupt me. So this is the balsamic implementation rendering of our Bulk Edit. This is something that we talked about--switching the order of the identifiers, putting them on top, and then having the query or the identifiers below. So this is what is represented here. The two links at the bottom, save the query as, and open the existing list; these are out of the scope of our conversation for now. This is something we will come back to later.

Image Added

For the purpose of this conversation, we are on the Query tab in bulk edit. There is a new button called Build query. When the user clicks the button, it opens a new pop-up that looks like this. It's a long data entry textbox with a test button disabled.

Erin  37:06  
So what you are proposing is a redesign from what's currently implemented?

Magda  37:11  
Not really redesigning, adding. We will get to this. Basically, I'm adding a new button that will allow us to build a query.

Magda  37:39  
And then, in the Poppy release, we will add additional record types. In Orchid, we will support just one record type. So in order to know what you have, there will be additional labels showing that you have access to users inventory items, circulation loads, and then circulation requests, etc., to build your query. You can still only edit one record type using Bulk Edit. But in Poppy, you will be able to access additional record types to build your query. Any questions or comments?

Orchid release

Image Added

Poppy release

Image Added

Magda  9:14  
The user clicks on the search textbox, and the list of available properties is displayed.

Image Added

Magda  39:42  
The user selects the patron group, one of the available properties. Then the list of available operators is displayed. The list of operators will depend on the selected field. We will start with equal, in, not equal, and not in.  The properties list is preserved so the user can go back and change it.

Image Added

Image Added

Erin 40:34  
Should we use words instead of symbols for consistency in that drop-down?

Thomas  41:05  
I think these are standard search operators, but still, I think I would just do equals, not equals, in, and not in.

Magda  41:17  
Okay, thank you. We selected the operator, and now we get the list the pre-populated list of available options.

Image Added

Magda 41:19
So that would obviously be in place only for the controlled vocabulary. So if, in some cases, the user selects a title, then obviously, we will not pre-populate the drop-down with all 10 million titles that the collection may have. Instead, we would present a text entry box. But for this example, we would pre-populate the combo with the existing values. For options with hundreds of options like locations, for example, the drop-down like what already exists in inventory where what you type narrows down the list. Any question?

Magda  42:58
The next option is the Boolean operator, which will automatically be displayed. And user can choose to continue to add to the query or just not select anything. Please note that once the first option is entered, the test button is enabled. It was disabled until now.

Image Added

And here's the example when the user continued to build the query by adding a boolean operator.

Image Added

Here's the example where the user wrote the query and click the test button. There is a gray progress bar that shows that the records are being counted. And in this example, we have more than 100,000 records returned with the notification, please narrow down your search. The question here is what the number should be and how we should handle that number. I do believe we talked about this at some point, and it was proposed that this would be configurable. However, we need to keep in mind that having a big number of records will definitely impact the performance of Bulk Edit.

Image Added

Erin  45:06  
Is this a number across the board for queries? Or is it by record type?

Magda  45:11  
That would be across the queries. So this is something we definitely will talk about later. But I would like you to keep it in mind. Does anyone have quick feedback on if there should be a limit for the Bulk Edit?

Erin  45:57  
I mean, I would think that it probably should if we're talking about our current example, where we have a 25,000 limit. I would want 10,000 as the limit here.

Magda  46:12  
Correction. Not 25,000 rather, it is 2,500. This is just a proposal. As I said, we'll definitely come back to this later. 

So we hit the test button for the query, and the preview of the resulting 1000 records displays. This preview does not show all the records, but just some of them. You have the option of selecting columns to see what you want to see by checking and unchecking the columns. Once the preview is populated, there is a new button, Run query.

Image Added

So the run the query button will close the pop-up and bring the user back to the landing page. And what is happening here is:

• The query that was created in the pop-up is displayed here on the left side as read-only.
• The list of the identified matching records is listed.
• We can paginate through the results similarly to how this is done in inventory by 100 records.
• There is also the option of checkboxes where the users can narrow down their results by selecting checkboxes. So in our example, we have 1000 records that were found to match the query. But out of those 1000, Only 100 were selected. And again, there is the option of changing the columns. In this example, the user selected 100 records, and with these 100 records, there will be the standard behavior of Bulk Edit, meaning you can save those 100 records into the CSV file. And then, when you start Bulk Edit, the edits will only affect those 100 selected records.

Image Added

Mark  49:24  
When you get your search results, are the checkboxes automatically prepopulated?

Magda  49:40  
I would say yes they will be populated. But this can be discussed. We will not get to check boxes in Orchid, that's for sure. But I would say yes, let's prepopulate and by selecting the top checkbox, it will unselect the whole 100 that are displayed.

Magda  50:25  
So the question I have for this group is, what is a easonable number to paginate? If you have 10,000 records that match your search criteria, will you be paginating through those 10,000? By hundreds?

Erin 51:18  
I'm only paginating to verify that I didn't do something dumb. I might page through the first part of it. But I'm not going to page through the entire set.

Leeda  51:35  
I would think our filtering and things like that would be done outside the system in a spreadsheet.

Erin  51:42  
You might want to download the entire set, which you'll be able to do, but you're not going to paginate through that in that UI?

Leeda 51:52  
Yeah, totally agree with that.

Magda  51:56  
So what will be the cutoff? How many records will be rendered?

Erin  52:10  
To me, that screen looks normal at paginating through 100. And then it pulled 100 records or pulled 1000 records. And I would, probably expect that if I clicked 10 times I would paginate through all the records. But I wouldn't probably do that very often. I'm sorry that I know that's kind of like a wishy-washy answer.

Erin 53:28  
Sure, that makes sense.

Bob  53:31  
The thing is, I might want to see the end of the results in the file just to make sure if I'm looking for any anomalies.

Erin  53:43  
Yeah, that also makes sense. You know, if I'm expecting to get results all the way through to Z.

Bob  53:49  
I don't know how the results are coming back. I just see the top of the file. So maybe there are no problems with the top of the file. But halfway down the file, there's a bunch of errors for some reason, where I'm getting things back that I shouldn't or that I didn't expect.

Magda  54:05  
So the other option would be to sort it the other way around. However, this is an expensive operation, the sorting of 10,000 records that we need to be mindful of. So it may make more sense maybe to add another link that will take you to the last one.

Bob 54:29  
Can we export this to look at it in a spreadsheet?

Magda 54:32  
Yes. The existing behavior of the spreadsheet that we have is preserved. So you can still save the list of matching records, as you do right now in Bulk Edit.

Bob  54:51  
I would probably do that. You know, because I would want to see a whole bunch just look for any problems.

Unknown Speaker  55:00  
So do you see the value in adding this preview? Or not?

Unknown Speaker  55:09  
I think people were expressing an interest in seeing more than 10. You know, so I think this is valuable, but it doesn't have to go to 200,000 Records. In my opinion.

Amanda 55:25  
As long as we will continue to be able to export the data into an Excel spreadsheet, I don't see it as a problem because we're all used to manipulating data in Excel. So it would probably be easier for us to find those anomalies. I'm probably going to flip through a few pages just to make sure that it looks like I expect it to, but I will still download the data. So you know, I am not going to. This is not the hill that I'm going to die on about how many records we need to be able to scroll through.

Bob 56:54  
Yeah, I mean, it seems to me that both are valuable tools for looking at the data and what should dictate how much data should be displayed is the system performance.

Magda 57:10  
So this is a very good point, Bob, because we have to keep two things in mind, the performance displaying those records will definitely have an impact on the performance. But if this is something that needs to needs to happen, we will need to take it into account. But also to develop this functionality. It's not a simple thing, this is a lot of UI development. So if you feel that you need this, then we will obviously allocate resources to build it. If you think this is nice to have but you can live without it, then we will allocate the resources differently.

Erin  28:13  
If we can show the first 500 records with the option to download the entire record set if you got more than that in your results. If we can say something concretely in the documentation, I think that is fine for where we are at right now. I think, eventually, people will expect the full pagination because that's what you get in applications like Gmail and other online tools.

Magda  58:57  
I see Sara's question in the chat. That. Yeah, this is only my rendering of a few records. In fact, as Erin suggested, mentioned, this is the scroll down that will show you the rest. This is actually how the inventory works as well. On the screen, you don't see that 100 Unless you usually have a very small result. And I know Sarah, you were the person who was asking mostly about this. Do you have any other comments?

Sara  59:41  
No, I just wanted to make sure because somebody mentioned 10 or something at some point. I just wanted to ensure that we were that it was actually, as I was seeing it, that there was a scroll bar that would take us through the first 100 and that it wasn't in debate about the 100 but more about whether we can do pagination. I guess I'm from other systems and others here on Aleph there is a pop-up modal that you could scroll through. And then, you could also download it and put it in Excel to manipulate it in detail. And so I guess the pagination, while pagination works, it is a very, very cumbersome and FOLIO. It takes forever to get from one page to the next page and to the next page, right?

Magda 60:01
So would you rather have a thousand on one page instead of pagination?

Sara 60:33
I would prefer that, but I understand this is costly. I just did not want to go back to 10 selected.

Magda 60:48

I hear you. The simpler way right now is to change the preview from 10 to 100 and then download the rest to see more. We have two minutes left, so we are not going to get to permissions. We will get to that at the next meeting, or I will reach out to you on Slack. If you have more feedback, feel free to reach out to me. Thank you. Bye.