Page Comparison

Housekeeping

• Attendees  - please add your name to the list of attendees
• Meeting host -  please turn on Transcript option for the meeting  - do we still need that?
• Updates on bulk edit rancher (aka bulk-edit-perf environment)

We still need someone to take on upkeep / edits of the Bulk Edit app documentation ((OLD ACCOUNT) Erin Nettifee ) - can you ask at your institutions / SIGs if you are not able to do so?

Magda Zacharska] 10:11:10
The next topic is the review of functionality.

[Magda Zacharska] 10:11:20
I have a Powerpoint presentation that I would like to walk through and then do a quick demo of the functionality we have done.

[Magda Zacharska] 10:11:37
Please feel free to interrupt me while I go through the slides.

[Magda Zacharska] 10:11:52
So in the Nolana there are 3 main functional features, one to support user data updates using in app approach. For item records we implemented locations and statuses. We also added loan types.

[Magda Zacharska] 10:12:18
Records we added to already a implemented holdings, locations, and statuses.

[Magda Zacharska] 10:12:25
And for holdings, we started with the locations as well.

[Magda Zacharska] 10:12:34
Those 3 functional areas are completed. We are still testing. Possibly some bugs will be discovered. But the main development has been completed.

[Magda Zacharska] 10:12:47
In addition to that, we have a series of NFS testing which includes automated tests for the backend.

[Magda Zacharska] 10:12:57
This is Karate for front and releasing the modules and updating that technology, especially for the spring application we needed.

[Magda Zacharska] 10:13:18
So what have we done while working on those features and some changes to the Ui?

[Magda Zacharska] 10:13:31
Some of them were mentioned during user acceptance testing at the end of Morning Glory.

[Magda Zacharska] 10:13:39
One of them was to change the order of components in the left pane, where the user selects the record type first, and then selects the identifiers.

[Magda Zacharska] 10:14:01
The next suggestion from user acceptance testing was preserving the selection of the columns.

[Magda Zacharska] 10:14:10
Now the user can select the columns that are displayed on the landing page when the matching records are displayed, and the selection of those columns is preserved on the are you sure form where we preview the changes that will be made and also on the confirmation screen once the changes are committed.

[Magda Zacharska] 10:14:39
And the last Ui improvement is on the export manager.

Magda Zacharska] 10:14:50
We can now filter out Bulk Edit jobs and also by statuses: successful, in progress, or failed.

[Erin Nettifee] 10:15:05
Bulk Edit does not use scheduled right?

[Magda Zacharska] 10:15:28
Correct. In this case it means that something went wrong with the uploading the identifiers and the job never progress further.

[Magda Zacharska] 10:15:38
So this is something that is in a not expected behavior. When you click on the export manager on the row, it will tell you the details of what went wrong. But you are correct. The scheduling of the Bulk Edite Jobs is not implemented yet. It is planned for the later releases.

[Magda Zacharska] 10:16:01
And the other questions, the comments.

[Magda Zacharska] 10:16:08
The nice thing I will move to that user data in the in-app approach. We will be supporting, in addition to the CSV approach that was delivered in the scope of Morning Gloria, on the request of community, we also added support for the in-app approach, starting with the most common changes: change in email hosting, expiration date update, and patron group.

[Magda Zacharska] 10:17:12
On the item records Bulk Edit, we can also do the changes to temporary loan type and permanent loan type.

[Magda Zacharska] 10:17:17
In addition to already implemented support for a temporary item location, permanent item location, and item status.

[Magda Zacharska] 10:17:35
For the holdings records. we added support of temporary and permanent holdings locations.

[Erin Nettifee] 10:17:50
And this is just the inventory records this is that's that is correct? Not SRS?

[Magda Zacharska] 10:18:02
That is correct. This is only for records that have a source folio.

[Magda Zacharska] 10:18:04
Okay, So let's do a quick demo

Note: see recording for demo. Notes below are just regarding questions and comments that arose.

[Erin Nettifee] 10:19:34
Is the the fact that the actions are great out here mean that is the only choice? Or could you choose a different action? So if I wanted to, for example, erase remove the expiration date from a set of user accounts, I would have to use the CSV approach?

[Magda Zacharska] 10:20:19
We don't support setting it to empty Also the same with Patron Group. Patron group which is a required field, and it needs to be populated.

[Erin Nettifee] 10:20:30
Expiration data is not required on the user record.

[Magda Zacharska] 10:20:34
So for now we can only support replace with something.

[Erin Nettifee] 10:20:45
We will need to add this functionality separately.

[Magda Zacharska] 10:21:47
So you can still download the CSV file and see all the records. And you see the records in its the entire record, not only the columns you have selected.

[Magda Zacharska] 10:22:13
And we are committing the changes, and here is the confirmation of the of the changes we have a committed It's also the the option of downloading the the change records.

[Magda Zacharska] 10:22:33
Any questions on on this functionality

10:28:10
Note: There was an error when Magda tried to demo Bulk Edit for holdings records. She said that this was due to a code change on the back end that the UI did not handle correctly. This will be address in bugfest. the issue.

[Magda Zacharska] 10:29:02
So in I would like now to move to the user acceptance testing and the scenarios.

[Magda Zacharska] 10:29:19
I change a little bit. approach this time. Instead of sending the survey and awaiting for your responses, I would like to use the Excel spreadsheet approach.

Magda Zacharska] 10:29:38
This is with a spreadsheet that contains the steps for each record type and each step that needs to happen.

[Magda Zacharska] 10:30:12
Start, and all the steps that we that we have that I showed you today.

[Magda Zacharska] 10:30:19
I added column for each institution that is on the Bulk Edit Working Group list. If your institution has more than one representative, if you want to add your comments, please clone the column and and add your comments in this empty fields. Type Pass, Not Pass, or any other comments that are applicable.

[Magda Zacharska] 10:30:59
Since we agreed that we are going to use the bugfest environment and bugfest starts next week, I will re-post the link to this document in our slack channel with the logging information to the bugfest environment and we will have 2 weeks to complete the the user acceptance testing.

[Erin Nettifee] 10:31:50
Will we all be doing the same editing on the same records?

[Magda Zacharska] 10:32:16
It's up to you because if we I can provide you with the list of records identifiers, then you will be using the same files, and you will be most likely stepping on each other toes, if you do those at the same time.

[Magda Zacharska] 10:32:25
If you can use your own for identifiers or I can provide you with the multiple list of identifiers, and then you will be marking which one you use. If you use your own file please drop those on the Google Drive in the folder.

[Jennifer Eustis (she/her)] 10:34:26
Magda, I just have a quick question. Is the UAT testing in addition to the tests we are doing TestRails?

[Magda Zacharska] 10:34:34
That's correct.

[Magda Zacharska] 10:34:43
We do have a lot of added tests in in TestRails, but this is a slightly different point of view. You can use your

UXPROD-3842  Bulk edit - architectural improvements

The changes are needed because the current implementation is not scalable enough to support larger data.  

• Results of current performance tests conducted by the team
• Proposed redesign

[Magda Zacharska] 10:36:18
We have less than 30 min left and I would like to spend this time talking about that and getting your feedback regarding architectural improvements. This is a a big endeavor to make architectural changes, we need to start them early to make sure we have enough time to test it properly.

[Magda Zacharska] 10:36:50
And this will definitely affect the Bulk Edit road map I don't think we will be able to talk about this today.

[Magda Zacharska] 10:36:59
I'm also in the process of updating a Bulk Edit road map, so we will return to to this next time we meet.

[Magda Zacharska] 10:37:10
So why did we decided to make architectural improvements? In Morning Glory when we did performance testing, the goal was to support editing at least 100,000 records?

[Magda Zacharska] 10:37:31
We realize that the architecture does not support that, and the performance is even worse if there are other application using export manager: exporting EDIFACT records, Bursar exprot, or circulation log exports.

[Magda Zacharska] 10:37:59
All of those when they are happening they have an impact on bulk edit.

[Magda Zacharska] 10:38:03
So, after further analysis of the code our Solution Architect proposed we redesign and separate Bulk Edit from Export Manager so that it is not affected by all the other things that are happening while other exports export the data. I added a link to the performance results from Morning Glory, and you can take a look at them when you have a chance.

[Magda Zacharska] 10:38:50
I added the link to the proposed proposed design, and I would like to spend a couple of minutes talking about it. I will not be going into the technical details, but there is one part I would like to spend a couple of minutes on. It's a bulk operation states. Because this describes how the data is being processed.

• New - This is the the moment when the user uploads the identifiers or runs the query.

• Retrieving

  records

  -

  This

  is

  when

  the

  user

  retreives

  retrieves the

  records.So

  we

  submitted

  the

  the

  way

  we

  want

  to

  identify

  the

  records,

  either

  identifiers

  or

  the

  query,

  and

  the

  records

  are

  being

  retrieved

  from

  the

  storage

  storage

  modules

  .

Image Removed

• . This is the second state.

• Saving records - This is the third state. This is the snapshot of the records that you get when you run the query.

• Those files

• are written to a CSV file, but this is basically a file that is created in export manager.

• But we will move this file to the

• Bulk Edit shared storage. This will be not model internal storage.

• This will be a storage in

• S3 Armenia for those who are not using Amazon

• Web Services. This is the file where

• it will be stored.

• This is not the

• CSV file

• you are downloading to your local machine.

• This is something that is still happening

• in the system.

• Data Modification - This is when the user specifies what changes will happen either in the in

• -app approach or in the

• CSV approach for user records.

• Review Changes - This is where the records are being displayed on

• the are you sure

• form.
• Apply Changes - This is when the changes are being committed to the

• records stored in the

• storage modules.

• So this is

• saving the data.

• Suspended -  This is one of the statuses that we will not support

• in Orchid. This will be implemented later, when the user will be able to suspend

• the bulk edit. This will obviously require further discussion.

• Completed - This is self self-explanatory

• . It is when the job that was completed

• with no errors.

[Magda Zacharska] 10:42:23
The next one. This will errors when you will get the notification, and that it completed.

[Magda Zacharska] 10:42:30
But they were errors, and you will still see the errors in the error accordion, and the the next one is cancelled.

• Completed with errors - This is when the job is completed with errors and you will still see the errors in the error accordion.
• Cancelled - We are not going to support canceling jobs in Orchid. This is something to discuss later.
• Scheduled - We are not going to support canceling jobs in Orchid.

• This is something to discuss later

• .
• Failed - This is when something went wrong

• and

• the operation did not complete.

Image Added

[Erin Nettifee] 10:43:08
Can I ask some questions, Magna: Sure. So this list of statuses is this something that is just part of the new design? [Magda Zacharska] 10:43:19
Is And is this something that would end up being exposed in an that the bulkheaded Bulk Edit app or both?

[Magda Zacharska] 10:43:25
Yes, so th. This is a part of the new design, and it will be something that will be . exp.[Magda Zacharska] 10:43:32
Then exposed in Bug edit, and the way we are exposed in Bulk Edit. We are going to do this .[Erin Nettifee] 10:43:37
It's by by an adding additional tab to back edit. Please let me know if you can see this is probably a little bit so small.[Erin Nettifee] 10:43:49
But yeah, no, this is better. now. yeah, that's a little bit betterBulk Edit. In addition to the identifier and query tabs there will be the logs tab which will again be driven by permissions.

[Magda Zacharska] 10:44:10
But if you have permission, you can see the logs and the standard filtering by status, by record type, and by type of Bulk Edit operation--if it was Bulk Edit edit or delete. And also when the job started when the job ended.

Image Added

[Erin Nettifee] Okay, So this is kind of like deciding to build our own version of the export manager just for Bulk Edit so that it can be customized to Bulk Edit's particular needs.

[Magda Zacharska] 10:4344:53
Okay, so you'll see this in other logs. tab. So in addition to identifier query and logs, this would be again driven by permissions.Exactly. It has a more information. You get more data here.

Image Added

[Magda Zacharska] 10:4445:1012
But if  So what you have permission, you can see the log and the standard for your filtering and the functionality by status by record type.

[Magda Zacharska] 10:44:24
What recorded type was export and modified.

[Magda Zacharska] 10:44:28
And then type of bucketed operation it was if it was bucketed or bug delete.

[Erin Nettifee] 10:44:35
When the job started when the job ended. Okay, So this is kind of like deciding to build our own version of the export manager stuff just for bulk edit, so that it can be customized to bulk at

[Magda Zacharska] 10:44:53
its ,

[Erin Nettifee] 10:44:59
It has a more information. You get more data here. So what you have here you have okay, barg, edit operation, type, record type status.

[Magda Zacharska] 10:45:12
Those are the status that we mentioned. this is who ran it when it started running when it ended running the number of records.

[Erin Nettifee] 10:45:24
That was okay affected. How many records like how many records were requested?

[Magda Zacharska] 10:45:32
How many records were were processed. Was it the editing in app or Manual Manual? is the

[Magda Zacharska] 10:45:41
Csv. approach, name a replacement, and here you have dogs, information or empty.

[Magda Zacharska] 10:45:48
If the actions are empty, it means no files were created, and it's at this point no files are available.

[Magda Zacharska] 10:45:57
When you click on, Doc, this will be the next screen shot up.

[Magda Zacharska] 10:46:02
This is this: again. so clicking on the action menu will give you the option to download the file, and you can download each of the files that are being created in the process of of bug editing so

[Magda Zacharska] 10:46:25
First it starts with the with the file. you can retrieve the file that you use to trigger the bug edit.

[Magda Zacharska] 10:46:33
Then the matching records errors that were accounted during the matching the the preview of the changes.

[Magda Zacharska] 10:46:42
This is what is the equivalent on our issue? Form, then the list of it.

[Magda Zacharska] 10:46:50
Updated records and the file with er errors that were encountered by when the changes were committed.

[Erin Nettifee] 10:47:02
And so Thomas is asking about in the chat about whether the status is, and information that we're seeing here would be useful in the export and the export manager.

[Erin Nettifee] 10:47:15
App. But I think what i'm guessing is that when this is implemented, there will no longer be bulk at it.

[Magda Zacharska] 10:47:21
Stuff in export manager will just be all encompassed in its own interface.

[Magda Zacharska] 10:47:31
So when we go to the statuses there's one part that happens still in in the export manager.

[Magda Zacharska] 10:47:39
So when when the records are being retrieved from the back end.

[Erin Nettifee] 10:47:44
Okay, So this is what is happening in the export manager.

[Magda Zacharska] 10:47:50
But the file that is being created is being moved to debug edit short storage, so the file will not be accessible.

[Magda Zacharska] 10:47:59
We will see that the records are being retrieved.

[Magda Zacharska] 10:48:02
But we will not have those files at all in export management.Bulk Edit Operation Type, Record Type, Status, those are the status that we mentioned, who ran it, when it started running, when it ended running , the number of records requested, how many records were were processed., was it the editing in app or Manual/CSV, and Actions. If Actions is empty it means no files were created and no files are available. When you click on the three dots you will have the option to download the files, and you can download each of the files that are being created in the process of of Bulk Editing. 

[Erin Nettifee] 10:4847:11
Okay, that feels a little disjointed to me but I don't know if others have thoughts or on it.

[Erin Nettifee] 10:48:18
I think I would want to be able to see all the stuff just in one app.

[Magda Zacharska] 10:48:25
So you will see all the stuff in bug edit so you don't, You would see all the files in in bulk edit02
Thomas is asking about in the chat about whether the information that we're seeing here would be useful in the export manager. But I am guessing that when this is implemented, there will no longer be Bulk Edit data in Export Manager. It will just be all be encompassed in its own interface.

[Magda Zacharska] 10:4847:39
You don't need to go to export. manager we can hide whatever is in export.

[Erin Nettifee] 10:48:46
Manager. So it's. not confusing for anyone okay, I mean this is all kind of abstract, so you i'm not advocating for a particular decision.

[Thomas Trutt] 10:48:56
I don't think but yeah go ahead thomas sounded like you were about to say something

[Thomas Trutt] 10:49:05
Popping up in the export manager. Then other things popping up hidden here are duplicated over here.

[Thomas Trutt] 10:49:12
My big worry about this is I like the different statuses.

[Thomas Trutt] 10:49:18
I had, like all the information as being shown here. But are we solving the issue?31
So when we go to the Bulk Edit Operational Statuses there's one part that still happens in the Export Manager. When the records are being retrieved from the back end this is happening in the Export Manager, but the file that is being created is being moved to Bulk Edit shared  storage, so the file will not be accessible. You will be able to see that the records are being retrieved, but we will not have those files at all in Export Manager.

[Erin Nettifee] 10:48:11
Okay, that feels a little disjointed to me but I don't know if others have thoughts or on it.

[Erin Nettifee] 10:48:18
I think I would want to be able to see all the stuff just in one app.

[Magda Zacharska] 10:4948:24
Are we just moving it to another app? So they were. Yeah, we we first of all, we are not competing for resources in with another applications.[Magda Zacharska] 10:49:37
That was one of the issues we are facing. when25
So you would see all the files in bulk edit. You don't need to go to Export Manager. We can hide whatever is in Export Manager. So it's. not confusing for anyone.

[Erin Nettifee] 10:48:35
I mean this is all kind of abstract, so you I'm not advocating for a particular decision.

[Thomas Trutt] 10:4948:44
When you look at this, the this is the status. This is just a top of what the skipping on the on the flow here is what is happening right now before Bug Bulk added.56
My big worry about this is are we just moving the problem to another app?

[Magda Zacharska] 10:5049:01
All the updates, everything that the most resource, greedy part of bug edit 37
First of all, we are not competing for resources in with another applications.That was one of the issues we are facing. The most resource greedy part of Bulk Edit is actually saving the changes to the database and handling those changes occur accurately. [Thomas Trutt] 10:50:14
This was until and Nolana. This Until Nolana, this has been done by .[Magda Zacharska] 10:50:20
But export managerExport Manager, which is not the place where those things should happen. [Magda Zacharska] 10:50:28
This should be a bulk-added responsibilityBulk Edit responsibility. Does it answer your question?

[Thomas Trutt] 10:50:36
Does it answer your question a little bit? it it it's 43
A little bit. I still have a concern. [Thomas Trutt] 10:50:43
The only reason why? I said is Yes, it's it's It's getting more access to the bulk at it Bulk Edit back end process. [Thomas Trutt] 10:50:48
Because now it's his own app and it's able to do all this processing on its own my . My worries is that it's still going to be computing with the export manager competing with Export Manager and other processes, because it's [Thomas Trutt] 10:50:58
still going to be hitting the same Api API endpoints on the other internal apps. [Thomas Trutt] 10:51:03
So it might add, even compound it, because you now have 2 large apps hitting the same api's APIs at the same time.

[Thomas Trutt] 10:51:12
I guess I could see where this might make a little bit more sense, because at the X data export, the Data Export App was doing the updates moving that component out of it. [Magda Zacharska] 10:51:23
And that does make sense. Because , like, why would you have updating data in the actually export ?[Magda Zacharska] 10:51:28
App and I see your point that we are hitting the the same like in case of hi, Thomason, holding.

[Magda Zacharska] 10:51:42
So we are. We are hitting the same environment with different apis

[Magda Zacharska] 10:51:51
But this will happen if somebody also exports, or does the oapm H.

[Magda Zacharska] 10:51:58
Or make any other inventory changes. this is not what we can prevent.

[Magda Zacharska] 10:52:06
What we know, however, that if something goes wrong we can identify the module, and we can handle this within a module.

[Magda Zacharska] 10:52:16
One module. and this is a good point. We probably will be coming to this once.

[Magda Zacharska] 10:52:22
We have a a performance test for the in place for the new for the new design.

[Magda Zacharska] 10:52:31
I was told this will resolve our problem with the limit of 10,000 records.

[Magda Zacharska] 10:52:42
At this time we should be able to go up in the number of records we can update through the bulk operation, but this definitely will not be a superb blood for inventory or circulation modules performance.

[Magda Zacharska] 10:53:06
I would like we have some of the minutes left. There is one more mockup that I would like to show.

[Magda Zacharska] 10:53:17
Is those

[Magda Zacharska] 10:53:22
Hi Tim, that have expired because we will be putting the record.

[Magda Zacharska] 10:53:28
The the files, a large number of files in the external in the external storage.

[Magda Zacharska] 10:53:37
We will keep them for a month. So for that bug edits that happen in the previous month, you will be able to access them.

[Magda Zacharska] 10:53:47
But then we will remove them, and the files that were removed will be marked with this information, as you see on the screen, plus unavailable for download, because 30 days have elapsed since the job was run

[Magda Zacharska] 10:54:04
in export manager currently you are getting the Xml error that tells you that the exploration talking elapsed, which is not very user friendly.

[Magda Zacharska] 10:54:15
That's why we decided to go this route to to notify the user that defaults and no longer, and

[Erin Nettifee] 10:54:31
So the 30 days then is hard coded. we can make it.

[Magda Zacharska] 10:54:41
We can make it configurable through the api because if it turns out it's a preferred approach.

[Magda Zacharska] 10:54:52
We will start with hardcoded value of of 30 records 30 days.

[Magda Zacharska] 10:54:58
This was the the value that was at some point proposed for the files that are being generated by data export as well.

[Erin Nettifee] 10:55:06
Does anyone have any comments? Sure, I mean that that and 30 days makes as much sense as anything to me?

[Erin Nettifee] 10:55:12
It might be worth a question to Sisops or to, you know, ebsco hosting or index data, just to get a sense.

[Erin Nettifee] 10:55:21
This, too, what they think of that number and configuration and stuff like that.

[Erin Nettifee] 10:55:27
But here I mean the the stuff that we would have to retain would be things like financial records, and we're just not.

[Erin Nettifee] 10:55:34
That's not what's happening here. so I I think 30 days, I think would would be okay.

[Erin Nettifee] 10:55:40
But it I it's probably worth just asking around

[Magda Zacharska] 10:55:50
So I run this obviously by obscure hosting team.

[Magda Zacharska] 10:55:55
And I do also believe that that may depends on size of the institution.

[Erin Nettifee] 10:56:04
Sure, the large institution that have larger about edits, and they have a larger file size same way.

[Magda Zacharska] 10:56:13
Also want to cop on calls of storing those files because they will be eating much more space than for smaller institution when they have a smaller and smaller file size, they will not be eating that much of this storage space and

[Thomas Trutt] 10:56:35
30 days may be not even required for them. They could go longer without running out of space or incurring costs, for it sounds like this is almost a from what you even just described.

[Thomas Trutt] 10:56:56
Now as this might move around based on the institution, and that would make more sense to have. This is a tenant level setting, or have it as an api endpoint that could be hit.

[Magda Zacharska] 10:57:05
You could say delete all files after the State that could be set as a cron job or something.

[Magda Zacharska] 10:57:11
I, Rather let the user the tenant to specify their grace period and yeah, that would be.

[Thomas Trutt] 10:57:23
That would be my preference, as well but if that's not possible.

[Magda Zacharska] 10:57:27
The second one would be having something that a host hit an Api and say, Remove these files up after this date sounds good.

[Magda Zacharska] 10:57:36
I will bring it up to the to the development team

[Magda Zacharska] 10:57:45
So this is it what I had for today?

[Magda Zacharska] 10:57:49
Do you have any comments, questions.

[Erin Nettifee] 10:57:59
I know how many i've

[Magda Zacharska] 10:58:12
Thank you all, and i'll see you in in 2 weeks and We will start with the roadmap updates.

[Erin Nettifee] 10:58:20
Thank you. Hey, Erin, Can you save that file?

[Erin Nettifee] 10:58:25
Yes, I will save it for you. Thank you. have a good one.