Inventory
(UXPROD-785)
|
|
| Status: | Draft |
| Project: | UX Product |
| Components: | None |
| Affects versions: | None |
| Fix versions: | TBD | Parent: | Inventory |
| Type: | New Feature | Priority: | P2 |
| Reporter: | Debra Howell | Assignee: | Ryan Taylor |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | Support, bug-fest, changed-expectations, delegate_candidate, industry-standard, loc | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||
| Release: | Not Scheduled | ||||||||||||||||||||||||
| Epic Link: | Inventory | ||||||||||||||||||||||||
| Front End Estimate: | XL < 15 days | ||||||||||||||||||||||||
| Front End Estimator: | Khalilah Gambrell | ||||||||||||||||||||||||
| Front-End Confidence factor: | 20% | ||||||||||||||||||||||||
| Back End Estimate: | XL < 15 days | ||||||||||||||||||||||||
| Back End Estimator: | Khalilah Gambrell | ||||||||||||||||||||||||
| Back-End Confidence factor: | 20% | ||||||||||||||||||||||||
| Development Team: | Folijet | ||||||||||||||||||||||||
| PO Rank: | 80 | ||||||||||||||||||||||||
| Rank: Cornell (Full Sum 2021): | R2 | ||||||||||||||||||||||||
| Description |
|
Overview: "space" should be considered as a category, not a literal character, e.g., leading and trailing carriage returns & tabs should also be ignored Laura Daniels can provide additional details Steps to Reproduce:
Expected Results: Developer Notes Before this can be worked on, we need more information. Which record types should this apply to? Is it every record type in inventory? |
| Comments |
| Comment by egerman [ 20/Aug/21 ] |
|
From TAMUs testing:
|
| Comment by Zak Burke [ 31/Aug/21 ] |
|
I think you could make a strong case for trimming leading and trailing whitespace from all search queries and probably from all fields when creating/editing entries, but those are UI/front-end stories, not MODINV/back-end stories. Charlotte Whitt, egerman, Debra Howell, do you think SIGs would want to weigh on that kind of a change, or should the UI teams follow Nike and Just Do It? Do you think SIGs may have different expectation for different apps, e.g. whitespace should be left as-is in users but trimmed in inventory, or do you think trimming could (should) be consistently implemented across the board? |
| Comment by Charlotte Whitt [ 01/Sep/21 ] |
|
Hi Zak Burke, I am leaning towards, that is an industry standard requirement, and something all current systems do already. Otherwise it could be discussed in the Cross App SIG, Martina.Schildt? Alternatively in each of the domain specific SIG, e.g. MM-SIG lew235, Acquisition SIG susan.martin@mtsu.edu, Resource Access SIG Jana Freytag. |
| Comment by Debra Howell [ 01/Sep/21 ] |
|
Zak Burke I am in agreement with Charlotte. |
| Comment by egerman [ 01/Sep/21 ] |
|
Also agree with Charlotte. |
| Comment by lew235 [ 01/Sep/21 ] |
|
Agree with Charlotte. |
| Comment by Martina.Schildt [ 16/Sep/21 ] |
|
Agree with Charlotte as well |
| Comment by Holly Mistlebauer [ 24/Sep/21 ] |
|
Debra Howell: The freeze date for Kiwi was September 17, but given this is a P2 bug we will try to get it in. Thanks... |
| Comment by Holly Mistlebauer [ 24/Sep/21 ] |
|
P.S. to Debra Howell: The team was set to "Prokopovych" just 4 days ago. That is why we didn't work on it before the freeze date. |
| Comment by Kyle Felker [ 25/Oct/21 ] |
|
Debra Howell Holly Mistlebauer This issue needs more specific parameters before it can be worked on. We need to know what fields in what records you want leading and trailing whitespace cut out of. Do you want this done ONLY to call number fields in the holdings record? Call number fields anywhere in any record? Other fields elsewhere? All fields everywhere (that's gonna be a tall order, and will take some time)? You also mention double spaces, and it isn't clear (to me at least) wether you mean double spaces within the text or only at the ends. If you mean at the ends, then thats not a separate concern-any stripping process we employ would remove multiple whitespace characters from the end. |
| Comment by Debra Howell [ 25/Oct/21 ] |
|
Kyle Felker please see Charlotte Whitt's note above. She outlines it. Zak Burke would probably have some thoughts, as well. |
| Comment by Marc Johnson [ 26/Oct/21 ] |
Which note are you referring to? The only comment from Charlotte I read above is in reference to an industry standard, is it that one? If so, I don't think that answers Kyle Felker question about which fields are affected by this, unless the implication is all text fields? |
| Comment by Marc Johnson [ 28/Oct/21 ] |
|
As requested during our sprint planning, I have added some questions to the description (rather than a comment) and moved the issue to draft until we can be more specific about what is involved in this work. cc: Oleksiy_Lemeshko |
| Comment by Anya [ 01/Nov/21 ] |
|
Support: Zak Burke will follow up with Marc Johnson and Charlotte Whitt. |
| Comment by Ann-Marie Breaux (Inactive) [ 04/Nov/21 ] |
|
Just adding a comment and a linked bug. This appeared in Kiwi Bugfest in relation to the reference data for Inventory Instance_type (called Resource Type in Inventory Settings). Folijet made a change to the default MARC-Instance map so that the proper Instance Type can be derived from the 336$a or the 335$b (previously was only 336$b). When I checked it on the hosted ref envs, all was fine. When I checked it on Kiwi Bugfest, there was a problem with 336$a spoken word, but not 336$b spw. Given the revised mapping, both should have resulted in the same value being assigned to the records. When the developer reviewed, they discovered that several of the reference values in Kiwi Bugfest had trailing spaces, which caused the lookup by 336$a to fail. We adjusted the reference values in Kiwi Bugfest, which resolved the discrepancy. I added this as an example of a situation where having a check for trailing spaces on the BE would be helpful (since these were not user-created reference values). I would also advocate for a check/removal on the UI for user-created values. Before implemented, it would be important to evaluate each field in each record type, to figure out if there's ever a valid use case for leading spaces, trailing spaces, or double spaces. For example, Indicator 1 and Indicator 2 in MARC records are often spaces (blanks). We wouldn't want the system to try to resolve that from 2 spaces to 1. And as it happens, I talked to Zak Burke and Marc Johnson to think through the Kiwi Bugfest problem and resolution, so thank you to Marc for alerting me to this Jira! |
| Comment by Anya [ 08/Nov/21 ] |
|
Support: Zak Burke to follow up |
| Comment by Anya [ 22/Nov/21 ] |
|
Support : Zak Burke to follow up with Charlotte Whitt and Marc Johnson |
| Comment by Anya [ 06/Dec/21 ] |
|
Support: Charlotte Whitt will set up a meeting |
| Comment by Anya [ 20/Dec/21 ] |
|
Support : Charlotte Whitt how did the meeting go? - What release should this issue be assigned to |
| Comment by Holly Mistlebauer [ 21/Dec/21 ] |
|
Charlotte Whitt: How do we get this ticket out of DRAFT? Thanks... |
| Comment by Charlotte Whitt [ 03/Jan/22 ] |
|
Holly Mistlebauer - Zak Burke Marc Johnson lew235 Debra Howell Felix Hemme and I need to review this ticket. I have send out a doodle poll |
| Comment by Zak Burke [ 06/Jan/22 ] |
|
In a JS regular expression, whitespace is considered to be the following: space \f form feed \n new line \r carriage return \t tab \v vertical tab \u00a0 non-breaking space \u1680 obscure early-Irish script space block; srsly I am not making this up \u2000-\u200a various unicode character separators \u2028 line separator \u2029 paragraph separator \u202f narrow non-breaking space \u205f mathematical white space \u3000 ideographic white space, used in CJK \ufeff zero-width non-breaking space Stripping all these characters from the beginning and end of a string would be easy. Coalescing characters within a string is ... messier. I think a simple action that is likely to have good results would be to coalesce only multiple spaces into a single space and leave everything else alone. Otherwise, we have to decide what to do with situations like "tab-tab" (reduce to a single tab or a single space?) or "tab-space-tab" where it's even less obvious what should happen whereas "multiple spaces become one space" is a simple, easy to understand rule. |
| Comment by lew235 [ 07/Jan/22 ] |
|
Zak Burke that sounds completely reasonable to me (and conveniently addresses the one situation where I know this is occurring for us at Cornell) |
| Comment by Erin Nettifee [ 21/Feb/22 ] |
|
Hi all. This has come up in the context of https://folio-org.atlassian.net/browse/CIRC-1293 which is a bug in Vega's queue. There had been work done previously to allow barcodes with leading and trailing white space to be able to be checked out, but that appears to have regressed. Their developer's initial assessment is that the best approach would be to trim the leading and trailing whitespace off the barcode when it's created. (And I cannot tell from any of the related initial tickets how the issue was initially fixed.) In talking to Charlotte Whitt I can see the work on this ticket, and I'm wondering whether a meeting happened in January and if there was a decision made about the approach that would be done when this can be developed, and the list of fields that would be involved. |
| Comment by Marc Johnson [ 01/Mar/22 ] |
There was a meeting about this subject. My memory of the decisions is a little hazy, I think the outcome was:
I believe the reasoning behind this was that folks weren't fully sure that every organisation wants to remove the leading / trailing whitespace. And if that turned out to be the case, it would be easier to change than if it were done in the back end. I believe Zak Burke had an action to investigate what the UI's definition of whitespace was. Charlotte Whitt Felix Hemme Zak Burke lew235 How wrong is my recollection of this meeting? |
| Comment by Marc Johnson [ 01/Mar/22 ] |
|
Oh, and I forgot, I think folks were going to decide later which fields would be affected. |
| Comment by lew235 [ 01/Mar/22 ] |
|
Your recollection matches mine, Marc Johnson Except maybe I imagined we decided all fields would be affected? |
| Comment by Marc Johnson [ 01/Mar/22 ] |
Thanks. I couldn't remember whether a decision was made on that. |
| Comment by Erin Nettifee [ 01/Mar/22 ] |
|
Hi all - Sherzod Nurjonov with Vega has been looking at how to address checkouts of items with spaces in barcode - what appears to be a regression - https://folio-org.atlassian.net/browse/CIRC-1293 This problem was identified as part of Missouri State's adoption, and addressed in the Checkout app as part of https://folio-org.atlassian.net/browse/CIRC-284 and https://folio-org.atlassian.net/browse/UICHKOUT-506, but appears to have regressed. It doesn't sound like this is a firm decision? Or is it? Is there any sense as to when this work might occur? Vega's initial thought was that it made sense to strip the whitespace when the item was saved, but it sounds like Metadata management can identify cases where that wouldn't work for a particular library. |
| Comment by Charlotte Whitt [ 02/Mar/22 ] |
|
Marc Johnson Erin Nettifee - back in last August @zak proposed:
And all SMEs agreed to do this - as this is the Industry standard, and all old ILS systems already have similar functionality. Following in January 2022, here Zak Burke wrote up:
So not sure if we need further investigation from Zak Burke - or we are good to go to solve this as a general solution. And then the solution would also fix the regression bug ticket raised by Erin Nettifee
If we are good to go for a general solution, would that then be work solved by the Stripes Force team - or do we need each team to do this for each of their apps? |
| Comment by Marc Johnson [ 02/Mar/22 ] |
|
Charlotte Whitt Zak Burke Erin Nettifee
Given that the agreed solution (to do this only in the UI) deliberately allows organisations to still define fields with leading and trailing whitespace, I doubt that it completely resolves
cc: lew235 |
| Comment by lew235 [ 02/Mar/22 ] |
|
MM can only imagine there might be cases where a library would choose to allow leading/trailing white spaces to be saved. I'm not sure I understand how this solution would not resolve
|
| Comment by Marc Johnson [ 03/Mar/22 ] |
I think that depends upon how the folks who have item barcodes with whitespace are going to maintain them.
I took from Erin Nettifee comments that Missouri State have barcodes that contain whitespace
I am assuming that changing their barcodes is not an option and hence FOLIO needs to support barcodes with whitespace (otherwise
If the whitespace is within the barcode, then these two issues are not related at all (because this issue only covers removing leading / trailing whitespace). If the whitespace is leading / trailing then the solution presented here means that Missouri State cannot use the reference inventory UI as any edit to an item could corrupt the barcode. I'm basing this on a bunch of inferences and assumptions. lew235 Erin Nettifee maybe you can help me understand what I'm getting wrong or misunderstanding. |
| Comment by Marc Johnson [ 03/Mar/22 ] |
|
Having read some of the comments on
Is that an acceptable decision for Missouri State (to change all / some of their barcodes during adoption of FOLIO)?
|
| Comment by Erin Nettifee [ 03/Mar/22 ] |
|
Missouri State ended up changing their barcodes after the fix was put in, but there are other adopters that have this scenario. I know Duke (my institution) is one of them, and we have at least several thousand records with spaces at the end, though we're still trying to get an accurate count, so rebarcoding is not really an option. We no longer get items with barcodes like this, but there were several years where they were possible. The UI concern is definitely there, though I'm more concerned about the SIP2 and NCIP aspects - I think we'd be unlikely to edit a barcode unless we were replacing it with another one. |
| Comment by Marc Johnson [ 03/Mar/22 ] |
Thanks, that is useful context. Does that mean that FOLIO is going to have to support barcodes with trailing spaces for Duke to adopt FOLIO? |
| Comment by lew235 [ 03/Mar/22 ] |
|
If leading and trailing spaces are stripped both when data are saved and from searches, would that not allow for the barcodes with trailing spaces to exist? |
| Comment by Marc Johnson [ 03/Mar/22 ] |
The stripping when saving would mean that no barcodes with leading or trailing spaces can exist. If an organisation needs barcodes that have meaningful leading trailing spaces, the current solution would mean their data is corrupted if the reference UI was used. |
| Comment by Erin Nettifee [ 04/Mar/22 ] |
|
Right - cause even if you were editing an item with leading or trailing spaces and you weren't editing the barcode field, saving the record would erase the spaces from the underlying item. |
| Comment by Erin Nettifee [ 04/Mar/22 ] |
I don't know :-| - I need to follow up on some questions on this side to try to figure out how many we are talking about. |
| Comment by Ann-Marie Breaux (Inactive) [ 07/Mar/22 ] |
|
Hi Charlotte Whitt and Marc Johnson I'm a little unclear on the plans for this. This feature only covers Inventory record types. Is that correct? Is this being considered as a Morning Glory feature? The main thing I want to understand - if this is picked up by Prokopovych for Morning Glory, and automated stripping of spaces, plus reindexing happens, with that require any coordination with Source Record Storage? (if the Inventory record's source = MARC). If not, then no worries. If so, then I would like to understand what may be required ASAP. Thank you! |
| Comment by Marc Johnson [ 08/Mar/22 ] |
Yes, this feature was only intended to be for instances, holdings and items.
I'm not able to answer that, that's for Charlotte Whitt
I don't think we have an answer to that. There won't be a mass-scale automated stripping of whitespace or re-indexing because the decision made for this work is to do the stripping in the UI only when a record is created or edited. Thus, once this is implemented, there will be a mixture of records where the whitespace has been stripped and some that haven't. This may well affect how data import does it's matching (depending upon how that works).
That is a really good question. Charlotte Whitt lew235 For records where only some fields are editable in the UI, when those are saved, would you expect only the editable fields to have been stripped or all text fields (as currently decided)? cc: Zak Burke |
| Comment by lew235 [ 08/Mar/22 ] |
|
Definitely only the editable fields. We should not strip anything from source data or the data created from source records. |
| Comment by Debra Howell [ 11/Nov/22 ] |
|
Charlotte Whitt Can I get an update on this feature please? It's still causing us quite a bit of frustration, and it seems to be languishing in limbo. Thanks! cc: Anya lew235 |
| Comment by Charlotte Whitt [ 15/Nov/22 ] |
|
@Khalilah Zak Burke this is not just Inventory, but a more general issue. Can this be solved in the central Smart stripes component. Or do we need a UI developer to trawl through every entry data element one by one, to fix this? |
| Comment by Zak Burke [ 15/Nov/22 ] |
|
Charlotte Whitt, it's likely we can solve this centrally for components based on stripes-connect for insert/POST and update/PUT requests since both work the same way, and we could probably inject a pre-submit filter just before the objects get serialized. IIRC react-query and/or OkapiKy have similar hooks. That's all good news! However, I know there are also > 50 instances of one-off fetch calls that don't leverage stripes-connect or okapi-ky (from stripes-core) that will need to be handled on a case-by-case basis. |
| Comment by Erin Nettifee [ 17/Nov/22 ] |
|
Thomas Trutt asked if I could revisit the Duke analysis on this issue to see what behavior was happening with regards to my comments from this spring. On a Lotus environment, testing an item with a scanned barcode "D00687485 ", where the item record in Inventory was actually loaded with the whitespace stripped by Jeff, our developer: 2) Check out succeeds, the UI appears to strip the whitespace after you hit enter, this is the JSON payload that it sends to checkout-by-barcode: {"itemBarcode":"D00687485","userBarcode":"6033006990017076","servicePointId":"aebfafc3-0b07-445f-939d-1b47f75dd83c","loanDate":"2022-11-17T15:53:10.606Z","id":"dfe09d73-b2a3-49b6-ac24-b4cd24f59fc1"}3) Inventory search fails because the scan includes the whitespace + an enter key, and the underlying inventory record does not. If I modify the underlying inventory record to add the space in, the search succeeds. 4) Course lookup fails because whitespace is included, this is the JSON that it sends, presumably the failure is actually when the code tries to look up the item, and the barcode here has a space and the barcode in Inventory does not. ,"id":"949829f1-c6e1-4b46-bb42-083a58eb0d3b"} On Bugfest Nolana, since I don't have a Duke environment with NCIP installed: I was unable to figure out how to test SIP2. So in terms of what I was seeing with Duke in Fall 2021 that resulted in the circ ticket that I filed, either something changed between now and then that resolved the issue, or I was missing something in Fall 2021 and this actually was not a regression. I have resolved
Regardless of all of this, Duke still has many thousands of records with spaces in the scanned barcode from several years where for some reason space was allowed as a printed barcode check digit. This is no longer the case for barcodes, but it is unfeasible right now for us to rebarcode the books with the offending spaces. We will either have to reprogram all of our barcodes to strip the whitespace (which is likely something we can do, but requires research), or we will need to load our records with whitespace that actually matches what the scanned barcode has. |
| Comment by Marc Johnson [ 18/Nov/22 ] |
|
I'm trying to understand the implications of your last comment.
Based upon later comments, I get the sense that the barcodes were intended to be loaded with a trailing space and this was done in error. And that led the system to work when entering the barcode for searching without the space. Have I understood that reasonably?
I think this means that we need to allow a trailing space in barcodes for the system to work for Duke (assuming the rebarcoding / reprogramming isn't done). How wrong is my interpretation of this? |
| Comment by Charlotte Whitt [ 18/Nov/22 ] |
|
Erin Nettifee - would you mind do the same test also in Bugfest Nolana environment. |
| Comment by Erin Nettifee [ 18/Nov/22 ] |
You know I'm actually not sure that we didn't intend to strip the trailing space when we imported the barcode values, I haven't had those conversations with people here.
I'm fairly confident we could do the reprogramming, but if it didn't, either we would need that search to work or we would need some more meaningful error message than simply that the barcode wasn't found. All else being equal, I think the fact that this was allowed for a few years is viewed as an aberration/mistake, so not something we would envision happening in the future - which to me implies that we would be open to different ways to make this functional for us, whether that meant it worked with the trailing space or that we compensated for the data in other ways. |
| Comment by Marc Johnson [ 21/Nov/22 ] |
|
Thank you for the follow up.
What kind of different error message would be wanted? I'm asking because I'm unsure how we would differentiate once all of the spaces have been stripped from the data. |
| Comment by Autumn Faulkner [ 09/Aug/23 ] |
|
We are having spine labeling and scanning issues in Inventory due to leading spaces. These spaces get introduced sometimes by our label maker when we're interacting with some irregular barcodes we use at MSU. (It basically wants to add spaces for barcodes that don't have the full amount of expected characters.) While we are working on a solution to the label maker issue (which of course turns out to be more nasty than expected), this was not a problem in Sierra because Sierra ignores/normalizes spaces altogether. Also, outside of Inventory, we find the spaces are not an issue. So we assume other Folio modules have this capability. Hoping this can be replicated in Inventory as well. |
| Comment by Thomas Trutt [ 09/Aug/23 ] |
|
Khalilah Gambrell and Christine Schultz-Richert I was hoping that we could get this feature re-evaluated and hopefully prioritized for development. The one prior block for this issue was barcode data that Duke had, in that their barcodes legitimately contained trailing spaces. Even though that was a fringe case it would have greatly impacted them and created a showstopper. Since Duke will not be moving to FOLIO at this time, we were hoping this feature could be looked at again and implemented. |
| Comment by Raegan Wiechert [ 14/Sep/23 ] |
|
Not sure if this matters or not, but https://folio-org.atlassian.net/wiki/display/MM/2023-09-14+Metadata+Management+Meeting+notes could be added under the "Mentioned in" section above. |