Inventory (UXPROD-785)

[UXPROD-1685] Search by normalized ISBN number - in Inventory (thin thread, dedicated search option) Created: 15/May/19  Updated: 16/Sep/20  Resolved: 15/Apr/20

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Q2 2020
Parent: Inventory

Type: New Feature Priority: P3
Reporter: Charlotte Whitt Assignee: Charlotte Whitt
Resolution: Done Votes: 0
Labels: cap-mvp, chalmers, chalmers_debut_followup, inventory, metadatamanagement, po-mvp, q4-2019-at-risk, split
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Skärmavbild 2019-10-14 kl. 16.03.49.png    
Issue links:
Blocks
is blocked by UXPROD-1394 ISBN normalization: Refinement Closed
Defines
is defined by MODINVSTOR-413 add "isbn" normalised index Closed
is defined by MODINVSTOR-474 Add "invalidIsbn" normalized index Closed
is defined by RMB-499 Add "normalizeDigits" function Closed
is defined by RMB-575 add "sqlExpressionQuery" parameter Closed
is defined by UIIN-647 Normalize ISBNs for ISBN searching in... Closed
Relates
relates to MODINV-122 ISBN normalization API Closed
Epic Link: Inventory
Analysis Estimate: Very Small (VS) < 1day
Analysis Estimator: Charlotte Whitt
Front End Estimate: Small < 3 days
Front End Estimator: Niels Erik Nielsen
Front-End Confidence factor: Medium
Back End Estimate: Medium < 5 days
Back End Estimator: Jakub Skoczen
Estimation Notes and Assumptions: Outstanding questions:
* Implemented as a search index or as a persisted field containing the normalized version of the ISBN number?
Development Team: Prokopovych
PO Rank: 110
PO Ranking Note: CW: Aligned PO rank with Calculated Total rank
Rank: Chalmers (Impl Aut 2019): R1
Rank: Chicago (MVP Sum 2020): R1
Rank: Cornell (Full Sum 2021): R1
Rank: Duke (Full Sum 2021): R1
Rank: 5Colleges (Full Jul 2021): R1
Rank: GBV (MVP Sum 2020): R1
Rank: Lehigh (MVP Summer 2020): R1
Rank: MO State (MVP June 2020): R1
Rank: TAMU (MVP Jan 2021): R1
Rank: U of AL (MVP Oct 2020): R1

 Description   

Goal/Purpose: Library staff need to search Inventory by ISBN both with or without dashes and/or spaces and retrieve ISBN entered either with or without dashes/spaces.

Use cases: Staff member scans UPC code on back cover to enter ISBN (contains dashes) – this should retrieve the record whether the string is entered with or without dashes.

Staff member enters digits only to search--this should retrieve the record whether the string is entered with or without dashes.

FOLIO displays the ISBN number in the Instance record. The ISBN number is one of the main elements when ordering a resource, match of incoming records and more.

The ISBN element is in Inventory defined as a resource identifier number set by a resource identifier type labelled ISBN/Invalid ISBN in a reference table in the Settings > Inventory > Instance.

Documentation:
Slide deck: https://docs.google.com/presentation/d/1zsgykxOAKSdjikm8Lg56lamAv9u6o7ms1dTIHx1SDnI/edit#slide=id.p

Out of scope for MVP:
Work on relationship between 10/13 digit ISBNs - Implementation to search on calculated ISBN will be addressed post-MVP; while this has some not intended complications for Special collections. Requirements to be discussed further post-mvp..

ISBN utility code library: https://github.com/folio-org/folio-isbn-util

  • - - - - - - - - - - - - - - - - - - - - - - -

Some examples

Charlotte Whitt This is feedback from Chalmers. This is high priority for them. Please review and determine next step (finalize into a story, mark as will not implement, need more info, delete, etc.). The label chalmers_debut_followup helps us keep these issues all findable as a group.



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 17/Jul/19 ]

HI Charlotte Whitt I noticed this doesn't have a Q3 label. Do you think it will get handled in Q3 2019? Not sure how it ranks against the many other Inventory features. We're also working on it from the Orders side. Relevant UIOR story is linked.

Comment by Charlotte Whitt [ 17/Jul/19 ]

Hi Ann-Marie Breaux - that would be a question for Cate Boerema?

This is such an everyday (general) search, and something almost all staff are doing on a daily basis. So it definitely needs to be in if not before, then for Q4 2019.

Comment by Cate Boerema (Inactive) [ 17/Jul/19 ]

Well, Charlotte Whitt, you had previously put UXPROD-1420 at the top of your Q3 priorities, but I think once we break that down, we'll see it's not all really needed for mvp. We should then take a look at what other Inventory features might be able to be pulled in. The calculated rank on this is relatively low, but that's clearly because not all institutions have ranked it (all who have put it at go-live). Charlotte this might be one you bump up with your PO rankings and tag with po-mvp (if you think that's appropriate)

Comment by Ann-Marie Breaux (Inactive) [ 17/Jul/19 ]

Charlotte Whitt Cate Boerema FWIW - this is definitely a Chalmers pain point. They want to be able to scan a UPC on a book and use the ISBN of the UPC to execute a search in Orders and/or Inventory. There's a separate story for Orders ( UIOR-274 Closed ). Whatever solution we come up with in Orders will likely be extensible to Inventory.

Comment by Charlotte Whitt [ 17/Jul/19 ]

I have already assigned `PO-mvp`
This has nothing to do with UXPROD-1420

Comment by Marie Widigson [ 16/Sep/19 ]

Still very high priority for us at Chalmers!

Comment by Cate Boerema (Inactive) [ 19/Sep/19 ]

Charlotte Whitt and Ann-Marie Breaux this is cap-mvp, thus Q4 work for Core Functional. Will it be dev-ready soon?

Comment by Ann-Marie Breaux (Inactive) [ 19/Sep/19 ]

Cate Boerema we're getting all our ducks in a row this week over in Orders, and then I think we will be able to copy that work for Inventory, so yes, I think this can be dev ready next week

cc: Charlotte Whitt

Comment by Ann-Marie Breaux (Inactive) [ 30/Sep/19 ]

Charlotte Whitt and Cate Boerema Here's the current ISBN situation in Inventory, and what we're doing in Orders to make this work. If it woudl be helpful to talk through any of this or look at it together in FOLIO, please let me know.

Inventory situation:

  1. The MARC-related ISBNs in Inventory have been mapped (by Folijet) to be valid (identifier type = ISBN) if in 020 $a of the MARC record, or invalid (identifier type = Invalid ISBN) if in 020 $z. As far as I know, there's no validation being done on the actual calculation of "valid" ISBNs in Inventory to ensure they actually calculate properly.
  2. When a user manually enters an ISBN into an Instance (not controlled by an underlying MARC), there's no validation that the ISBN calculates properly.
  3. There's usually no calculation/validation check on MARC 020 $a. If Inventory begins validating "valid" ISBNs (ISBNs with Identifier type = ISBN) to ensure they calculate properly, it will find some (maybe many?) that do not calculate properly, either having come from the underlying MARC or input manually. If Inventory imposes a validation check, there will need to be a way to deal with ones that do not calculate properly. That can be fairly easy for manually entered ISBNs, but not so easy for ISBNs coming from underlying MARC, since it means the underlying MARC will need to be updated.
  4. Further, many "valid" ISBNs have a qualifier after the number itself (e.g. hardcover, ebook, vol. 1, etc). That qualifier needs to be ignored when doing any validation check, but needs to be retained to help understand the differences between multiple ISBNs in the same record
  5. And finally, ISBNs may be 10 digit or 13 digit, but there's nothing requiring all 10-digit versions to be changed to their equivalent 13 digit version in Inventory.

Order situation

  1. We need clean ISBNs to transact in orders, preferably 13-digit, and retaining the qualifier so that the vendor understands what version of a book is being ordered. We also don't want the clutter of invalid ISBNs on POLs, specifically eBook ISBNs on print-book orders and vice versa. To that end, we have done the following (some stories still in process, but most done):
  2. For users manually entering POL data, require them to enter the ISBN as a valid 13 digit ISBN before the POL can be saved
  3. For ISBNs coming over from Inventory, only bring over ones with identifier type = ISBN, not identifier type = Invalid ISBN
  4. For ISBN-10s coming over from Inventory, translate them to their 13 digit equivalents; then delete that newly-translated ISBN if it duplicates an existing ISBN-13 already in the POL. That way we end up with one ISBN-13 for each different ISBN that came from the Inventory instance.
  5. If the ISBN that came from Inventory is not valid (does not calculate properly), then the POL cannot be saved until the ISBN is adjusted or removed. Any changes in the ISBN fields do NOT flow back to Inventory
  6. Any qualifier for the ISBN goes in a separate field from the number itself. That way we can calculate the number cleanly, but still have the qualifier for identification purposes, and for output in the order to the vendor
  7. All of that allows us to end up with clean ISBNs in the POL. Now, for searching:
  8. If someone searches in the POL/Keyword Search, they can only search by the 13 digit, valid ISBN, with no qualifiers
  9. But if someone searches in the POL/Product ID/ISBN search, they can search by ISBN-10, ISBN-13, with or without hyphens, with or without qualifiers. FOLIO will normalize their input to the ISBN-13 without hyphens or qualifiers and bring up the appropriate search results.

So that's how we're handling it in Orders, and we'll probably implement similar handling in other acq apps that have ISBN searches. It'll likely be easier since most of the other apps' data flows from the POL, so all of the work we've done to clean up the ISBNs in the POL will help with the downstream searches.

cc: ing Craig McNally and Dennis Bridges in case they have anything to add.

Comment by Marc Johnson [ 07/Oct/19 ]

For users manually entering POL data, require them to enter the ISBN as a valid 13 digit ISBN before the POL can be saved

Where is this constraint enforced, in the reference UI, mod-orders and/or mod-orders-storage?

But if someone searches in the POL/Product ID/ISBN search, they can search by ISBN-10, ISBN-13, with or without hyphens, with or without qualifiers. FOLIO will normalize their input to the ISBN-13 without hyphens or qualifiers and bring up the appropriate search results.

Does this mean that it is only possible to search for an entire ISBN-13 in acquisitions?

How does the orders UI know that what has been entered into the POL/Product ID/ISBN search is intended to be a ISBN number and not a product ID?

Comment by Marc Johnson [ 07/Oct/19 ]

When a user manually enters an ISBN into an Instance (not controlled by an underlying MARC), there's no validation that the ISBN calculates properly.

Should the reference inventory UI or mod-inventory only allow valid ISBN's to be provided for an identifier with the type ISBN (rather than Invalid ISBN)?

Further, many "valid" ISBNs have a qualifier after the number itself (e.g. hardcover, ebook, vol. 1, etc). That qualifier needs to be ignored when doing any validation check, but needs to be retained to help understand the differences between multiple ISBNs in the same record

Does inventory have a separate field for the qualification information?

If not, how are we intending for the qualifier to be ignored?

(I find the qualifier examples a little odd, as I think of ebooks and hardcover books etc as being separate instances)

Comment by Theodor Tolstoy (One-Group.se) [ 08/Oct/19 ]

[Comment removed]

Comment by Charlotte Whitt [ 14/Oct/19 ]

Notes from meeting with Ann-Marie Breaux Craig McNally lew235 Marc Johnson Charlotte Whitt - 10/14/2019
Inventory:
ISBN search option:

  • search on valid ISBN (020 $a)
  • search on invalid ISBN (020 $z)
  • search on 10 digit (or 9 digit and an x or X) - with the possibility to do truncation ( * )
  • search on 13 digit - with the possibility to do truncation ( * )
  • search with hyphens
  • search without hyphens

Example from Chalmers bugfest environement:

Comment by Ann-Marie Breaux (Inactive) [ 14/Oct/19 ]

Hi Charlotte Whitt One small correction on this line:

  • search on 13 digit (or 12 digit and an x or X) - with the possibility to do truncation ( * )

13 digit ISBNs will never have an X check digit. When the ISBN expansion happened, they also revised the check algorithm, so that X would no longer be assigned, only digits. The only place you'll see an X is on the end of a 10-digit ISBN.

Also, in case it's useful, here's a tool from Bowker (the US ISBN agency) that allows for quick translation back and forth between 10s and 13s: https://www.isbn.org/ISBN_converter

Comment by Ann-Marie Breaux (Inactive) [ 15/Oct/19 ]

Theodor Tolstoy (One-Group.se)'s comments from last week, which I asked him to remove until after we got over some of the other fires:

IMHO, In order for FOLIO to work long-term, you need to be able to trust Inventory identifiers to be exactly that and not strings formatted in a way that is imposed by MARC21. A valid ISBN should be an ISBN that other modules can trust right out of the box (the box being inventory).

Otherwise, every single module that wants to interact with Inventory needs to setup a similar logic and validation for their identifiers as mod-orders. And what if the source of the data has another logic when populating Inventory ? Will not having MARC-import formatted identifiers break the contract? Perhaps another importer puts the qualifier in front of the ISBN instead? Then every module need to be rewritten to accommodate that.

If the ISBN does not validate, it could be put in as an Invalid ISBN and search would not be broken.

Perhaps, what we really should do is to introduce the GTIN Identifier type? The GTIN should always be the 13 digit version of the ISBN, validated and trustworthy.

If there is a strong drive for having the qualifiers together with the identifiers, perhaps they could be a added as a separate note field so the identifier?

Not the best place of opening this discussion. But i feel this Jira touches on a lot of things that should be a bit more clear.

Comment by Ann-Marie Breaux (Inactive) [ 15/Oct/19 ]

Here's a link to examples of many product identifiers, including various configurations of ISBNs: https://drive.google.com/open?id=13KA5SXmev5qApIofCUyKvDRt4reAqNFI

Comment by Charlotte Whitt [ 22/Oct/19 ]

Based on the feed back we have gotten from RA, MM, and RM SIG last week - https://docs.google.com/presentation/d/1zsgykxOAKSdjikm8Lg56lamAv9u6o7ms1dTIHx1SDnI/edit#slide=id.p - we'll split this feature into two tracks:
1) search by normalized ISBN with/without hyphens, incl. qualifier data - MVP
2) search by transformed ISBN - where the ISBN has been calculated from 10 to 13 digits, and maybe also vise versa (TBD) - there was some push back from e.g. the Special Collection SMEs.

Comment by Charlotte Whitt [ 12/Dec/19 ]

Hi Jakub Skoczen - will you confirm/update the estimates of the remaining work on making search on normalized ISBN numbers possible. Thanks much
Getting this in is critical for the cap-mvp planning.

CC: Adam Dickmeiss Julian Ladisch

Comment by Jakub Skoczen [ 12/Dec/19 ]

Charlotte Whitt I've updated the estimate to represent remaining work – MODINVSTOR-413 Closed and RMB-499 Closed (which should be completed this sprint). Note that the front-end part (if any) cannot be executed by the Platform team.

Comment by Charlotte Whitt [ 12/Dec/19 ]

Awesome, thank you Jakub Skoczen

Generated at Fri Feb 09 00:17:37 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.