Custom Fields (UXPROD-396)

[UXPROD-2211] Custom Fields in Inventory Created: 18/Dec/19  Updated: 28/Dec/23

Status: Draft
Project: UX Product
Components: None
Affects versions: None
Fix versions: TBD
Parent: Custom Fields

Type: New Feature Priority: P3
Reporter: Charlotte Whitt Assignee: Ryan Taylor
Resolution: Unresolved Votes: 0
Labels: delegate_candidate, loc, round_iv
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File screenshot-1.png     PNG File screenshot-2.png    
Issue links:
Blocks
blocks UXPROD-2848 Search by Custom Fields (Instance, Ho... Draft
blocks UXPROD-2248 Tokens - Add custom fields to patron ... Draft
Cloners
is cloned by UXPROD-3580 Customization of Reference values in ... Draft
Defines
defines UXPROD-396 Custom Fields In Progress
Relates
relates to UXPROD-2178 Inventory. Item record. Make item mat... Draft
relates to UXPROD-2212 Settings page. Inventory. Each instit... Open
relates to UXPROD-33 Custom Fields (for User Record and as... Closed
relates to UXPROD-2143 Q1 2020/Q2 2020 | Custom Fields (for ... Closed
Release: Not Scheduled
Epic Link: Custom Fields
Front End Estimate: XL < 15 days
Front End Estimator: Khalilah Gambrell
Front-End Confidence factor: 20%
Back End Estimate: XL < 15 days
Back End Estimator: Khalilah Gambrell
Back-End Confidence factor: 20%
Development Team: Folijet
Kiwi Planning Points (DO NOT CHANGE): 33
PO Rank: 81
Rank: Chicago (MVP Sum 2020): R1
Rank: Cornell (Full Sum 2021): R2
Rank: Duke (Full Sum 2021): R1
Rank: 5Colleges (Full Jul 2021): R2
Rank: GBV (MVP Sum 2020): R2
Rank: hbz (TBD): R2
Rank: MO State (MVP June 2020): R2
Rank: TAMU (MVP Jan 2021): R1
Rank: U of AL (MVP Oct 2020): R2
Score: 30.5

 Description   

Ability to add custom fields to records in Inventory (Instance, Holdings, Item). Custom fields has been developed to the Users app. This is highly sought after by the SMEs and early implementer libraries.

Inventory requires custom fields to support

  • Data Migration: Given each ILS provider has unique fields it is critical we support custom fields to ensure that all valuable data is migrated to Folio.
  • Affiliation/Statistical groupings
  • Workflow
  • Interaction with locally used 3rd party systems

Slidedeck:



 Comments   
Comment by Holly Mistlebauer [ 17/Jun/20 ]

Chicago comment from Round IV Outliers spreadsheet: Inventory instance notes are not available for editing if there is an underlying MARC record. We are trying to get away from managing data in MARC 9XX fields.Currently we have good flexibility in holdings and items.We de-prioritized the tags and notes apps because we thought we could add custom fields at the instance level, but if there is underlying MARC we are locked into maintaining that data in the MARC record, not in the instance or in even the inventory app.This means instances with underlying MARC will need to be treated differently than instances without underlying MARC, which greatly complicates workflows and procedures. -Tod Olson

Comment by jackie.gottlieb@duke.edu (Inactive) [ 15/Sep/20 ]

Reason Duke has determined this is a go-live/R1 for us is because we need the 'previous/former' call number (call number 2) to be searchable in inventory and a custom field may be the only feasible way to support this requirement.

Comment by Marc Johnson [ 18/Sep/20 ]

Charlotte Whitt jackie.gottlieb@duke.edu Cate Boerema

Reason Duke has determined this is a go-live/R1 for us is because we need the 'previous/former' call number (call number 2) to be searchable in inventory and a custom field may be the only feasible way to support this requirement.

Is previous/former call number intended to be a custom field? And does that mean that folks are expecting custom fields to be searchable?

Comment by Ann-Marie Breaux (Inactive) [ 18/Sep/20 ]

Maybe a stupid question from me - in lieu of a custom field, why couldn't there be a call number type for previous call number, and the previous call number goes there? I realize that call number/call number type is not repeatable on the holdings or item, but it is repeatable (classification) on the instance.

Comment by jackie.gottlieb@duke.edu (Inactive) [ 18/Sep/20 ]

Marc Johnson
"Is previous/former call number intended to be a custom field? And does that mean that folks are expecting custom fields to be searchable?"
We are not expecting FOLIO to have additional call number fields and so have assumed we will need to support this with a custom field. Yes, we expect custom fields to be searchable.

Ann-Marie Breaux, I am not clear how a previous call number can be supported by using a call number type. We do need the previous call number to be on the item record. (further detail: we do anticipate storing our previous call number in Marc 583 and using that to have is appear in our discovery layer.) Jessica Janecki, can you envision how this might work?

Comment by Jessica Janecki [ 18/Sep/20 ]

Ann-Marie Breaux we need this information at the item level.
Marc Johnson we need custom fields to be searchable

Comment by Marc Johnson [ 21/Sep/20 ]

Jessica Janecki jackie.gottlieb@duke.edu

we need this information at the item level.

Given that inventory searches for instances, would this be something like find the instances where an item exists with a previous call number of ABC?

We are not expecting FOLIO to have additional call number fields and so have assumed we will need to support this with a custom field. Yes, we expect custom fields to be searchable.

we need custom fields to be searchable

Khalilah Gambrell I believe you were involved in the original custom fields work. Was searching custom fields something that was discussed / designed in the scope of that work?

Comment by Khalilah Gambrell [ 21/Sep/20 ]

Marc Johnson, I believe the design accounts for these fields to be searchable. The work to make fields searchable needs to be done by the app team.

Comment by Marc Johnson [ 21/Sep/20 ]

Khalilah Gambrell

I believe the design accounts for these fields to be searchable

Thanks for your quick response.

I'm curious about this because our current tooling (RAML Module Builder) defines databases indexes during design / coding, which effectively makes them static to every tenant. As I understand it, specific custom fields on the other hand, as designed to be defined on a per tenant basis and are not known during the design / coding part of the work.

Did part of that design involve discussions about how database indexing could be done for custom fields? If so, please could you refer me to documentation or to someone who could help me understand that better.

Is there an example implemented in users, which I believe was the first place custom fields were built, that can be used for reference?

Comment by jackie.gottlieb@duke.edu (Inactive) [ 21/Sep/20 ]

Marc Johnson RE: Given that inventory searches for instances, would this be something like find the instances where an item exists with a previous call number of ABC?

The answer is yes. I understand that the search capabilities don't take you directly to an item, they take you to an instance, but that is fine for now.

Comment by Dima Tkachenko [ 22/Sep/20 ]

Marc Johnson
Custom fields assigned to a record (like users) are searchable as any other fields, because they look inside Json as regular attributes.
Sample user record with custom fields ("customFields" attribute in Json):

{
  "username" : "fred",
  "id" : "956f39c5-92e3-4c26-bcdc-1827674710cf",
  "barcode" : "890127749735764",
  "active" : true,
  "type" : "patron",
  "patronGroup" : "bdc2b6d4-5ceb-4a12-ab46-249b9a68473e",
  "departments" : [ ],
  "proxyFor" : [ ],
  "personal" : {
    "lastName" : "Hilll",
    "firstName" : "Santino",
    "middleName" : "Travon",
    "email" : "kariane@douglas-hoeger-and-stokes.py",
    "phone" : "(234)617-6257 x0630",
    "mobilePhone" : "1-363-752-3867 x6224",
    "dateOfBirth" : "1995-04-02T00:00:00.000+0000",
    "addresses" : [ {
      "countryId" : "US",
      "addressLine1" : "12402 Leon Passage",
      "city" : "Rolling Hills Estates",
      "region" : "MI",
      "postalCode" : "98879-8861",
      "addressTypeId" : "93d3d88d-499b-45d0-9bc7-ac73c3a19880",
      "primaryAddress" : true
    } ],
    "preferredContactTypeId" : "002"
  },
  "enrollmentDate" : "2017-06-05T00:00:00.000+0000",
  "expirationDate" : "2020-09-28T00:00:00.000+0000",
  "createdDate" : "2020-09-22T11:04:36.825+0000",
  "updatedDate" : "2020-09-22T11:04:36.825+0000",
  "metadata" : {
    "createdDate" : "2020-09-22T01:41:26.447+0000",
    "updatedDate" : "2020-09-22T11:04:36.815+0000",
    "updatedByUserId" : "016bdad7-0959-5e33-af02-84a45a108e29"
  },
  "customFields" : {
    "phone" : "602-265-5968",
    "graduated" : true
  }
}

As you can see the user user has 2 CFs with attribute names "phone" and "graduated". These names are defined in CF section of user settings:

You can take a look at mod-users code to see how CFs integrated into Users.There is also a recording related to CFs overview conducted by Spitfire team. Please ask Khalilah Gambrell for a link

Regarding the indexes, there are no predefined indexes but you can add some in schema.json as you do in other cases

Comment by jackie.gottlieb@duke.edu (Inactive) [ 22/Sep/20 ]

Dima Tkachenko, Thank you! This is very helpful information.

Comment by Marc Johnson [ 22/Sep/20 ]

Dima Tkachenko Thank you for your response.

Custom fields assigned to a record (like users) are searchable as any other fields, because they look inside Json as regular attributes.

In inventory, due to the quantity of records, we typically do not support searching without an underlying database index, as that results in very poor performance. There is an ongoing debate about whether this should always be the case on FOLIO-2573 Closed .

Regarding the indexes, there are no predefined indexes but you can add some in schema.json as you do in other cases

I'm not sure I understand how that would work in practice.

Schema.json is used to define database indexes during development of a module version. As custom fields are defined specifically for a tenant, during the operation of the system, they are not known during development. How would developers know to define a database index for a property that does not exist during development, and may not exist for many organisations (tenants)?

Comment by Dima Tkachenko [ 22/Sep/20 ]

Marc Johnson
I'm not an expert in json indexes. As I understand if you what to create a specific index on a specific field, like "phone" in my example, this is not possible beforehand unless you know the name. May be you can some how build an index on the content of "customFields" element..
Cannot give you the immediate recommendations/guidance, it has to be investigated further

Comment by Marc Johnson [ 22/Sep/20 ]

Dima Tkachenko Thank you.

As I understand if you what to create a specific index on a specific field, like "phone" in my example, this is not possible beforehand unless you know the name.

That fits with my understanding.

Maybe you can some how build an index on the content of "customFields" element.

I don't think that indexing the custom fields object as a whole is likely to give folks the behaviour they expect. I think they expect to be able to search specifically on a particular field.

Cannot give you the immediate recomendations/guidance, it has to be investigated additionally

Who is going to do that investigation?

Comment by Mikhail Fokanov [ 22/Sep/20 ]

Searching over custom fields can be done smoothly using Elastic Search. Two approaches can be used depending on requirements:
1.) If there is a requirement to have facets for custom fields or to search only over one of this custom fields, https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html could be leveraged. In such case tenants can work with custom fields like with ordinary one even not caring about that they are custom.
The drawback of it can be that there could be a plenty of fields. And ES is not supposed to handle thousands of different fields in one index.
2.) More simple approach is to make heap of all custom fields in one ES field. It will be good in terms of searching performance. But it won't allow to search only over one field. Although there could be different fields for different types of values (e.g. string, number etc), it is pretty limited and I don't recommend to go with it.

Comment by jackie.gottlieb@duke.edu (Inactive) [ 22/Sep/20 ]

Marc Johnson, Thank you for your engagement and exploration of this. If I am understanding this so far, we can 'sort of' get the custom field indexed but the behavior may not be what is expected; investigation is needed. Mikhail is offering Elastic Search as possible solution, though as far as I am aware, Elastic Search hasn't yet been confirmed to be implemented. Charlotte Whitt, Where do we go from here? What should be the next step? I felt the need to explain why Duke ranked it R1 since Chicago and us are the only 2.

Comment by Khalilah Gambrell [ 22/Sep/20 ]

jackie.gottlieb@duke.edu and Charlotte Whitt, I recommend a separate feature = Inventory App | Search custom fields.

Comment by Marc Johnson [ 25/Sep/20 ]

jackie.gottlieb@duke.edu

Thank you for your engagement and exploration of this. If I am understanding this so far, we can 'sort of' get the custom field indexed but the behavior may not be what is expected;

At the moment, our approach to indexing requires developers to know the field to be indexed during development. As custom fields are inherently intended to be configured after development, the current approach doesn't make sense for them (we'd be adding indexes for fields that aren't there), without compromising their intent (that custom fields are custom and configured by tenants dynamically).

as far as I am aware, Elastic Search hasn't yet been confirmed to be implemented

The Technical Council are intending to discuss this next week.

Comment by Marc Johnson [ 25/Sep/20 ]

Mikhail Fokanov

Searching over custom fields can be done smoothly using Elastic Search

I have no doubt that elastic search can index custom fields. What I don't understand is how it inherently resolves the challenge with custom fields. Does the current Elastic Search proposal include information about how indexes in Elastic Search might be defined dynamically after development?

Comment by Mikhail Fokanov [ 25/Sep/20 ]

Does the current Elastic Search proposal include information about how indexes in Elastic Search might be defined dynamically after development

It doesn't include such information due to absence of this requirements at time of its writing. Dynamic mapping for Elastic Search indexes is needed only if there is such a requirement. So if Elastic is selected as the solution for this issue, I will add such implementation notes to the Search page.

Comment by Marc Johnson [ 25/Sep/20 ]

Mikhail Fokanov Thank you for the clarification

Comment by jackie.gottlieb@duke.edu (Inactive) [ 30/Nov/20 ]

Khalilah Gambrell, is there a reason for this feature/story not to be directly linked to the custom fields epic (https://folio-org.atlassian.net/browse/UXPROD-396)? What is the next step in making progress on this? Thanks, Jackie

Comment by Jon Miller [ 09/Dec/20 ]

Personally, I would like to see a property of some kind that lets you add in whatever extra properties you need, and not even necessarily have to be defined through Settings like custom fields do. For example, I need to add a date/time value property to holdings that indicate when a holding was stored to OCLC's database. It's not something that would need to be rendered in the FOLIO UI. It's just a field that an external batch program would make use of. Not sure if the customFields property could do that, or, if it should be some other property, like "extraFields" or something along those lines. It wouldn't need validation or anything. I have a need to do something similar in locations as well for the spine label printing app that I'm working on. It would be good if it were implemented across the board for all objects (or at least the top-level ones that appear as tables).

Comment by Charlotte Whitt [ 10/Dec/20 ]

Hi Jon Miller - I agree. I think the current implementation having the custom fields in a dedicated accordion, might just be a short term solution.

I'll loop in Khalilah Gambrell who will be more able to answer your question, and explain the short term solution and possible plans for more long term solution.

Comment by Khalilah Gambrell [ 10/Dec/20 ]

jackie.gottlieb@duke.edu, it is up to Charlotte Whitt and Core-Functional to move forward on this feature.

Generated at Fri Feb 09 00:22:04 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.