Custom Fields (UXPROD-396)

[UXPROD-33] Custom Fields (for User Record and as General Platform Feature): Phase 1 Backend Development Created: 18/Jan/18  Updated: 21/Mar/22  Resolved: 06/Mar/20

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Q1 2020
Parent: Custom Fields

Type: New Feature Priority: P3
Reporter: Cate Boerema (Inactive) Assignee: Khalilah Gambrell
Resolution: Done Votes: 3
Labels: crossplatform, q4-2019-at-risk, split, usermanagement
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Screenshot 2020-02-14 at 18.19.06.png     PNG File Screenshot 2020-02-14 at 18.25.05.png     PNG File Screenshot 2020-02-17 at 13.56.15.png    
Issue links:
Blocks
blocks UXPROD-2248 Tokens - Add custom fields to patron ... Draft
Cloners
is cloned by UXPROD-2143 Q1 2020/Q2 2020 | Custom Fields (for ... Closed
Defines
is defined by MODCFIELDS-40 Make RefId a read-only field Closed
is defined by MODCFIELDS-38 Security update jackson-databind >= 2... Closed
is defined by MODCFIELDS-39 Prepare for Custom Field backend demo Closed
is defined by MODKBEKBJ-346 Resource is not removed if request ha... Closed
is defined by MODCFIELDS-36 Text field can be created without req... Closed
is defined by MODUSERS-179 PUT /custom-fields endpoint is not av... Closed
is defined by MODUSERS-183 Update mod-users to use new version o... Closed
Relates
relates to UXPROD-396 Custom Fields In Progress
relates to UICFIELDS-23 Spike: Define Custom Field UI design ... Closed
relates to UIU-1492 Custom fields: Create UI permissions ... Closed
relates to UXPROD-2211 Custom Fields in Inventory Draft
relates to UXPROD-2212 Settings page. Inventory. Each instit... Open
relates to MODCFIELDS-10 User Record | Import Custom Field | U... Closed
relates to MODCFIELDS-32 Custom Fields: Support search by refI... Closed
relates to MODCFIELDS-33 Custom Fields: Support update by refId Closed
relates to MODCFIELDS-34 Custom Fields: Support delete by refId Closed
relates to MODCFIELDS-35 User Record | Import Custom Field | U... Closed
relates to MODKBEKBJ-334 Custom labels: Expand POST title payl... Closed
relates to MODKBEKBJ-341 Custom labels: update custom labels list Closed
relates to MODKBEKBJ-343 Custom labels: delete custom label Closed
Requires
requires MODUSERS-134 [SPIKE] Custom Fields | Define archi... Closed
requires MODUSERS-146 Integrate custom fields to mod-users Closed
requires MODCFIELDS-3 Define DB schema Closed
requires MODCFIELDS-4 Implement POST custom field endpoint Closed
requires MODCFIELDS-5 Implement GET custom fields collectio... Closed
requires MODCFIELDS-6 Implement GET custom field by id endp... Closed
requires MODCFIELDS-7 Implement PUT custom field by id endp... Closed
requires MODCFIELDS-8 Implement DELETE custom field by id e... Closed
requires MODCFIELDS-12 Document X-Okapi-Module-Id multiple i... Closed
requires MODCFIELDS-13 Custom fields: Common fields Definiti... Closed
requires MODCFIELDS-14 Spike: Custom fields validation Closed
requires MODCFIELDS-15 Custom Fields: Text field/area Values... Closed
requires MODCFIELDS-16 Custom Fields: Dropdown Definition Va... Closed
requires MODCFIELDS-17 Custom Fields: Radio Button Definitio... Closed
requires MODCFIELDS-18 Support custom field usage statistic Closed
requires MODCFIELDS-19 Custom Fields: Checkbox Values Valida... Closed
requires MODCFIELDS-27 Custom Fields: Support fields order o... Closed
requires MODUSERS-149 Add Custom fields to user schema Closed
requires MODUSERS-172 Remove related values when custom fie... Closed
is required by STSMACOM-286 Custom Field: Input Option: Text area Closed
is required by UXPROD-2288 Q2 2020 | Custom Fields (for User Rec... Closed
is required by UXPROD-2289 Q3 2020 | Custom Fields (for User Rec... Closed
Potential Workaround: repurpose existing fields (e.g., notes) -- Laura W.
Epic Link: Custom Fields
Analysis Estimate: Medium < 5 days
Analysis Estimator: Khalilah Gambrell
Front-End Confidence factor: Medium
Back End Estimate: Large < 10 days
Development Team: Spitfire
PO Rank: 95
Rank: BNCF (MVP Feb 2020): R1
Rank: Chalmers (Impl Aut 2019): R5
Rank: Chicago (MVP Sum 2020): R1
Rank: Cornell (Full Sum 2021): R4
Rank: Duke (Full Sum 2021): R1
Rank: 5Colleges (Full Jul 2021): R1
Rank: FLO (MVP Sum 2020): R1
Rank: GBV (MVP Sum 2020): R1
Rank: hbz (TBD): R1
Rank: Hungary (MVP End 2020): R1
Rank: Lehigh (MVP Summer 2020): R1
Rank: Leipzig (Full TBD): R1
Rank: Leipzig (ERM Aut 2019): R1
Rank: MO State (MVP June 2020): R1
Rank: TAMU (MVP Jan 2021): R4
Rank: U of AL (MVP Oct 2020): R2

 Description   

5/30/2018: Per UM SIG - we need custom fields. The SIG will leverage tagging too but custom defined fields supports the need of more permanent storing of information, key/value pairs, and supports data migration too. Priority - Critical

Documentation

https://docs.google.com/presentation/d/1zXvDSw-zJ36dKWv1C-W3-x8yOmKNu01c3LV0prJh1F8/edit#slide=id.p



 Comments   
Comment by Cate Boerema (Inactive) [ 13/Jun/18 ]

Hi Khalilah Gambrell, I hope it's okay that I extended the description of this feature to make it a general platform feature. I assume you agree that, if we build custom field functionality, we should do it in such a way that it can be leveraged for different record types (beyond Users). LMK if you see issues with this.

Comment by Ann-Marie Breaux (Inactive) [ 19/Jul/18 ]

Had a discussion about statistical search code as a standard field in Inventory instance, holdings, item in MM SIG mtg on 19 July 2018. See notes/recording from that meeting. If custom fields are available, that would serve the purpose for statistical search code. Discuss more with Charlotte Whitt, Christie Thomas. Perhaps have a standard field for now in the inventory records and deprecate once custom fields are available.

Comment by Charlotte Whitt [ 20/Jul/18 ]

Hi Khalilah Gambrell - This feature is set in Draft mode - do we know when Custom Fields are planned for?

Comment by Cate Boerema (Inactive) [ 20/Jul/18 ]

I know this question was to Khalilah Gambrell, but I thought I'd chime in. We are still working on our plans for beyond Q3 2018 (custom fields isn't targeted for Q3)

Comment by VBar [ 09/Nov/18 ]

I'm not convinced that custom fields should be a platform-level feature. Given microservices and the bounded-contexts they depend on, the case should be made that each domain would not be better suited to provide its own separate custom field implementation. Are custom fields for users really the same as custom fields for an order record or an item record, etc?

Comment by Cate Boerema (Inactive) [ 15/Nov/18 ]

I'm not convinced that custom fields should be a platform-level feature. Given microservices and the bounded-contexts they depend on, the case should be made that each domain would not be better suited to provide its own separate custom field implementation. Are custom fields for users really the same as custom fields for an order record or an item record, etc?

VBar, the intention behind the "as a General Platform Feature" note was to convey that we would not look at the User record needs in a vacuum and that would strive to develop a UX and code that could be leveraged for other apps and record types.

Comment by Anya [ 29/Mar/19 ]

Comment from March Meeting: This could be needed for Users

Comment by Erin Nettifee [ 03/Apr/19 ]

This is definitely needed for users. This also has major implications for the work being done right now about the reporting data model being developed now as well.

Comment by Taras Spashchenko [ 02/Aug/19 ]

I agree with VBar. It would make sense to make it a platform-level feature if FOLIO was a monolith. But despite the fact that it looks like one big app on UI, FOLIO is microservices by its nature. And such platform-level features bring strong coupling between components of the platform, which directly contradicts its architecture.

Comment by Erin Nettifee [ 02/Aug/19 ]

I think Cate's comment upthread addresses this - the intention is to develop something that is not considered in a vacuum and could be implemented for other apps if desired, not necessarily something that is guaranteed to be available at the platform level (thus requiring a higher level of connection and support that FOLIO isn't supposed to do.) We're seeing that happening right now with notes, for example. It was initially built for eHoldings, and is being repurposed into Users with tweaks for the specific needs that Users has.

Comment by Erin Nettifee [ 26/Aug/19 ]

Is there a reason why this lost the cap-mvp label?

Comment by Khalilah Gambrell [ 06/Nov/19 ]

Changing estimates to represent remaining backend development work.

Comment by Jon Miller [ 12/Feb/20 ]

Is there a description somewhere on what is being proposed implementation-wise on how custom fields will be implemented?

I see the following property in user.json

    "customFields": {
      "description": "Object that contains custom field",
      "type": "object"
    }

What I'm hoping for is set

  "additionalProperties": true,

on that object. And then allow custom properties to appear there.

What I'm hoping it is not doing, is storing things in an array in an odd format.

The format I would like to see is something like the following where you can simply add properties in the customFields property.

{
  "username": "jemiller"
  "personal": {
    "firstName": "Jon"
  },
  "customFields": {
    "departmentName": "Library",
    "divisionName": "Library"
  }
}

I am concerned that if the fields appear in an array, it will make updates difficult. For example, I have an application that updates users. It uses the standard FOLIO user JSON schema as the file format. I want to use the standard schema because I don't want to define a custom one. I want keep things simple. With regard to doing updates, you need to merge the data with what's already in the table. For arrays, if the property for the array exists, the array is overwritten in it's entirety.

I would like to see it defined like it is above, that way it is kept in a tabular format without repeating fields.

Also, I am against creating another module to store this data. IMHO, it should be stored in the entity's JSON itself.

Comment by Jon Miller [ 12/Feb/20 ]

I'm wondering if when a custom field is defined, it is actually added to the JSON schema. Maybe you could have

{entity}

CustomFields.json JSON schemas that are included which is modified by the institution.

Comment by Jon Miller [ 12/Feb/20 ]

Sorry, somehow I missed the links at the top. I will have a look now.

Comment by Khalilah Gambrell [ 12/Feb/20 ]

Hi Jon Miller, we will discuss backend development at the Users Management SIG on Wednesday, Feb 26 at 10am ET. Also all dev work is linked to this feature. I appreciate the questions and I will ask the dev team to review and respond.

Comment by Jon Miller [ 12/Feb/20 ]

Khalilah Gambrell Thanks. It looks like you're doing a nice job on the UI. I'm just curious how the data is stored. I'm hoping it's stored in a straightforward fashion so that the fields basically work the same way non-custom fields work.

Comment by Jon Miller [ 12/Feb/20 ]

I just realized that it appears that JSON schema sets additionalProperties to true by default. So, the customFields property in users.json already allows adding additional properties. I thought maybe that was just a stub for something that wasn't yet implemented.

Comment by Jon Miller [ 14/Feb/20 ]

Khalilah Gambrell Can you post a sample JSON document that has custom fields in it?

Comment by Natalia Zaitseva [ 14/Feb/20 ]

Hi Jon Miller the idea under custom fields is to have the same interface which can be used in different modules. The assignment part expect to have the following mapping

 "refId": "value"

. As we do not know what refId names and values to expect, it was decided to use

"additionalProperties": true

Once you create a custom field

you will get a response with `refId` field and then you are able to assign a value to it

and the particular JSON documents used in samples are below
custom field:

{
  
  "visible": true,
  "required": true,
  "type": "TEXTBOX_SHORT",
  "name": "my field",
  "order": 1,
  "entityType": "user",
  "helpText": "Choose your favorite food"
}

and assignment

{
  "username": "test",
  "id": "59793cf1-dbb8-4f8d-bc47-3d23afbf1f71",
  "active": true,
  "patronGroup": "503a81cd-6c26-400f-b620-14c08943697c",
  "proxyFor": [],
  "personal": {
    "lastName": "test",
    "firstName": "test",
    "middleName": "test",
    "email": "marek.kamarie@sadd.us",
    "addresses": [],
    "preferredContactTypeId": "002"
  },
  "customFields": {
    "my-field_2": "qw"
  }
}

Here is the case with several custom fields assignment

"my-field_1": "qwqw",
"my-field_2": "pizza"

If you need any additional info about implementation you can ask me.

Comment by Jon Miller [ 14/Feb/20 ]

Thanks Natalia Zaitseva That seems to look good to me. I just wanted to make sure that the custom fields weren't stored in some weird array or something with a bunch of other metadata. They basically look just like all the other fields do, just under the customFields property. So, that's fine with me. That's what I was hoping for. So, the definition of the fields will be stored in a table in some other module? It might be nice if there was a way for it to generate a JSON schema that was included in the objects JSON schema as a reference under customFields. That's not really hugely important IMHO though. As long as the fields can be easily stored and retrieved for the given object, the same way all the other standard properties are, it looks good to me.

Comment by Jon Miller [ 14/Feb/20 ]

Actually, how would validation of those fields work? For example, say the field is something like a drop down list where you are only allowed to specify certain values? If it has to query a table in another module every time you insert or update a record, I could see that causing performance problems.

Comment by Natalia Zaitseva [ 17/Feb/20 ]

Jon Miller

So, the definition of the fields will be stored in a table in some other module?

If it has to query a table in another module every time you insert or update a record, I could see that causing performance problems.

Each module, which uses custom fields will have a table with definitions in the same schema for the module. For instance, mod-users will have its own table 'custom-fields'

so, there is no need to query a table in another module, because all data needed will be tied to the module schema in the database, i.e '<tenant>_mod_users>', see an example on the screenshot above.

It might be nice if there was a way for it to generate a JSON schema that was included in the objects JSON schema as a reference under customFields.

If you mean the case when a user selects the custom field value, then based on our model where we expect to have mapping like

"refId": "value"

where 'refId' - is the key constant, we can not predict what 'refId' custom field will have, as well as the value, whether it will be text field or multi-select dropdown.

Actually, how would validation of those fields work?

Well, we split up the validation of custom field definition and values assignment. Based on the type of the custom field there are a set of properties allowed. Then, we validate common fields, like name or description and the fields for a particular custom field type. During the assignment phase, when the custom field is assigned to the user, we perform the validation of the value(s).

Comment by Jon Miller [ 17/Feb/20 ]

Thanks Natalia Zaitseva. So, you are saying that users POST and PUT don't validate custom fields? If I understand you correctly, there are separate APIs that are used to validate custom fields (I mean when I do a POST to the users endpoint, does that endpoint validate the values in the customFields property)?

Comment by Natalia Zaitseva [ 17/Feb/20 ]

Jon Miller
Both parts(custom fields creation and custom field assignment) have separate validation
I mean, when a user creates/updates a custom field(we call it custom field definition) i.e. request

POST or PUT /custom-fields

backend validates fields defined for that custom field type
when a user sets the custom field value to a user record(we call it assignment) i.e

POST or PUT /users/<user_id>

backend have another validation to check if the entered value is allowed for that type of the custom field.

Comment by Jon Miller [ 17/Feb/20 ]

OK, so, the standard /users POST/PUT does validate custom fields. So, the question at that point, is, does that end point have to query the custom_fields table each time through in order to do the validation? The concern is that it is creating extra overhead and will slow down user POST/PUT. If it's doing extra database look ups, it will be slower than simply doing a JSON schema validation.

Comment by Marc Johnson [ 17/Feb/20 ]

Natalia Zaitseva

POST or PUT /custom-fields

Is /custom-fields an endpoint in users storage module, or is the definition of custom fields done centrally within it's own module?

Comment by Jon Miller [ 17/Feb/20 ]

I see a mod-custom-fields module in the developer documentation. And I see the JSON schemas for the custom field tables in there.

Comment by Marc Johnson [ 17/Feb/20 ]

Jon Miller

I see a mod-custom-fields module in the developer documentation. And I see the JSON schemas for the custom field tables in there.

As I understand it, from a previous conversation, mod-custom-fields isn't a module rather a shared library for use in Java modules that use custom fields.

Natalia Zaitseva Please correct me if I'm wrong?

Comment by Natalia Zaitseva [ 17/Feb/20 ]

Marc Johnson yes, it is a library, that any module can add it.

Comment by Natalia Zaitseva [ 17/Feb/20 ]

Jon Miller

So, the question at that point, is, does that endpoint have to query the custom_fields table each time through in order to do the validation?

yes. it has to, as well as user address-type, patron group validation.

Comment by Jon Miller [ 17/Feb/20 ]

The users module should not have to do extra look ups for address type or patron group. It can simply rely on database foreign key constraints for that. That is a more efficient way to do it. Any API that is doing a lot of extra look ups on each insert/update, is going to be slow with regard to bulk insert/updates.

Maybe it won't be that big of a deal for the users module assuming institutions don't have a whole lot of users, but, I think it will cause performance problems on bigger tables. For example, if you had it on instances, holdings, or items. I don't know if there is a need for that off hand. I'm just saying that it's a concern that I have.

IMHO, checks that can be done using foreign key constraints, should be done that way rather than doing extra look ups.

Regarding, validating custom fields, one way to get around doing extra look ups would be to store the validation rules as JSON schema in a separate file that is referenced from the main user.json. When a user adds a custom field through settings, write the JSON schema for it to the referenced file. I don't know how JSON schemas are read in FOLIO. I'm assuming they are read by file? There is overhead parsing JSON schema files. It would be best to read the files once into a static variable. Then, just use that variable each time an insert/update happens. You could store the custom field JSON schema in a table, or as a file. You could have it where it's written to disk and read when FOLIO starts, but, assuming you don't want to require a restart of the module after a custom field has been added, you could have it refresh the the JSON schema static variable.

Basically, what I'm getting at, is that extra table look ups should be avoided where possible. And that should be taken into consideration when designing how custom field validation works.

Comment by Jon Miller [ 17/Feb/20 ]

Natalia Zaitseva So, do the APIs listed at https://s3.amazonaws.com/foliodocs/api/mod-custom-fields/p/custom-fields.html get mapped as URLs under /users for the users module?

Comment by Jon Miller [ 17/Feb/20 ]

Although I haven't thought it through all that much, I almost think it would be better if the custom_fields table only occurred once in it's own schema/module. I'm wondering what good it is having it each module?

Comment by Oleksii Petrenko [ 18/Feb/20 ]

Hi Jon Miller
I suggest to move answering to question to demo session to 2/26
And also please continue put your question here.
We will compose all of them before demo then discuss all of them during demo one by one.
CC Khalilah Gambrell

Comment by Marc Johnson [ 18/Feb/20 ]

Oleksii Petrenko

I suggest to move answering to question to demo session to 2/26

Who is the audience of the demo? Is it an open invite?

Comment by Khalilah Gambrell [ 18/Feb/20 ]

Marc Johnson, we are presenting at the 2/26 User Management SIG meeting. I will forward details

Comment by Marc Johnson [ 18/Feb/20 ]

Khalilah Gambrell Thanks

Generated at Fri Feb 09 00:05:17 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.