Custom Fields
(UXPROD-396)
|
|
| Status: | Closed |
| Project: | UX Product |
| Components: | None |
| Affects versions: | None |
| Fix versions: | Q1 2020 | Parent: | Custom Fields |
| Type: | New Feature | Priority: | P3 |
| Reporter: | Cate Boerema (Inactive) | Assignee: | Khalilah Gambrell |
| Resolution: | Done | Votes: | 3 |
| Labels: | crossplatform, q4-2019-at-risk, split, usermanagement | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Potential Workaround: | repurpose existing fields (e.g., notes) -- Laura W. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Epic Link: | Custom Fields | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Analysis Estimate: | Medium < 5 days | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Analysis Estimator: | Khalilah Gambrell | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Front-End Confidence factor: | Medium | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Back End Estimate: | Large < 10 days | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Development Team: | Spitfire | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| PO Rank: | 95 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: BNCF (MVP Feb 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Chalmers (Impl Aut 2019): | R5 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Chicago (MVP Sum 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Cornell (Full Sum 2021): | R4 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Duke (Full Sum 2021): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: 5Colleges (Full Jul 2021): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: FLO (MVP Sum 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: GBV (MVP Sum 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: hbz (TBD): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Hungary (MVP End 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Lehigh (MVP Summer 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Leipzig (Full TBD): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: Leipzig (ERM Aut 2019): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: MO State (MVP June 2020): | R1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: TAMU (MVP Jan 2021): | R4 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank: U of AL (MVP Oct 2020): | R2 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
5/30/2018: Per UM SIG - we need custom fields. The SIG will leverage tagging too but custom defined fields supports the need of more permanent storing of information, key/value pairs, and supports data migration too. Priority - Critical Documentationhttps://docs.google.com/presentation/d/1zXvDSw-zJ36dKWv1C-W3-x8yOmKNu01c3LV0prJh1F8/edit#slide=id.p |
| Comments |
| Comment by Cate Boerema (Inactive) [ 13/Jun/18 ] |
|
Hi Khalilah Gambrell, I hope it's okay that I extended the description of this feature to make it a general platform feature. I assume you agree that, if we build custom field functionality, we should do it in such a way that it can be leveraged for different record types (beyond Users). LMK if you see issues with this. |
| Comment by Ann-Marie Breaux (Inactive) [ 19/Jul/18 ] |
|
Had a discussion about statistical search code as a standard field in Inventory instance, holdings, item in MM SIG mtg on 19 July 2018. See notes/recording from that meeting. If custom fields are available, that would serve the purpose for statistical search code. Discuss more with Charlotte Whitt, Christie Thomas. Perhaps have a standard field for now in the inventory records and deprecate once custom fields are available. |
| Comment by Charlotte Whitt [ 20/Jul/18 ] |
|
Hi Khalilah Gambrell - This feature is set in Draft mode - do we know when Custom Fields are planned for? |
| Comment by Cate Boerema (Inactive) [ 20/Jul/18 ] |
|
I know this question was to Khalilah Gambrell, but I thought I'd chime in. We are still working on our plans for beyond Q3 2018 (custom fields isn't targeted for Q3) |
| Comment by VBar [ 09/Nov/18 ] |
|
I'm not convinced that custom fields should be a platform-level feature. Given microservices and the bounded-contexts they depend on, the case should be made that each domain would not be better suited to provide its own separate custom field implementation. Are custom fields for users really the same as custom fields for an order record or an item record, etc? |
| Comment by Cate Boerema (Inactive) [ 15/Nov/18 ] |
VBar, the intention behind the "as a General Platform Feature" note was to convey that we would not look at the User record needs in a vacuum and that would strive to develop a UX and code that could be leveraged for other apps and record types. |
| Comment by Anya [ 29/Mar/19 ] |
|
Comment from March Meeting: This could be needed for Users |
| Comment by Erin Nettifee [ 03/Apr/19 ] |
|
This is definitely needed for users. This also has major implications for the work being done right now about the reporting data model being developed now as well. |
| Comment by Taras Spashchenko [ 02/Aug/19 ] |
|
I agree with VBar. It would make sense to make it a platform-level feature if FOLIO was a monolith. But despite the fact that it looks like one big app on UI, FOLIO is microservices by its nature. And such platform-level features bring strong coupling between components of the platform, which directly contradicts its architecture. |
| Comment by Erin Nettifee [ 02/Aug/19 ] |
|
I think Cate's comment upthread addresses this - the intention is to develop something that is not considered in a vacuum and could be implemented for other apps if desired, not necessarily something that is guaranteed to be available at the platform level (thus requiring a higher level of connection and support that FOLIO isn't supposed to do.) We're seeing that happening right now with notes, for example. It was initially built for eHoldings, and is being repurposed into Users with tweaks for the specific needs that Users has. |
| Comment by Erin Nettifee [ 26/Aug/19 ] |
|
Is there a reason why this lost the cap-mvp label? |
| Comment by Khalilah Gambrell [ 06/Nov/19 ] |
|
Changing estimates to represent remaining backend development work. |
| Comment by Jon Miller [ 12/Feb/20 ] |
|
Is there a description somewhere on what is being proposed implementation-wise on how custom fields will be implemented? I see the following property in user.json
"customFields": {
"description": "Object that contains custom field",
"type": "object"
}
What I'm hoping for is set "additionalProperties": true, on that object. And then allow custom properties to appear there. What I'm hoping it is not doing, is storing things in an array in an odd format. The format I would like to see is something like the following where you can simply add properties in the customFields property.
{
"username": "jemiller"
"personal": {
"firstName": "Jon"
},
"customFields": {
"departmentName": "Library",
"divisionName": "Library"
}
}
I am concerned that if the fields appear in an array, it will make updates difficult. For example, I have an application that updates users. It uses the standard FOLIO user JSON schema as the file format. I want to use the standard schema because I don't want to define a custom one. I want keep things simple. With regard to doing updates, you need to merge the data with what's already in the table. For arrays, if the property for the array exists, the array is overwritten in it's entirety. I would like to see it defined like it is above, that way it is kept in a tabular format without repeating fields. Also, I am against creating another module to store this data. IMHO, it should be stored in the entity's JSON itself. |
| Comment by Jon Miller [ 12/Feb/20 ] |
|
I'm wondering if when a custom field is defined, it is actually added to the JSON schema. Maybe you could have {entity}CustomFields.json JSON schemas that are included which is modified by the institution. |
| Comment by Jon Miller [ 12/Feb/20 ] |
|
Sorry, somehow I missed the links at the top. I will have a look now. |
| Comment by Khalilah Gambrell [ 12/Feb/20 ] |
|
Hi Jon Miller, we will discuss backend development at the Users Management SIG on Wednesday, Feb 26 at 10am ET. Also all dev work is linked to this feature. I appreciate the questions and I will ask the dev team to review and respond. |
| Comment by Jon Miller [ 12/Feb/20 ] |
|
Khalilah Gambrell Thanks. It looks like you're doing a nice job on the UI. I'm just curious how the data is stored. I'm hoping it's stored in a straightforward fashion so that the fields basically work the same way non-custom fields work. |
| Comment by Jon Miller [ 12/Feb/20 ] |
|
I just realized that it appears that JSON schema sets additionalProperties to true by default. So, the customFields property in users.json already allows adding additional properties. I thought maybe that was just a stub for something that wasn't yet implemented. |
| Comment by Jon Miller [ 14/Feb/20 ] |
|
Khalilah Gambrell Can you post a sample JSON document that has custom fields in it? |
| Comment by Natalia Zaitseva [ 14/Feb/20 ] |
|
Hi Jon Miller the idea under custom fields is to have the same interface which can be used in different modules. The assignment part expect to have the following mapping "refId": "value" . As we do not know what refId names and values to expect, it was decided to use "additionalProperties": true Once you create a custom field {
"visible": true,
"required": true,
"type": "TEXTBOX_SHORT",
"name": "my field",
"order": 1,
"entityType": "user",
"helpText": "Choose your favorite food"
}
and assignment
{
"username": "test",
"id": "59793cf1-dbb8-4f8d-bc47-3d23afbf1f71",
"active": true,
"patronGroup": "503a81cd-6c26-400f-b620-14c08943697c",
"proxyFor": [],
"personal": {
"lastName": "test",
"firstName": "test",
"middleName": "test",
"email": "marek.kamarie@sadd.us",
"addresses": [],
"preferredContactTypeId": "002"
},
"customFields": {
"my-field_2": "qw"
}
}
Here is the case with several custom fields assignment "my-field_1": "qwqw", "my-field_2": "pizza" If you need any additional info about implementation you can ask me. |
| Comment by Jon Miller [ 14/Feb/20 ] |
|
Thanks Natalia Zaitseva That seems to look good to me. I just wanted to make sure that the custom fields weren't stored in some weird array or something with a bunch of other metadata. They basically look just like all the other fields do, just under the customFields property. So, that's fine with me. That's what I was hoping for. So, the definition of the fields will be stored in a table in some other module? It might be nice if there was a way for it to generate a JSON schema that was included in the objects JSON schema as a reference under customFields. That's not really hugely important IMHO though. As long as the fields can be easily stored and retrieved for the given object, the same way all the other standard properties are, it looks good to me. |
| Comment by Jon Miller [ 14/Feb/20 ] |
|
Actually, how would validation of those fields work? For example, say the field is something like a drop down list where you are only allowed to specify certain values? If it has to query a table in another module every time you insert or update a record, I could see that causing performance problems. |
| Comment by Natalia Zaitseva [ 17/Feb/20 ] |
Each module, which uses custom fields will have a table with definitions in the same schema for the module. For instance, mod-users will have its own table 'custom-fields'
If you mean the case when a user selects the custom field value, then based on our model where we expect to have mapping like "refId": "value" where 'refId' - is the key constant, we can not predict what 'refId' custom field will have, as well as the value, whether it will be text field or multi-select dropdown.
Well, we split up the validation of custom field definition and values assignment. Based on the type of the custom field there are a set of properties allowed. Then, we validate common fields, like name or description and the fields for a particular custom field type. During the assignment phase, when the custom field is assigned to the user, we perform the validation of the value(s). |
| Comment by Jon Miller [ 17/Feb/20 ] |
|
Thanks Natalia Zaitseva. So, you are saying that users POST and PUT don't validate custom fields? If I understand you correctly, there are separate APIs that are used to validate custom fields (I mean when I do a POST to the users endpoint, does that endpoint validate the values in the customFields property)? |
| Comment by Natalia Zaitseva [ 17/Feb/20 ] |
|
Jon Miller POST or PUT /custom-fields backend validates fields defined for that custom field type POST or PUT /users/<user_id> backend have another validation to check if the entered value is allowed for that type of the custom field. |
| Comment by Jon Miller [ 17/Feb/20 ] |
|
OK, so, the standard /users POST/PUT does validate custom fields. So, the question at that point, is, does that end point have to query the custom_fields table each time through in order to do the validation? The concern is that it is creating extra overhead and will slow down user POST/PUT. If it's doing extra database look ups, it will be slower than simply doing a JSON schema validation. |
| Comment by Marc Johnson [ 17/Feb/20 ] |
Is /custom-fields an endpoint in users storage module, or is the definition of custom fields done centrally within it's own module? |
| Comment by Jon Miller [ 17/Feb/20 ] |
|
I see a mod-custom-fields module in the developer documentation. And I see the JSON schemas for the custom field tables in there. |
| Comment by Marc Johnson [ 17/Feb/20 ] |
As I understand it, from a previous conversation, mod-custom-fields isn't a module rather a shared library for use in Java modules that use custom fields. Natalia Zaitseva Please correct me if I'm wrong? |
| Comment by Natalia Zaitseva [ 17/Feb/20 ] |
|
Marc Johnson yes, it is a library, that any module can add it. |
| Comment by Natalia Zaitseva [ 17/Feb/20 ] |
yes. it has to, as well as user address-type, patron group validation. |
| Comment by Jon Miller [ 17/Feb/20 ] |
|
The users module should not have to do extra look ups for address type or patron group. It can simply rely on database foreign key constraints for that. That is a more efficient way to do it. Any API that is doing a lot of extra look ups on each insert/update, is going to be slow with regard to bulk insert/updates. Maybe it won't be that big of a deal for the users module assuming institutions don't have a whole lot of users, but, I think it will cause performance problems on bigger tables. For example, if you had it on instances, holdings, or items. I don't know if there is a need for that off hand. I'm just saying that it's a concern that I have. IMHO, checks that can be done using foreign key constraints, should be done that way rather than doing extra look ups. Regarding, validating custom fields, one way to get around doing extra look ups would be to store the validation rules as JSON schema in a separate file that is referenced from the main user.json. When a user adds a custom field through settings, write the JSON schema for it to the referenced file. I don't know how JSON schemas are read in FOLIO. I'm assuming they are read by file? There is overhead parsing JSON schema files. It would be best to read the files once into a static variable. Then, just use that variable each time an insert/update happens. You could store the custom field JSON schema in a table, or as a file. You could have it where it's written to disk and read when FOLIO starts, but, assuming you don't want to require a restart of the module after a custom field has been added, you could have it refresh the the JSON schema static variable. Basically, what I'm getting at, is that extra table look ups should be avoided where possible. And that should be taken into consideration when designing how custom field validation works. |
| Comment by Jon Miller [ 17/Feb/20 ] |
|
Natalia Zaitseva So, do the APIs listed at https://s3.amazonaws.com/foliodocs/api/mod-custom-fields/p/custom-fields.html get mapped as URLs under /users for the users module? |
| Comment by Jon Miller [ 17/Feb/20 ] |
|
Although I haven't thought it through all that much, I almost think it would be better if the custom_fields table only occurred once in it's own schema/module. I'm wondering what good it is having it each module? |
| Comment by Oleksii Petrenko [ 18/Feb/20 ] |
|
Hi Jon Miller |
| Comment by Marc Johnson [ 18/Feb/20 ] |
Who is the audience of the demo? Is it an open invite? |
| Comment by Khalilah Gambrell [ 18/Feb/20 ] |
|
Marc Johnson, we are presenting at the 2/26 User Management SIG meeting. I will forward details |
| Comment by Marc Johnson [ 18/Feb/20 ] |
|
Khalilah Gambrell Thanks |