Tags - Basic (UXPROD-250)

[FOLIO-1194] design and implement tag backend Created: 10/Apr/18  Updated: 01/Feb/19  Resolved: 02/May/18

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None
Parent: Tags - Basic

Type: Task Priority: P3
Reporter: Jakub Skoczen Assignee: Heikki Levanto
Resolution: Done Votes: 0
Labels: core, phase_1, sprint36, sprint37, tags
Remaining Estimate: Not Specified
Time Spent: 3 hours
Original estimate: Not Specified

Issue links:
Blocks
blocks STSMACOM-113 Tags on individual records: Assign, U... Closed
Sprint:
Epic Link: Tags - Basic

 Description   

We are leaning towards a solution where the literal string value is kept on the record. As such, there are two elements to the proposed solution:

1. a new JSON schema for tags (similar to the metadata schema, but not readonly), implementation of the handling in RMB, filtering will be automatically possible through CQL by using the array fields workaround

2. central module (mod-tags) for keeping a list of values used for tagging across FOLIO, there is a list of questions here that will guide the implementation:

  • deleting tags: should tags from the central list get removed when the last tags from tagged records is removed? (preferably not as it increases complexity) No, tags remain on the central list until removed via a central administrative action. That functionality comes in phase-2.
  • deleting tags: if tags are centrally managed, should removal of the tag from the central storage be prevented if the tag is used in any of the modules/records (again, preferably not as it requires consistency checks between the tag module and the specific module) Tag cannot be removed from the central tag list unless it is not assigned to any record in any FOLIO app. The central tag list will also be a way to review records across all apps that have a specific tag, so it's going to need to be aware of how each tag is being used in each app anyway.
  • allowing/disallowing tags: do we validate tag values at the point of assignment or only provide values as an auto-completion aid? (again preferably no validation happens) No validation, just auto-suggest, I think. However, we're going to have a permission that allows a user to only assign existing tags (tag must already be on the central tag list) versus a permission that allows the user to assign a completely new tag (which will then need to be automatically added to the central tag list). Not sure if that has an impact on the design you're considering or not.

As discussed on the call, to provide validation or prevent removal we need to create a dependency between a given storage module and the tag module (to perform lookup at the point of assignment/removal), or construct a business logic module to provide it.

Alternatively, we could look at some sort of push approach where the business logic modules would provide updates to the interested modules that keep a list of all tags. This is the most complex solution.

In case it's helpful, for phase 2 (working with a tags central admin area), we're also considering the idea of a "tag record" that would have the tag, plus also a description field, where someone could describe how the tag is to be used. Any tags created in phase 1 would not have descriptions, but the descriptions could be added later, once phase 2 allows access to the central tag list and individual tag records.



 Comments   
Comment by Ann-Marie Breaux (Inactive) [ 11/Apr/18 ]

Hi Jakub Skoczen See answers to your questions in-line above.

Comment by Heikki Levanto [ 11/Apr/18 ]

The tags module can not really be aware of all the other modules that use tags - Anyone can write new modules. So it has no easy way to check if anyone is using a given tag, in order to check if it can be deleted. There may be ways to work around that, but it won't be easy...

Comment by Ann-Marie Breaux (Inactive) [ 11/Apr/18 ]

Thanks, Heikki Levanto. The intention for phase 2 (which will follow ASAP after phase 1) is that you will be able to see a comprehensive list of where a particular tag is used across all apps. So somehow the central tags app will need to be able to understand where that tag is being used and allow a user to drill down into each record in each app that has a particular tag. Since the apps will be checking tags against the central tag list (for auto-suggest purposes, and to know whether a newly-assigned tag is new or existing), it seems like communication will needs to go both ways, somehow.

Comment by shale99 [ 11/Apr/18 ]

hi Ann-Marie Breaux
read through STSMACOM-113 Closed , i did not see any search functionality on tags other then autocomplete. did i miss something? is there a search requirement that tags can be searched and that the search on tags can be combined with other fields in a record for filtering?

Comment by Jakub Skoczen [ 11/Apr/18 ]

Ann-Marie Breaux preventing removal from the central list unless a tag is not use is tricky because FOLIO is a distributed system and different modules (that can include records tagged with the value) can be enabled/disabled dynamically. I would propose a solution where we mark a given tag as disabled, effective removing it from the cetral list and preventing any future tagging to include the value. But the values can remain in use for the records tagged previously.

Heikki Levanto I am hoping this would mean that we don't need to validate tag assignment in any other way than through the UI component that is used for tagging.

Comment by Heikki Levanto [ 11/Apr/18 ]

I think it is fine to mark a tag as disabled. We might even want two markers, one to disable assigning the tag to any object, and one for disabling search/filtering by that tag. Philosophically, the central list will then be more like a list of tags the users may use, rather than a fully controlled vocabulary of all allowed tag values.

I was thinking that it might be useful to restrict some tags to some types of records, many of them make sense only in some contexts (a person can be underage, vip, or deceased, a book can be missing or in need of repair). This kind of stuff could be kept in the central list, together with an explanation of the meaning of the tag etc. Probably not in the first version(s).

I have nothing against validating it only in the UI. It will need a pull-down/autocomplete list anyway. It may be possible that some mass-import routines will create new tags, it will be up to them to insert those in the central list if needed. At some point we may need special tools to check all tags, but that can wait well beyond v2.

Comment by Ann-Marie Breaux (Inactive) [ 11/Apr/18 ]

Hi guys,

Right now STSMACOM-113 Closed is linked to this backend work. Should UITAG-9 Closed (filtering by tags) and UITAG-18 Closed (tag permissions) also be linked to this? I still need to do some work on UITAG-18 Closed to break it into a few separate stories, but UITAG-9 Closed is ready, I think. And for that matter, do we want to build new project codes to get all of these stories out of UXPROD?

In regards to the deleting/marking as disabled - can we hold that part of the discussion until phase 2? I hope to be working on the phase 2 stories next week. We need to discuss a few more things in the tags subgroup first.

Thank you!

Comment by Jakub Skoczen [ 12/Apr/18 ]

Ann-Marie Breaux Yes, makes sense to link those stories here as well.

We can't really do any work on the tagging backend (which is prerequisite for all tagging implementation) unless we answer those questions and validate the assumptions. If it needs to wait until next week we are essentially blocked.

Here's what I propose though: validating deletion can be "faked" by searching if the tag is used in any of the modules. It's not trivial and it would need to include some additional functionality in Okapi to discover appropriate modules but can be done. With this assumption we can relax the consistency requirements (foreign key type of relationship between a tag and a record) and proceed with a solution where tags can be applied to records even if they are not in the central list. We can further limit this from the UI/client point of view and ensure that the tag is added to the list when it is applied (through a BL module or in the UI). So the only "side-effect" would be that records could be tagged directly through the API with tags that may not be included in the central list (subject to specific permissions) but exposure to this will be minimal. Makes sense?

Comment by Heikki Levanto [ 13/Apr/18 ]

In any case, it seems clear that we need some kind of mod-tags for the central list. We could as well start the stuff on Jira and GitHub already now, so we can have open issues and discuss the design details, and place to put the first docs.

We need to start by putting tags on one kind of objects. I was thinking of starting with tagging of users or items, but they both are a bit complex things already, and very critical to the functioning of the system. It might be better to use something simple and harmless as a testbed while developing. Maybe notes? How would that work on the UI side of things?

The internal representation of tags can almost not be anything else than an array of strings, no matter if we decide to allow free tags, or restrict it to be chosen from the central list. It looks like that would work well with our CQL querying, we have solved a similar problem earlier with language codes.

Comment by Jakub Skoczen [ 17/Apr/18 ]

Heikki Levanto Yep, let's start hacking on the backend, mod-tags, tags should be objects with UUIDs and possibility for attaching other metadata than labels:

POST /tags
GET|PUT|DELETE /tags/{id}

Tag.json
{
  "id" : "UIID",
   "label": "sprint36",
}

(we might consider using the labels as the ids, but I would vote against that for the sake of consistency)

As far as the candidate records for tagging are concerned, I would start with Users. It's a regular endpoint with a regular UI (SearchAndSot) so the integration will be fairly representative. As discussed, we should define a new schema for tagging:

tagged.json

"tags": []

Comment by Ann-Marie Breaux (Inactive) [ 17/Apr/18 ]

Hi all,

I'm sorry to be delayed replying to various comments on this. Will try to address all of the outstanding ones today, plus get the permissions stories for phase 1 finalized.

Heikki Levanto - thank you for picking up the work! If it would help to talk through any of this, let's plan a bit of time this week.

Which records: Ideally we want tags to apply to any "records" in FOLIO - we were trying to figure out how to define "records" - whether it would be anything that has a UUID, or can be viewed in a screen of metadata. This would include transactions, which may not be considered records, but do have a certain batch of metadata associated with them and can be displayed as an endpoint. Ultimately, in phase 2, each tag will have a little record in the central tag app, with the name of the tag, space for an optional description, and possibly and active/inactive flag.

As for which records to start with, we want to use acquisitions as our proving ground, as explained in STSMACOM-113 Closed . We would like to start with Orders, but that's not ready yet. Second choice is Vendors, which went into production last week. Third choice is one of the inventory record types: instance, item, or holdings (in that order).

Comment by Ann-Marie Breaux (Inactive) [ 17/Apr/18 ]

Jakub Skoczen Thank you for the additional comments about deleting tags. It will be helpful to talk through this a bit more when we get to Phase 2. Hopefully I'll have those user stories for you next week.

I had considered tags being adding to records (like orders or items) via MARC batch import, or API, but hadn't thought about the fact that the incoming data could be adding tags not already in the central tag list. Definitely something we'll need to consider in the MARC batch load work. For those incoming tags, we're intending that they will be in one field or string, separated by a pipe (|). The GOBI Library Solutions folks are starting work on some changes to GOBI to allow for this type of pipe-separated output within a MARC field/subfield, and we'll need to be able to consume it once we start parsing MARC records and consuming them in detail via the MARC batch loader. For that reason, I'm guessing we should add pipe to [space] as the only two characters disallowed from being in a tag itself. Does that sound correct?

Comment by Ann-Marie Breaux (Inactive) [ 17/Apr/18 ]

Heikki Levanto

Heikki: "I think it is fine to mark a tag as disabled. We might even want two markers, one to disable assigning the tag to any object, and one for disabling search/filtering by that tag. Philosophically, the central list will then be more like a list of tags the users may use, rather than a fully controlled vocabulary of all allowed tag values."
I'm not sure about this - we want to keep it as simple as possible. Either you can assign the tag or not. Ideally if the tag is not to be used at all, on any record (e.g. a typo, or a "duplicate" (new_york vs. new-york), then you could reassign all the records using the wrong tag and delete it from the central tag list. That is intended as Phase 2 functionality, which we're trying to finalize this week.

Heikki: "I was thinking that it might be useful to restrict some tags to some types of records, many of them make sense only in some contexts (a person can be underage, vip, or deceased, a book can be missing or in need of repair). This kind of stuff could be kept in the central list, together with an explanation of the meaning of the tag etc. Probably not in the first version(s)."
Definitely not in first versions. Users have talked about different tag lists/validation lists for different apps. I think the simpler we can keep it, the better. Presumably, I would never have cause to use "vip" or "deceased" unless I was talking about users. On the other hand, those values might be useful if I were tagging gift funds from specific donors. Since this is very new functionality in the context of a library system, the fewer restrictions we put on tags, the better. And so long as I can tell which types of records a tag is being used on (deceased used 154 times on user records, 6 times on fund records), then that gives context to the use. In phase 2, where we further define the tag record itself, we definitely do want a description field, where the creator of a tag can explain their intent.

Heikki: "I have nothing against validating it only in the UI. It will need a pull-down/autocomplete list anyway. It may be possible that some mass-import routines will create new tags, it will be up to them to insert those in the central list if needed. At some point we may need special tools to check all tags, but that can wait well beyond v2."
And thank you for reminding me about the mass import bit. I'll be working on that in my MARC batch loader PO role, so we'll definitely plan to address it there.

Comment by Ann-Marie Breaux (Inactive) [ 17/Apr/18 ]

shale99

"read through STSMACOM-113 Closed , i did not see any search functionality on tags other then autocomplete. did i miss something? is there a search requirement that tags can be searched and that the search on tags can be combined with other fields in a record for filtering?"

Not in phase 1. Within a particular app, you'll be able to filter by tags, but not search by them (unless, I guess, there's a "search all" option that takes into account every field in a record). In phase 2 (requirements being finalized this week), the intent is being able to search for tags from within a central tag app, both to find that tag in the central tag list and to see how/where that tag is being used in records in various apps.

Comment by Heikki Levanto [ 18/Apr/18 ]

Ann-Marie Breaux Just a comment about restricting tags to different types of objects. I agree there is no need for that in the first version. But later, when a library has accumulated hundreds of tags, it may make sense to have the pull-down list show the most relevant tags first, and there the kind of object may be a good indicator of what is relevant. (Also, on a pure technical side, it would be helpful to have some ideas about what kind of objects to check when looking if a tag is in use, etc). But all this can wait.

Comment by Heikki Levanto [ 02/May/18 ]

mod-tags v0.1.0 released today. If there are bugs or missing features, please open new issues in MODTAGS

Generated at Thu Feb 08 23:11:31 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.