[FOLIO-2445] SPIKE: Preceding/succeeding titles data model Created: 04/Feb/20 Updated: 03/Jun/20 Resolved: 24/Feb/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | P2 |
| Reporter: | Cate Boerema (Inactive) | Assignee: | Svitlana Zmiivska (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | back-end | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||
| Sprint: | Core: F - Sprint 82, Core: F - Sprint 81 | ||||||||||||||||||||||||||||||||
| Development Team: | Prokopovych | ||||||||||||||||||||||||||||||||
| Description |
|
Goal is to explore alternate data models so we can compare
|
| Comments |
| Comment by Marc Johnson [ 05/Feb/20 ] |
Is this the hybrid approach where we have two separate models for connected and unconnected, or where we weaken instance relationships so that only the parent or the child is required (which, as discussed, would make it easier for relationships to be broken / misdefined accidentally)? |
| Comment by Cate Boerema (Inactive) [ 05/Feb/20 ] |
|
I was thinking hybrid, but do you think it would make sense to actually compare all three options, outlining the pros and cons of each (including estimated size)? |
| Comment by Marc Johnson [ 05/Feb/20 ] |
I mostly wanted to clarify what you meant. I think we could explore all three options. I think we mostly need to decide which trade offs of those two options we are willing to take. I'll try to write something up here and we can go from there. |
| Comment by Cate Boerema (Inactive) [ 05/Feb/20 ] |
|
Thank you Marc Johnson! |
| Comment by Marc Johnson [ 06/Feb/20 ] |
OptionsInstance Relationships for Connected and UnconnectedApproachMake both parent and child instance optional for relationships and introduce new fields for the unconnected properties for either parent or child. Characteristics
Two Models for Connected and UnconnectedApproachUse instance relationships as is for connected preceding or succeeding titles. Use a new model specifically for unconnected. Characteristics
Single New Model for Connected and UnconnectedApproachStop using instance relationships for connected preceding or succeeding titles. Use a new model for both connected and unconnected. Characteristics
|
| Comment by Svitlana Zmiivska (Inactive) [ 11/Feb/20 ] |
|
The spike result can be found in the attachment |
| Comment by Marc Johnson [ 11/Feb/20 ] |
|
Svitlana Zmiivska Thank you. Please can the output of the spike be converted to a format that more folks can readily access (maybe comments here, or a page on the wiki) |
| Comment by Svitlana Zmiivska (Inactive) [ 11/Feb/20 ] |
|
Instance Relationships for Connected and Unconnected Steps to implement:
"title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ]
{
"precedingTitles": [
{
"id": "3b9592cc-55b1-4f19-89e2-d5bf484039b2",
"subInstanceId": "5bf370e0-8cca-4d9c-82e4-5170ab2a0a39",
"superInstanceId": "69640328-788e-43fc-9c3c-af39e243f3b7",
"instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990"
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
},
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "9780262012102"
}
]
}
],
"succeedingTitles": [
{
"id": "3b9592cc-55b1-4f19-89e2-d5bf484039b2",
"subInstanceId": "5bf370e0-8cca-4d9c-82e4-5170ab2a0a39",
"superInstanceId": "69640328-788e-43fc-9c3c-af39e243f3b7",
"instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990"
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
},
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "9780262012102"
}
]
}
]
}
Estimation: mod-inventory-storage: 3 story point Two Models for Connected and Unconnected Steps to implement:
{
"precedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
],
"succeedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
]
}
For connected titles (the same as we’ve already had):
{
"precedingTitles": [
{
"id": "3b9592cc-55b1-4f19-89e2-d5bf484039b2",
"subInstanceId": "5bf370e0-8cca-4d9c-82e4-5170ab2a0a39",
"superInstanceId": "69640328-788e-43fc-9c3c-af39e243f3b7",
"instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990"
}
],
"succeedingTitles": [
{
"id": "3b9592cc-55b1-4f19-89e2-d5bf484039b2",
"subInstanceId": "5bf370e0-8cca-4d9c-82e4-5170ab2a0a39",
"superInstanceId": "69640328-788e-43fc-9c3c-af39e243f3b7",
"instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990"
}
]
}
For unconnected titles:
{
"precedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
],
"succeedingTitles": [
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
]
}
Estimation: Single New Model for Connected and Unconnected Steps to implement:
{
"precedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
],
"succeedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
]
}
{
"precedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
],
"succeedingTitles": [
{
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
]
}
Estimation: mod-inventory-storage: 5 story point |
| Comment by Marc Johnson [ 11/Feb/20 ] |
|
Svitlana Zmiivska Thank you for providing your thoughts in the comments. I'm not sure I'm following all of the details of this investigation.
Why would the instance relationship type need to be optional? It seems that the proposed changes for the two models and the single new model are the same. Is that the case? This design suggests that the intention is to store the preceding and succeeding titles for each instance in the instance record? Is that a reasonable understanding? If it is, how would this model handle a change to a connected preceding/succeeding title? Would the instance records for both be changed, e.g. for the instance representing the succeeding title, will the other instance be removed from the preceding instances, and vice versa? Where in this system will this coordination of changes happen? |
| Comment by Svitlana Zmiivska (Inactive) [ 12/Feb/20 ] |
|
Marc Johnson Thank you for your questions. To save unconnected titles. 2) It seems that the proposed changes for the two models and the single new model are the same. Is that the case? I mean:
Single New Model for Connected and Unconnected connected and unconnected titles will be stored in an instance record in mod-inventory-storage (Now I see that the Single New Model should be changed. I didn't take into account that if a linked instance was changed that connected titles should also be changed. I relied on a similar implementation in PO Line, and this implementation turned out not so similar as I expected 3)This design suggests that the intention is to store the preceding and succeeding titles for each instance in the instance record? Is that a reasonable understanding? Yes, it is. The functionality of connected/unconnected titles is based loosely on a similar implementation in PO Lines. 4) If it is, how would this model handle a change to a connected preceding/succeeding title? If connected preceding/succeeding title fields (subInstanceId, superInstanceId and instanceRelationshipTypeId) exist, we will be able to get linked instance id 5)Would the instance records for both be changed, e.g. for the instance representing the succeeding title, Could you be more specific? Do you mean Scenario 5 in the story https://folio-org.atlassian.net/browse/UIIN-960 |
| Comment by Marc Johnson [ 12/Feb/20 ] |
In my opinion, it is still a preceding / succeeding title relationship even if one end of the relationship is unconnected.
Ok, I think there are some key differences between how PO lines are related to instances and preceding / succeeding title relationships that might mean the models need to be different.
Does that make sense? Cate Boerema Charlotte Whitt Do you agree with those differences?
I think at the storage layer, this likely needs to be modelled as a many to many relationship. The complexity of managing those relationships likely needs to be hidden in the business logic API. Zak Burke Michal Kuklis Matt Connolly I imagine the UI expectations is that an instance will have a set of preceding titles and a set of succeeding titles (as this is nicest for presentation)? And the addition or removal of sub-records from these collections should add or remove a relationship behind the scenes (and affect the views of both instances for a connected relationship)? (Apologies, I've run out of time before more meetings. I hope these brief thoughts help. I'll try to reply better later or tomorrow) |
| Comment by Marc Johnson [ 12/Feb/20 ] |
These questions were intended to raise awareness of a challenge that I think you had already identified above. If the preceding / succeeding titles are embedded in each instance, if an entry from a preceding title set is removed from one instance, should the corresponding record to be removed from the succeeding title set in the other instance for a connected relationship. |
| Comment by Svitlana Zmiivska (Inactive) [ 18/Feb/20 ] |
|
Single New Model for Connected and Unconnected mod-inventory-storage 1)create table preceding-succeeding-titles with fields:
id- required 2)create crud endpoints
POST /instance-storage/preceding-succeeding-titles Request body
{
"id": "4b8093b8-8de9-41be-ad9a-0fbccb99052b",
"superInstanceId": "601a8dc4-dee7-48eb-b03f-d02fdf0debd0",
"subInstanceId": "83a50dc6-b887-43d9-93ee-28b2c4cd11f8",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
GET /instance-storage/preceding-succeeding-titles Request parameters:
Response:
{
"precedingSucceedingTitles": [
{
"id": "4b8093b8-8de9-41be-ad9a-0fbccb99052b",
"superInstanceId": "601a8dc4-dee7-48eb-b03f-d02fdf0debd0",
"subInstanceId": "83a50dc6-b887-43d9-93ee-28b2c4cd11f8",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
],
"totalRecords": 1
}
PUT /instance-storage/preceding-succeeding-titles/ {precedingSucceedingTitleId}Request body
{
"superInstanceId": "601a8dc4-dee7-48eb-b03f-d02fdf0debd0",
"subInstanceId": "83a50dc6-b887-43d9-93ee-28b2c4cd11f8",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
DELETE /instance-storage/preceding-succeeding-titles/{precedingSucceedingTitleId} mod-inventory 1)Change preceding and succeeding titles fields in instance endpoint to POST /inventory/instances … "precedingTitles": [ { "id": "cb4f8537-7643-4ef4-8cc4-6dbf7bbe8a6b", "superInstanceId": "1b74ab75-9f41-4837-8662-a1d99118008d", "title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ] } ], "succeedingTitles": [ { "id": "1b4f8537-7643-4ef4-8cc4-6dbf7bbe8a6b", "subInstanceId": "8be05cf5-fb4f-4752-8094-8e179d08fb99", "title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ] } ] … 2) change behavior that preceding and succeeding titles are created, updated, read and deleted using the new endpoints in mod-inventory storage /preceding-succeeding-titles instead of /instance-storage/instance-relationships 3) get title, hrid and identifier for connected preceding and succeeding titles when return an instance(can be done on UI side) Instance Relationships for Connected and Unconnected mod-inventory-storage 1)add fields title, hrid, identifier to instance_relationship table 2)do superInstanceId and subInstanceId fields optional mod-inventory 1)Change preceding and succeeding titles fields in instance endpoint to POST host/inventory/instances GET host/inventory/instances PUT host/inventory/instances/{instanceId} … "precedingTitles": [ { "id": "cb4f8537-7643-4ef4-8cc4-6dbf7bbe8a6b", "superInstanceId": "1b74ab75-9f41-4837-8662-a1d99118008d", "instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990" "title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ] } ], "succeedingTitles": [ { "id": "1b4f8537-7643-4ef4-8cc4-6dbf7bbe8a6b", "subInstanceId": "8be05cf5-fb4f-4752-8094-8e179d08fb99", "instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990" "title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ] } ] … 2)do superInstanceId and subInstanceId fields optional Two Models for Connected and Unconnected mod-inventory-storage 1)create table unconnected-preceding-succeeding-titles
id, title - required 2)create crud endpoints
{
"id": "4b8093b8-8de9-41be-ad9a-0fbccb99052b",
"superInstanceId": "601a8dc4-dee7-48eb-b03f-d02fdf0debd0",
"subInstanceId": "83a50dc6-b887-43d9-93ee-28b2c4cd11f8",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
GET /instance-storage/ unconnected-preceding-succeeding-titles Request parameters
Response:
{
"precedingSucceedingTitles": [
{
"id": "4b8093b8-8de9-41be-ad9a-0fbccb99052b",
"superInstanceId": "601a8dc4-dee7-48eb-b03f-d02fdf0debd0",
"subInstanceId": "83a50dc6-b887-43d9-93ee-28b2c4cd11f8",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
],
"totalRecords": 1
}
PUT /instance-storage/unconnected-preceding-succeeding-titles/ {precedingSucceedingTitleId}Request body
{
"superInstanceId": "601a8dc4-dee7-48eb-b03f-d02fdf0debd0",
"subInstanceId": "83a50dc6-b887-43d9-93ee-28b2c4cd11f8",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
DELETE /instance-storage/unconnected-preceding-succeeding titles/{precedingSucceedingTitleId} mod-inventory 1)Change preceding and succeeding titles fields in instance endpoint to … "precedingTitles": [ { "id": "cb4f8537-7643-4ef4-8cc4-6dbf7bbe8a6b", "superInstanceId": "1b74ab75-9f41-4837-8662-a1d99118008d", "instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990" "title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ] } ], "succeedingTitles": [ { "id": "1b4f8537-7643-4ef4-8cc4-6dbf7bbe8a6b", "subInstanceId": "8be05cf5-fb4f-4752-8094-8e179d08fb99", "instanceRelationshipTypeId": "cde80cc2-0c8b-4672-82d4-721e51dcb990" "title": "A semantic web primer", "hrid": "inst000000000022", "identifiers": [ { "identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422", "value": "0262012103" } ] } ] … 2) change behavior that |
| Comment by Svitlana Zmiivska (Inactive) [ 18/Feb/20 ] |
|
Marc Johnson Could you please take a look on the possible implementations? |
| Comment by Marc Johnson [ 18/Feb/20 ] |
|
Svitlana Zmiivska Thank you for your additional investigation. Has this further investigation changed your thoughts about the relative size of the work? I'm going to focus on the new model for connected and unconnected preceding / succeeding titles, because I think that is much the same as the hybrid and is of most interest.
Given that this model is specific to preceding and succeeding titles, I think we can refer to these more specifically, e.g. precedingInstanceId and succeedingInstanceId.
By null, do you mean that the properties will be omitted from the records? And is that only one of them can be omitted, or both can be?
I think this is the aspect I am most interested in. Are you suggesting that mod-inventory will compare the preceding / succeeding titles it already has, with the ones received from the client, and use that to delete, create or replace instance relationships in storage?
What do you mean by done on the UI side? Does that mean these won't be present in the API response for connected preceding / succeeding titles? Does it matter to the client if a preceding / succeeding title is connected or unconnected? Does it change their interaction with the records? |
| Comment by Bohdan Suprun (Inactive) [ 18/Feb/20 ] |
|
Hi Svitlana Zmiivska, Marc Johnson, For the separate model I think we should also take into account data migration (records in instance-relationships table have to be imported to the new table). As far as I remember we don't have separate properties in instance for preceding/succeeding but parent/child properties are used instead. Are we going to add separate properties for them now? If yes, that's going to be a breaking change I think, at least for data-import. What do you think? |
| Comment by Svitlana Zmiivska (Inactive) [ 18/Feb/20 ] |
|
Marc Johnson Only one of them can be omitted superInstanceId or subInstanceId.
Ex. Create unconnected preceding titles
{
"id": "4b8093b8-8de9-41be-ad9a-0fbccb99052b",
"superInstanceId": "",
"subInstanceId": "5bf370e0-8cca-4d9c-82e4-5170ab2a0a39",
"title": "A semantic web primer",
"hrid": "inst000000000022",
"identifiers": [
{
"identifierTypeId": "8261054f-be78-422d-bd51-4ed9f33c3422",
"value": "0262012103"
}
]
}
Preceding unconnected titles can be received using |
| Comment by Svitlana Zmiivska (Inactive) [ 18/Feb/20 ] |
What do you mean by done on the UI side? Does that mean these won't be present in the API response for connected preceding / succeeding titles? It seems to me the best way is to use the same API response for connected preceding / succeeding titles. |
| Comment by Marc Johnson [ 18/Feb/20 ] |
Ok, you were suggesting that doing it in the client was another alternative? I agree this is better done transparently in mod-inventory. It does mean that requests for connected preceding or succeeding titles should ignore those properties and not store them. |
| Comment by Marc Johnson [ 18/Feb/20 ] |
I agree, we do need to consider migrating connected preceding / succeeding titles from instance relationships to the new model.
Yes, that is my understanding of the option where we introduce a wholly separate model for preceding / succeeding titles.
How is this a breaking change? Do you mean that if data-import were creating instance relationships for preceding / succeeding titles, they need to stop and move to the new model? |
| Comment by Bohdan Suprun (Inactive) [ 18/Feb/20 ] |
|
Persisting of parent/child relationships is executed within instance-batch API, which is used by data-import I think. But it worth asking them if they are using preceding-succeeding relationship type. |
| Comment by Svitlana Zmiivska (Inactive) [ 19/Feb/20 ] |
|
Hi Oleksii Kuzminov, Ruslan Lavrov Could you please help with the following questions: |
| Comment by Svitlana Zmiivska (Inactive) [ 19/Feb/20 ] |
|
To sum up, we’ve chosen the new model for connected and unconnected preceding / succeeding titles. mod-inventory-storage Story point 8 mod-inventory-storage Story point 5 mod-inventory (mod-inventory will compare the preceding / succeeding titles it already has, with the ones received from the client, and use that to delete, create or replace preceding succeeding titles in storage) Story point 8 |
| Comment by Svitlana Zmiivska (Inactive) [ 19/Feb/20 ] |
|
Marc Johnson Bohdan Suprun The preceding-succeeding relationship type isn't used by data-import (confirmed by Ruslan Lavrov). |
| Comment by Svitlana Zmiivska (Inactive) [ 19/Feb/20 ] |
|
Marc Johnson Bohdan Suprun Do you have any other questions or suggestions? |
| Comment by Marc Johnson [ 19/Feb/20 ] |
I'm not sure if that decision has been made. I thought Cate Boerema had commented to suggest that if the work wasn't significantly larger then we go with the new model. Given that Cate Boerema is away, if we don't think this is significantly more work, I suggest we start with the development work on the new model. Svitlana Zmiivska Do you agree with that position, that it isn't significantly more work? |
| Comment by Svitlana Zmiivska (Inactive) [ 19/Feb/20 ] |
|
Marc Johnson Yes, I do. I think I can start. Should I create stories? |
| Comment by Marc Johnson [ 19/Feb/20 ] |
Please check if there are any existing stories already. If not, please create stories as needed. |