...
- working group has been working to find a common pattern for data sync use cases that have been collected in this group (and other groups)
- Raman has written a proposal based on Kafka < does not solve all given use cases, but most of them
- approach: ADR DR-000004 - Cross application data sync proposal
- approach was presented to the TC | recorded as ADR = Architecture Decision Record
- TC recommends this approach | use of Kafka to keep data consistency
- it was a long way around approving a practice that is already in use in FOLIO
- new pattern: inbox/outbox pattern | audit whether changes are read by subscriber | TC did not adopt that because it has not been used so far
- it seems as if teams just need to do something to have it approved by TC | we need to discuss whether this is a good way
- Brooks in chat: Not having outbox also means that you could encounter a situation where a record is not updated in the database, but an update event is published and consumed… | Charlotte agrees
- Owen & Marc: what we gain with this approach is better than what we have before
- Brooks agrees
- Marc in chat: Whilst the outbox pattern helps, it is not sufficient to guarantee the consistent state (which is generally very hard to achieve)
- E.g.: the Circ log app not knowing about an item being deleted: UXPROD-3738 Circulation log update: deleted item records
- Marc in chat: You can mitigate/minimize that possibility…
Not sure how that applies here… - Marc: all of the technical background stuff is in place (since the search engine based search work)
- Marc in chat: You can mitigate/minimize that possibility…
- ADR gives us a way of how changes can be communicated
- Up to POs and dev teams to decide when it is time to do this
- to keep in mind: this approach does not give us one way
- consumer is more interested in keeping data in sync than the producer | this is a challenge | need to talk about prioritization | apps are dependant on development on both sides | this is not solved through this solution
- what can users expect from this solution
- data update not in real time | most likely not very delayed (timely, but not instantaneous)
- cannot know definitely | will depend on lots of variables
- Marc in chat: For folks who want to understand the rough expectations of these designs, they can consider the search in inventory. For example, folks know there is a discrepancy when returning to a search immediately after editing a record.
- Dung-Lan in chat: What people can expect should be made as clear as possible to reduce confusion or frustration | the group agrees
- idea: expose to the user whether looking at original data or copy of data → discuss in this group in a future meeting
- new pattern is in use for communication between Inventory and Elastic Search | Marc will check and give more information
- possible next step: find dev team to start using the new approach for one of our use cases | maybe modorders-642 would be a good starting point | we need Dennis to decide on this | discuss in PO meeting? | not needed, promotion is sufficient
- Next steps:
- we need to promote these things are unblocked | PO meeting
- feedback somehow the frustrations of the process | maybe in PC with TC
- present approach and what to expect in PC call with SIG conveners are present (SIG convener updates)
...
- we need to promote new approach | in PO meeting
- feedback somehow the frustrations of the process | maybe in PC with TC
- present approach and what to expect in PC call with SIG conveners are present (SIG convener updates)
Chat
00:02:33 Martina Schildt: Agenda: https://wiki.folio.org/display/AppInt/2022-10-19+Meeting+notes%3A+Cross-app+data+sync+update+and+outcomes 00:15:50 Brooks Travis: Circular 00:16:22 Maura Byrne: Indeed. 00:17:14 Brooks Travis: Not having outbox also means that you could encounter a situation where a record is not updated in the database, but an update event is published and consumed… 00:18:38 Index Data: Yes, very frustrating 00:19:52 Index Data: E.g.: the Circ log app not knowing about an item being deleted 00:19:59 Maura Byrne: +1 00:20:16 Brooks Travis: You can mitigate/minimize that possibility… 00:20:28 Brooks Travis: Not sure how that applies here… 00:20:41 Martina Schildt: I see you Marc 00:21:08 Brooks Travis: Agreed 00:21:15 Marc Johnson: Indeed. Whilst the outbox pattern helps, it is not sufficient to guarantee the consistent state (which is generally very hard to achieve) 00:21:51 Index Data: https://wiki.folio.org/display/DD/UXPROD-3738+Circulation+log+update%3A+deleted+item+records 00:30:32 Dung-Lan Chen: Link to the page Owen is sharing on the screen - https://wiki.folio.org/display/TC/ADR-000004+-+Cross+application+data+sync+proposal 00:38:18 Brooks Travis: This was one of the cases that precipitated Raman’s original proposal that became this approach. 00:38:35 Brooks Travis: Dennis is unavailable today and tomorrow 00:38:44 Brooks Travis: Probably on Friday. 00:38:51 Brooks Travis: Next week 00:39:01 Owen Stephens: Precipitated is a lovely word with several meanings 🙂But in this case ‘led to’ 00:39:07 Brooks Travis: I mean he’ll probably be unavailable until next week 00:45:03 Raman Auramau: Need to hop off now. 00:45:14 Martina Schildt: Thanks Raman 00:45:16 Maura Byrne: +1 Marc 00:45:19 Owen Stephens: Thanks Raman 00:48:28 Marc Johnson: For folks who want to understand the rough expectations of these designs, they can consider the search in inventory. For example, folks know there is a discrepancy when returning to a search immediately after editing a record. 00:49:26 Marc Johnson: I should be clear, it’s not that nothing can be done, it’s that nothing has to be done beyond communicating this to POs and TLs 00:50:41 Dung-Lan Chen: What people can expect should be made as clear as possible to reduce confusion or frustration. 00:51:00 Maura Byrne: +1 Dung-Lan 00:51:17 Marc Johnson: There is no trivial way to find the places where this approach is used in the system today 00:51:33 Owen Stephens: Marc - do you know any situation? 00:51:52 Brooks Travis: Kafka queue names? 00:52:14 Brooks Travis: Mod-circulation publishes messages 00:52:42 Brooks Travis: Loans, requests, and check-in records all generate events 00:53:19 Marc Johnson: Inspecting Kafka in a running system could tell us this 00:53:27 Marc Johnson: I have no access to such a system 00:53:47 Brooks Travis: The primary consumer of those at the moment is mod-inn-reach, but circulation log could potentially switch from mod-pub sub for those records… 00:54:28 Brooks Travis: Same for mod-patron-blocks 00:54:30 Owen Stephens: I would like to feedback somehow the frustrations of the process but I'm not sure where to or whether we can expect that to be productive 00:55:11 Marc Johnson: And guess who gets to be the temporary liaison for tomorrow 😃 00:55:38 Dung-Lan Chen: +1 Kristin 00:58:38 Owen Stephens: It's a good point 00:59:33 Owen Stephens: I don't think we need to find a dev team ... we just need to promote these things are unblocked 01:00:33 Owen Stephens: I can't see we really have a direct role except promoting the outcome 01:00:55 Marc Johnson: I agree communicating that this mechanism is available is important 01:01:05 Index Data: I need to drop off - sorry 01:01:18 Owen Stephens: Thanks Charlotte 01:01:34 Owen Stephens: Thanks Martina 01:01:41 Maura Byrne: Thanks, Martina
Future topics
- Topic proposal by Owen Stephens for October:
- UX patterns for common cross-app tasks (e.g. would it be useful to have a UX for a 'quick add' task that could be used in different contexts when you need to create something in another app). I think it would be good to have UX specialists (Kimie, Gill) present for this, and possibly also some Stripes devs (e.g. John C/Zak)
- fast-add is a good model - Laura will demo
- Owen has some examples from invoice creation to talk about
- Martina S. will check whether Gill and Kimie will be available
- Date: Oct 24th
- Use of shortcut keys and macros for more effective cross-app working - it also be good to have UX and Stripes/dev knowledge for this discussion I think. I know @Laura (she/they) uses macros so might have insights into the potential for cross-app working
- Potential for external 'workflow' solutions for cross-app interactions
- I think 'workflow' is a dangerous term here - in this context it's more about automation than user workflows, although I think there is overlap
- I was particularly struck by the solution in production at TAMU (Jeremy Huff and Sebastian Hammer presented, the recording is at https://prod-zoom-recordings-openlibraryfoundation-org.s3.amazonaws.com/50dc6c87-3912-43fa-8287-56ec73b12bbb%2Fshared_screen_with_speaker_view%28CC%29.mp4 starting at 3 hrs, 14 min) - I think getting someone from TAMU to talk about how this is used would be v interesting
- There was also a presentation on the use of a tool called Airflow at Stanford for "bibliographic workflow" but I've not watched that yet so not 100% sure if it is completely applicable - I think the core use case there was systems migration but it may go beyond that
- UX patterns for common cross-app tasks (e.g. would it be useful to have a UX for a 'quick add' task that could be used in different contexts when you need to create something in another app). I think it would be good to have UX specialists (Kimie, Gill) present for this, and possibly also some Stripes devs (e.g. John C/Zak)
- UX/UI and implementers topics
- should be Wednesdays
...