2022-10-19 Meeting notes: Cross-app data sync update and outcomes

Date

19 Oct 2022

Housekeeping

• Convener and notes: Martina Schildt
• Next meeting: Oct 24th | topic: UX patterns for common cross-app tasks

Discussion items

1. • Cross-app data sync update and outcomes of working group Owen Stephens 
   • AI Slack channel: open vs closed - if time allows

Minutes

• working group has been working to find a common pattern for data sync use cases that have been collected in this group (and other groups)
• Raman has written a proposal based on Kafka < does not solve all given use cases, but most of them
• approach: DR-000004 - Cross application data sync proposal
• approach was presented to the TC | recorded as ADR = Architecture Decision Record
• TC recommends this approach | use of Kafka to keep data consistency
• it was a long way around approving a practice that is already in use in FOLIO
• new pattern: inbox/outbox pattern | audit whether changes are read by subscriber | TC did not adopt that because it has not been used so far
• it seems as if teams just need to do something to have it approved by TC | we need to discuss whether this is a good way
• Brooks in chat: Not having outbox also means that you could encounter a situation where a record is not updated in the database, but an update event is published and consumed… | Charlotte agrees
• Owen & Marc: what we gain with this approach is better than what we have before
• Brooks agrees
• Marc in chat: Whilst the outbox pattern helps, it is not sufficient to guarantee the consistent state (which is generally very hard to achieve)
• E.g.: the Circ log app not knowing about an item being deleted: UXPROD-3738 Circulation log update: deleted item records
  
  • Marc in chat: You can mitigate/minimize that possibility…
    Not sure how that applies here…
  • Marc: all of the technical background stuff is in place (since the search engine based search work)
• ADR gives us a way of how changes can be communicated
• Up to POs and dev teams to decide when it is time to do this
• to keep in mind: this approach does not give us one way
• consumer is more interested in keeping data in sync than the producer | this is a challenge | need to talk about prioritization | apps are dependant on development on both sides | this is not solved through this solution
• what can users expect from this solution
  • data update not in real time | most likely not very delayed (timely, but not instantaneous)
  • cannot know definitely | will depend on lots of variables
  • Marc in chat: For folks who want to understand the rough expectations of these designs, they can consider the search in inventory. For example, folks know there is a discrepancy when returning to a search immediately after editing a record.
  • Dung-Lan in chat: What people can expect should be made as clear as possible to reduce confusion or frustration | the group agrees
• idea: expose to the user whether looking at original data or copy of data → discuss in this group in a future meeting
• new pattern is in use for communication between Inventory and Elastic Search | Marc will check and give more information
• possible next step: find dev team to start using the new approach for one of our use cases | maybe modorders-642 would be a good starting point | we need Dennis to decide on this | discuss in PO meeting? | not needed, promotion is sufficient
• Next steps: 
  • we need to promote these things are unblocked | PO meeting
  • feedback somehow the frustrations of the process | maybe in PC with TC
  • present approach and what to expect in PC call with SIG conveners are present (SIG convener updates)

Next steps

• we need to promote new approach | in PO meeting
• feedback somehow the frustrations of the process | maybe in PC with TC
• present approach and what to expect in PC call with SIG conveners are present (SIG convener updates)

Chat

00:02:33	Martina Schildt:	Agenda: https://wiki.folio.org/display/AppInt/2022-10-19+Meeting+notes%3A+Cross-app+data+sync+update+and+outcomes
00:15:50	Brooks Travis:	Circular
00:16:22	Maura Byrne:	Indeed.
00:17:14	Brooks Travis:	Not having outbox also means that you could encounter a situation where a record is not updated in the database, but an update event is published and consumed…
00:18:38	Index Data:	Yes, very frustrating
00:19:52	Index Data:	E.g.: the Circ log app not knowing about an item being deleted
00:19:59	Maura Byrne:	+1
00:20:16	Brooks Travis:	You can mitigate/minimize that possibility…
00:20:28	Brooks Travis:	Not sure how that applies here…
00:20:41	Martina Schildt:	I see you Marc
00:21:08	Brooks Travis:	Agreed
00:21:15	Marc Johnson:	Indeed. Whilst the outbox pattern helps, it is not sufficient to guarantee the consistent state (which is generally very hard to achieve)
00:21:51	Index Data:	https://wiki.folio.org/display/DD/UXPROD-3738+Circulation+log+update%3A+deleted+item+records
00:30:32	Dung-Lan Chen:	Link to the page Owen is sharing on the screen  - https://wiki.folio.org/display/TC/ADR-000004+-+Cross+application+data+sync+proposal
00:38:18	Brooks Travis:	This was one of the cases that precipitated Raman’s original proposal that became this approach.
00:38:35	Brooks Travis:	Dennis is unavailable today and tomorrow
00:38:44	Brooks Travis:	Probably on Friday.
00:38:51	Brooks Travis:	Next week
00:39:01	Owen Stephens:	Precipitated is a lovely word with several meanings 🙂But in this case ‘led to’
00:39:07	Brooks Travis:	I mean he’ll probably be unavailable until next week
00:45:03	Raman Auramau:	Need to hop off now.
00:45:14	Martina Schildt:	Thanks Raman
00:45:16	Maura Byrne:	+1 Marc
00:45:19	Owen Stephens:	Thanks Raman
00:48:28	Marc Johnson:	For folks who want to understand the rough expectations of these designs, they can consider the search in inventory.

For example, folks know there is a discrepancy when returning to a search immediately after editing a record.
00:49:26	Marc Johnson:	I should be clear, it’s not that nothing can be done, it’s that nothing has to be done beyond communicating this to POs and TLs
00:50:41	Dung-Lan Chen:	What people can expect should be made as clear as possible to reduce confusion or frustration.
00:51:00	Maura Byrne:	+1 Dung-Lan
00:51:17	Marc Johnson:	There is no trivial way to find the places where this approach is used in the system today
00:51:33	Owen Stephens:	Marc - do you know any situation?
00:51:52	Brooks Travis:	Kafka queue names?
00:52:14	Brooks Travis:	Mod-circulation publishes messages
00:52:42	Brooks Travis:	Loans, requests, and check-in records all generate events
00:53:19	Marc Johnson:	Inspecting Kafka in a running system could tell us this
00:53:27	Marc Johnson:	I have no access to such a system
00:53:47	Brooks Travis:	The primary consumer of those at the moment is mod-inn-reach, but circulation log could potentially switch from mod-pub sub for those records…
00:54:28	Brooks Travis:	Same for mod-patron-blocks
00:54:30	Owen Stephens:	I would like to feedback somehow the frustrations of the process but I'm not sure where to or whether we can expect that to be productive
00:55:11	Marc Johnson:	And guess who gets to be the temporary liaison for tomorrow 😃
00:55:38	Dung-Lan Chen:	+1 Kristin
00:58:38	Owen Stephens:	It's a good point
00:59:33	Owen Stephens:	I don't think we need to find a dev team ... we just need to promote these things are unblocked
01:00:33	Owen Stephens:	I can't see we really have a direct role except promoting the outcome
01:00:55	Marc Johnson:	I agree communicating that this mechanism is available is important
01:01:05	Index Data:	I need to drop off - sorry
01:01:18	Owen Stephens:	Thanks Charlotte
01:01:34	Owen Stephens:	Thanks Martina
01:01:41	Maura Byrne:	Thanks, Martina

Future topics

• Topic proposal by Owen Stephens for October:
  • UX patterns for common cross-app tasks (e.g. would it be useful to have a UX for a 'quick add' task that could be used in different contexts when you need to create something in another app). I think it would be good to have UX specialists (Kimie, Gill) present for this, and possibly also some Stripes devs (e.g. John C/Zak)
    • fast-add is a good model - Laura will demo 
    • Owen has some examples from invoice creation to talk about
    • Martina S. will check whether Gill and Kimie will be available
    • Date: Oct 24th
  • Use of shortcut keys and macros for more effective cross-app working  - it also be good to have UX and Stripes/dev knowledge for this discussion I think. I know @Laura (she/they) uses macros so might have insights into the potential for cross-app working
  • Potential for external 'workflow' solutions for cross-app interactions
    • I think 'workflow' is a dangerous term here - in this context it's more about automation than user workflows, although I think there is overlap
    • I was particularly struck by the solution in production at TAMU (Jeremy Huff and Sebastian Hammer presented, the recording is at https://prod-zoom-recordings-openlibraryfoundation-org.s3.amazonaws.com/50dc6c87-3912-43fa-8287-56ec73b12bbb%2Fshared_screen_with_speaker_view%28CC%29.mp4 starting at 3 hrs, 14 min) - I think getting someone from TAMU to talk about how this is used would be v interesting
    • There was also a presentation on the use of a tool called Airflow at Stanford for "bibliographic workflow" but I've not watched that yet so not 100% sure if it is completely applicable - I think the core use case there was systems migration but it may go beyond that
• UX/UI and implementers topics
  • should be Wednesdays

Attendees

Action items