...
- Cross-app data sync update and outcomes of working group Owen Stephens
- AI Slack channel: open vs closed - if time allows
Minutes
- working group has been working to find a common pattern for data sync use cases that have been collected in this group (and other groups)
- Raman has written a proposal based on Kafka < does not solve all given use cases, but most of them
- approach: DR-000004 - Cross application data sync proposal
- approach was presented to the TC | recorded as ADR = Architecture Decision Record
- TC recommends this approach | use of Kafka to keep data consistency
- it was a long way around approving a practice that is already in use in FOLIO
- new pattern: inbox/outbox pattern | audit whether changes are read by subscriber | TC did not adopt that because it has not been used so far
- it seems as if teams just need to do something to have it approved by TC | we need to discuss whether this is a good way
- Brooks in chat: Not having outbox also means that you could encounter a situation where a record is not updated in the database, but an update event is published and consumed… | Charlotte agrees
- Owen & Marc: what we gain with this approach is better than what we have before
- Brooks agrees
- Marc in chat: Whilst the outbox pattern helps, it is not sufficient to guarantee the consistent state (which is generally very hard to achieve)
- E.g.: the Circ log app not knowing about an item being deleted: UXPROD-3738 Circulation log update: deleted item records
- Marc in chat: You can mitigate/minimize that possibility…
Not sure how that applies here… - Marc: all of the technical background stuff is in place (since the search engine based search work)
- Marc in chat: You can mitigate/minimize that possibility…
- ADR gives us a way of how changes can be communicated
- Up to POs and dev teams to decide when it is time to do this
- to keep in mind: this approach does not give us one way
- consumer is more interested in keeping data in sync than the producer | this is a challenge | need to talk about prioritization | apps are dependant on development on both sides | this is not solved through this solution
- what can users expect from this solution
- data update not in real time | most likely not very delayed (timely, but not instantaneous)
- cannot know definitely | will depend on lots of variables
- Marc in chat: For folks who want to understand the rough expectations of these designs, they can consider the search in inventory. For example, folks know there is a discrepancy when returning to a search immediately after editing a record.
- Dung-Lan in chat: What people can expect should be made as clear as possible to reduce confusion or frustration | the group agrees
- idea: expose to the user whether looking at original data or copy of data → discuss in this group in a future meeting
- new pattern is in use for communication between Inventory and Elastic Search | Marc will check and give more information
- possible next step: find dev team to start using the new approach for one of our use cases | maybe modorders-642 would be a good starting point | we need Dennis to decide on this | discuss in PO meeting? | not needed, promotion is sufficient
- Next steps:
- we need to promote these things are unblocked | PO meeting
- feedback somehow the frustrations of the process | maybe in PC with TC
- present approach and what to expect in PC call with SIG conveners are present (SIG convener updates)
Next steps
- we need to promote new approach | in PO meeting
- feedback somehow the frustrations of the process | maybe in PC with TC
- present approach and what to expect in PC call with SIG conveners are present (SIG convener updates)
Chat
00:02:33 Martina Schildt: Agenda: https://wiki.folio.org/display/AppInt/2022-10-19+Meeting+notes%3A+Cross-app+data+sync+update+and+outcomes
00:15:50 Brooks Travis: Circular
00:16:22 Maura Byrne: Indeed.
00:17:14 Brooks Travis: Not having outbox also means that you could encounter a situation where a record is not updated in the database, but an update event is published and consumed…
00:18:38 Index Data: Yes, very frustrating
00:19:52 Index Data: E.g.: the Circ log app not knowing about an item being deleted
00:19:59 Maura Byrne: +1
00:20:16 Brooks Travis: You can mitigate/minimize that possibility…
00:20:28 Brooks Travis: Not sure how that applies here…
00:20:41 Martina Schildt: I see you Marc
00:21:08 Brooks Travis: Agreed
00:21:15 Marc Johnson: Indeed. Whilst the outbox pattern helps, it is not sufficient to guarantee the consistent state (which is generally very hard to achieve)
00:21:51 Index Data: https://wiki.folio.org/display/DD/UXPROD-3738+Circulation+log+update%3A+deleted+item+records
00:30:32 Dung-Lan Chen: Link to the page Owen is sharing on the screen - https://wiki.folio.org/display/TC/ADR-000004+-+Cross+application+data+sync+proposal
00:38:18 Brooks Travis: This was one of the cases that precipitated Raman’s original proposal that became this approach.
00:38:35 Brooks Travis: Dennis is unavailable today and tomorrow
00:38:44 Brooks Travis: Probably on Friday.
00:38:51 Brooks Travis: Next week
00:39:01 Owen Stephens: Precipitated is a lovely word with several meanings 🙂But in this case ‘led to’
00:39:07 Brooks Travis: I mean he’ll probably be unavailable until next week
00:45:03 Raman Auramau: Need to hop off now.
00:45:14 Martina Schildt: Thanks Raman
00:45:16 Maura Byrne: +1 Marc
00:45:19 Owen Stephens: Thanks Raman
00:48:28 Marc Johnson: For folks who want to understand the rough expectations of these designs, they can consider the search in inventory.
For example, folks know there is a discrepancy when returning to a search immediately after editing a record.
00:49:26 Marc Johnson: I should be clear, it’s not that nothing can be done, it’s that nothing has to be done beyond communicating this to POs and TLs
00:50:41 Dung-Lan Chen: What people can expect should be made as clear as possible to reduce confusion or frustration.
00:51:00 Maura Byrne: +1 Dung-Lan
00:51:17 Marc Johnson: There is no trivial way to find the places where this approach is used in the system today
00:51:33 Owen Stephens: Marc - do you know any situation?
00:51:52 Brooks Travis: Kafka queue names?
00:52:14 Brooks Travis: Mod-circulation publishes messages
00:52:42 Brooks Travis: Loans, requests, and check-in records all generate events
00:53:19 Marc Johnson: Inspecting Kafka in a running system could tell us this
00:53:27 Marc Johnson: I have no access to such a system
00:53:47 Brooks Travis: The primary consumer of those at the moment is mod-inn-reach, but circulation log could potentially switch from mod-pub sub for those records…
00:54:28 Brooks Travis: Same for mod-patron-blocks
00:54:30 Owen Stephens: I would like to feedback somehow the frustrations of the process but I'm not sure where to or whether we can expect that to be productive
00:55:11 Marc Johnson: And guess who gets to be the temporary liaison for tomorrow 😃
00:55:38 Dung-Lan Chen: +1 Kristin
00:58:38 Owen Stephens: It's a good point
00:59:33 Owen Stephens: I don't think we need to find a dev team ... we just need to promote these things are unblocked
01:00:33 Owen Stephens: I can't see we really have a direct role except promoting the outcome
01:00:55 Marc Johnson: I agree communicating that this mechanism is available is important
01:01:05 Index Data: I need to drop off - sorry
01:01:18 Owen Stephens: Thanks Charlotte
01:01:34 Owen Stephens: Thanks Martina
01:01:41 Maura Byrne: Thanks, Martina
Future topics
- Topic proposal by Owen Stephens for October:
- UX patterns for common cross-app tasks (e.g. would it be useful to have a UX for a 'quick add' task that could be used in different contexts when you need to create something in another app). I think it would be good to have UX specialists (Kimie, Gill) present for this, and possibly also some Stripes devs (e.g. John C/Zak)
- fast-add is a good model - Laura will demo
- Owen has some examples from invoice creation to talk about
- Martina S. will check whether Gill and Kimie will be available
- Date: Oct 24th
- Use of shortcut keys and macros for more effective cross-app working - it also be good to have UX and Stripes/dev knowledge for this discussion I think. I know @Laura (she/they) uses macros so might have insights into the potential for cross-app working
- Potential for external 'workflow' solutions for cross-app interactions
- I think 'workflow' is a dangerous term here - in this context it's more about automation than user workflows, although I think there is overlap
- I was particularly struck by the solution in production at TAMU (Jeremy Huff and Sebastian Hammer presented, the recording is at https://prod-zoom-recordings-openlibraryfoundation-org.s3.amazonaws.com/50dc6c87-3912-43fa-8287-56ec73b12bbb%2Fshared_screen_with_speaker_view%28CC%29.mp4 starting at 3 hrs, 14 min) - I think getting someone from TAMU to talk about how this is used would be v interesting
- There was also a presentation on the use of a tool called Airflow at Stanford for "bibliographic workflow" but I've not watched that yet so not 100% sure if it is completely applicable - I think the core use case there was systems migration but it may go beyond that
- UX patterns for common cross-app tasks (e.g. would it be useful to have a UX for a 'quick add' task that could be used in different contexts when you need to create something in another app). I think it would be good to have UX specialists (Kimie, Gill) present for this, and possibly also some Stripes devs (e.g. John C/Zak)
- UX/UI and implementers topics
- should be Wednesdays
...
Present | Name | Home Organization |
---|---|---|
Ann-Marie Breaux | EBSCO | |
x | Brooks Travis | EBSCO |
x | Charlotte Whitt | Index Data |
Dennis Bridges | EBSCO | |
x | Dung-Lan Chen | Skidmore College |
Gill Osguthorpe | UX/UI Designer - K-Int | |
x | Heather McMillan Thoele | TAMU |
Ian Ibbotson | Developer Lead - K-Int | |
x | Jana Freytag | VZG, Göttingen |
Khalilah Gambrell | EBSCO | |
x | Kirstin Kemner-Heek | VZG, Göttingen |
x | Kristin Martin | Chicago |
x | Laura Daniels | Cornell |
Lloyd Chittenden | Marmot Library Network | |
x | Marc Johnson | K-Int |
x | Martina Schildt | VZG, Göttingen |
x | Martina Tumulla | hbz, Cologne |
x | Maura Byrne | Chicago |
Mike Gorrell | Index Data | |
x | Owen Stephens | Product Owner - Owen Stephens Consulting |
x | Patty WanningerRaman Auramau | EBSCO |
Sara Colglazier | Five Colleges / Mount Holyoke College Library | |
Kimie Kester | EBSCO | |
John Coburn | EBSCO | |
Zak Burke | EBSCO |
...