[FOLIO-2386] Add mod-pubsub to hosted envs and vagrant boxes Created: 09/Dec/19  Updated: 03/Jun/20  Resolved: 13/Jan/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Kateryna Senchenko Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: ci, data-import, devops, platform-backlog, pub-sub
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks MODINV-181 Integrate mod-inventory with mod-pubsub Closed
is blocked by MODPUBSUB-62 Add new LaunchDescriptor and use new ... Closed
Relates
relates to FOLIO-2359 Include source-record-manager and sou... Open
relates to MODINV-172 Apply event-driven approach for data-... Closed
relates to FOLIO-2353 Add kafka to vagrant boxes Closed
Sprint: CP: sprint 79, CP: sprint 80/81
Development Team: Core: Platform

 Description   

Add https://github.com/folio-org/mod-pubsub to hosted envs and vagrant boxes. It should be included in platform-core modules.



 Comments   
Comment by Kateryna Senchenko [ 09/Dec/19 ]

Hello Jakub Skoczen, Wayne Schneider, Ian Hardy,
To enable pub-sub usage by other modules it should be available on hosted envs and in vagrant boxes. Please take a look at this task. Thank you.

Comment by Ann-Marie Breaux (Inactive) [ 16/Dec/19 ]

Hi Jakub Skoczen We have documentation and everything all ready to go for PubSub. My understanding is that the Kafka work for the hosted environments has also been completed. Do you have a timeframe for when Core-Platform will be able to work on this ticket to add pubsub to the hosted envs? And once that's done, should we plan some sort of meeting with the backend team leads to go over any details with them? My understanding is that some of them are hoping to begin using it in their apps ASAP.

Thank you!

cc: Cate Boerema Hkaplanian VBar Mark Veksler

Comment by Ann-Marie Breaux (Inactive) [ 17/Dec/19 ]

Ann-Marie Breaux From Folijet standup notes 5 Dec: After documentation and adding to core module, announce, and invite questions in the #folio-pubsub Slack channel

Comment by Ann-Marie Breaux (Inactive) [ 18/Dec/19 ]

Copying from comments on MODINV-172:

Marc Johnson added a comment - Yesterday
When this work starts, it would be really useful to understand the events being published by the various modules and what the expected flow is.

Ann-Marie Breaux added a comment - Yesterday
Hi Marc Johnson Thank you for this suggestion - as soon as PubSub is added to the hosted ref envs (see FOLIO-2386 Closed ), I'm thinking it would be good to have an info/Q&A mtg with the tech leads for the various teams, so that Folijet can go over the documentation that is in place so far, answer questions, and have folks raise suggestions like this. What do you think?
================================================================================

Marc Johnson added a comment - 2 hours ago
Ann-Marie Breaux

As soon as PubSub is added to the hosted ref envs (see FOLIO-2386 Closed ), I'm thinking it would be good to have an info/Q&A mtg with the tech leads for the various teams, so that Folijet can go over the documentation that is in place so far, answer questions, and have folks raise suggestions like this. What do you think?

I think that is a great idea. I think we need to be careful about the scope of that conversation.

I think there are 3 topics in this area:

  • How to use mod-pubsub to publish and subscribe to events (including how to define events etc)
  • How FOLIO developers should use this facility to integrate between modules (the tech leads have started discussing this)
  • How the specific import process is likely to work (e.g. what parts of the system publishes which events and what the impact should be in other areas)
    I think what you described above most closely fits the first of those. I think the second is a topic the tech leads and Vince Bareau are working on.

The last topic I think is especially interesting as it will be the first process designed using events in FOLIO. I think that is especially important as to set an example of how we intend this facility / style of integration to be used.

================================================================

Ann-Marie Breaux added a comment - Just now
Hi Marc Johnson Yes, all sounds good.

As soon as FOLIO-2386 Closed is done (and after the holidays), we'll plan to address the first topic. Do you think it's something that could be discussed at a TeamLeads meeting, or will we need to call a special meeting? If special mtg, maybe after the holidays, we can work up a Doodle poll with possible times.

Comment by Ann-Marie Breaux (Inactive) [ 10/Jan/20 ]

Hi Ian Hardy Just checking on this ticket - it has been in progress for a while now. There's no points assigned, so it's not clear to me if this is a big task or not. Do you have an ETA on when it might be completed? Once this is done, Marc Johnson and I need to plan a developer meeting to go over it. Thank you!

cc: Oleksii Kuzminov

Comment by Ian Hardy [ 10/Jan/20 ]

Hi Ann-Marie Breaux, Here's an update on the progress of this ticket.

Done:

  • Did some test builds with kafka/pubsub on the regular folio-snapshot environments (for example, this build is running pubsub: https://folio-snapshot-test.aws.indexdata.com). Running a simple kafka configuration with docker-compose is OK.
  • We also have builds that take place in kubernetes, for this I've installed a more permanent kafka deployment which I'm fairly happy with

To Do:

  • enable automatic deployment of pubsub to kubernetes when a new snapshot/release is built (fairly straighforward).
  • Test k8s builds to make sure they don't break when pubsub is required by platform-core

I'll do my best to have this resolved in the next few days, sorry about the lack of updates. It's turned out to be slightly more complicated than the average add-a-new module type task.

Ian

Comment by Ann-Marie Breaux (Inactive) [ 10/Jan/20 ]

No worries, Ian Hardy - definitely helpful to have an update, so I really appreciate that. And thanks for continuing to work on it!

Comment by Ian Hardy [ 13/Jan/20 ]

Getting a memory error from mod-pubsub in the reference environments, I'd like to bump the default memory settings in the module to resolve this (works when I do this manually) Kateryna Senchenko is reviewing.

13:55:02.716 [vert.x-worker-thread-0] ERROR LiquibaseUtil        [10822eqId] Error while initializing schema for the module
liquibase.exception.ChangeLogParseException: liquibase.exception.SetupException: Out of memory
Comment by Ian Hardy [ 13/Jan/20 ]

Added kafka and pubsub to the snapshot/testing builds. I'm testing using directions from Kateryna below and things look OK. It's up on folio-snapshot-load and testing-core now and the rest of the builds after their regularly schedule refreshes.

1. POST to host:port/pubsub/event-types with body

{
  "eventType": "RECEIVED_SAMPLE_ENTITY",
  "eventTTL": 30
}

2. POST to host:port/pubsub/event-types/declare/publisher

{
  "moduleId": "mod-sample-publisher",
  "eventDescriptors": [
    {
       "eventType": "RECEIVED_SAMPLE_ENTITY",
       "eventTTL": 30
    }]
}
Comment by Ann-Marie Breaux (Inactive) [ 13/Jan/20 ]

Terrific - thanks very much, Ian Hardy. I'll get Folijet folks to check them out, and then talk with Marc Johnson about scheduling some time with developers.

Comment by Kateryna Senchenko [ 14/Jan/20 ]

Checked on folio-snapshot-load - everything works as expected, thank you Ian Hardy.

Generated at Thu Feb 08 23:20:16 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.