2023-02-27 - Tenant Collection Kafka Topics Discussion

Owned by Craig McNally
Last updated: Feb 27, 2023 by Florian Gleixner

Date

Attendees 

Discussion items

TimeItemWhoNotes
1 minScribeAll

Florian Gleixner 

*

RFC: Tenant Collection Kafka Topics

Olamide Kolawole 

Background:  

  • See https://github.com/folio-org/rfcs/pull/6
  • Discussion notes from TC meeting:
    • Goal is to advance RFC to public review process
    • Concern about tenant separation being out of scope.
      • Goal of RFC was to reduce the partition count and therefore reduce costs
      • Tenant separation seems to be another large issue and too much scope for this RFC
      • Original per-tenant Kafka messages at least allowed us to keep
      • Temporary Kafka security solution would be rendered obsolete/deprecated by this RFC (though it seems unlikely this will be done any time soon)
        • Does the ALL  tenant topic would close the door to the possibility of separating tenant messages?
        • RFC describes an option to combine tenant message, not a requirement
        • Major trade-off is that we are committing in the long term to support two fundamentally different approaches for the foreseeable future, but does keep the door open
      • RFC is intended to be practical was of addressing pain points, which includes limited development and funding; hurdle is very high to implement enhanced security
    • Other notes?
    • Inference: providers do not migrate all tenants to new flower releases all at once, and therefore need multiple versions of a module are active at once. This gives a scenario where multiple versions of Inventory will be consuming a message.
    • Response was cut off due to end of meeting
    • To continue in dedicated discussion on Monday

Discussion Notes:

Olamide Kolawole explains how Upgrades of modules using Kafka nowadays work:

  • If consumer modules are upgraded, both, the old and the new Module may consume messages. If for example the mod-search module in a updated version changes the index structure, the Upgrade cannot work without inconsistencies.
  • Kafka Tenant Collections don't make it worse neither offer a solution

Julian Ladisch solution can be to use the "environment" in the topic path and put the flower release name in there.

Olamide Kolawole Security could be done using message encryption on tenant basis. This is not part of this RFC, but a standard solution that would have to be implemented

Marc Johnson Astonishing, that this existing upgrade problem is not a issue for hosting providers today

Olamide Kolawole mod-pubsub solves the problem, but mod-pubsub has other problems with performance and lost messages

Marc Johnson pubsub is deprecated, we should not be relying on pubsub. Biggest concern with this proposal is having to support both possibilities. More configuration possibilities - more tests - Bugfest is run by Ebsco, so the Tenant Collections will be testet for sure.

Olamide Kolawole thinks that topics per tenant will be used in Bugfest, because it is default.

Craig McNally If there are no other concerns, we can move forward to the public review. 7 TC members are in the meeting

Approved by lazy consensus.

Julian Ladisch will resolve the discussion and merge the PR.





Action Items

  •