2023-02-27 - Tenant Collection Kafka Topics Discussion

1 min

Scribe

All

@Florian Gleixner 

*

RFC: Tenant Collection Kafka Topics

@Olamide Kolawole 

Background:  

• See https://github.com/folio-org/rfcs/pull/6

• Discussion notes from Feb 22, 2023 TC meeting:

  • Goal is to advance RFC to public review process

  • Concern about tenant separation being out of scope.

    • Goal of RFC was to reduce the partition count and therefore reduce costs

    • Tenant separation seems to be another large issue and too much scope for this RFC

    • Original per-tenant Kafka messages at least allowed us to keep

    • Temporary Kafka security solution would be rendered obsolete/deprecated by this RFC (though it seems unlikely this will be done any time soon)

      • Does the ALL  tenant topic would close the door to the possibility of separating tenant messages?

      • RFC describes an option to combine tenant message, not a requirement

      • Major trade-off is that we are committing in the long term to support two fundamentally different approaches for the foreseeable future, but does keep the door open

    • RFC is intended to be practical was of addressing pain points, which includes limited development and funding; hurdle is very high to implement enhanced security

  • Other notes?

  • Inference: providers do not migrate all tenants to new flower releases all at once, and therefore need multiple versions of a module are active at once. This gives a scenario where multiple versions of Inventory will be consuming a message.

  • Response was cut off due to end of meeting

  • To continue in dedicated discussion on Monday

Discussion Notes:

@Olamide Kolawole explains how Upgrades of modules using Kafka nowadays work:

• If consumer modules are upgraded, both, the old and the new Module may consume messages. If for example the mod-search module in a updated version changes the index structure, the Upgrade cannot work without inconsistencies.

• Kafka Tenant Collections don't make it worse neither offer a solution

@Julian Ladisch solution can be to use the "environment" in the topic path and put the flower release name in there.

@Olamide Kolawole Security could be done using message encryption on tenant basis. This is not part of this RFC, but a standard solution that would have to be implemented

@Marc Johnson Astonishing, that this existing upgrade problem is not a issue for hosting providers today

@Olamide Kolawole mod-pubsub solves the problem, but mod-pubsub has other problems with performance and lost messages

@Marc Johnson pubsub is deprecated, we should not be relying on pubsub. Biggest concern with this proposal is having to support both possibilities. More configuration possibilities - more tests - Bugfest is run by Ebsco, so the Tenant Collections will be testet for sure.

@Olamide Kolawole thinks that topics per tenant will be used in Bugfest, because it is default.

@Craig McNally If there are no other concerns, we can move forward to the public review. 7 TC members are in the meeting

Approved by lazy consensus.

@Julian Ladisch will resolve the discussion and merge the PR.