2024-05-31 Sys Ops & Management SIG Agenda and Meeting notes

Dear folks,

The Formalization Subgroup of the FOLIO Tri-Council is currently surveying the opinion of the Community about how we want to proceed with introducing the new Eureka Platform into the Project.

See Folio Eureka Platform Overview.

They have asked me to survey the opinion of the system administrators or people responsible for hosting the system. We and the Techincal Council will be the key informants of their survey.

So the question is, where do we, where does the FOLIO community collectively intend to go ?

To say that much in advance:

• I don't think this is about a collective "No" of the Community to Eureka. This will not be an option. It's about the how and when it will be introduced.

I imagine that we find answers to the following questions:

1. Shall the Project support both platforms (Okapi, Eureka) perpetually ?
2. Shall the Project move from Okapi to Eureka after some time of an overlap ? How long should that overlap period be ?
3. Do you / your institution understand the reasons why Eureka has been made up ? Do you understand the benefits ?
4. Do you / does your institution understand the implications of the Switch (Okapi → Eureka) ? If not, approximately how much time do you expect to need to sufficiently understand the implications (technically) ?
5. If your institution plans or wants to stay with Okapi for a long time (say at least for 3 years), what are the reasons ?
   1. your own Ressources
   2. Lack of (local, i.e. your) requirements (e.g. towards system security, performance)
   3. Distrust in (some of the) new Components, e.g. maybe distrust that Kong remains OpenSource, that Kong might not support FOLIO forever
   4. Lack of support or documentation
   5. Fear to lose clients; fear that clients don't understand the change because they won't see a change
   6. Want to use software made (and supported) by Index Data
   7. others
6. Does your institution already have a timetable for a Switch to the Eureka platform ? If yes, what time scale are we talking about, approximately ?
   1. Can your institution in the first place make a decision whether they would like to change the platform or not right now ? If not, how much time do you expect to need in order to be able to make a decision ? What information is missing in order for you to make a decision ?
7. Do you think that Eureka will introduce a larger complexity into your hosted system ?
   1. If yes, do you think this will only be in the initial, implementation stage or will it remain to be more complex ?
   2. Do you expect that system maintainance will eventually become easier with Eureka ? (compared to your current setup)
8. Do we expect a performance improvement by replacing Okapi with Kong + Sidecars ?
   1. if not, what other positive aspects do we see in replacing Okapi by Kong + Sidecars ?
   2. do we see negative aspects ?
9. Have you thought about migration (of your production systems) from Okapi to Eureka ?
   
   1. How much concern does that raise in your mind ?
   2. Do you consider this particularly difficult ? More difficult than the platform switch itself (without any production data) ?
10. How much effort do you think it will be to migrate permissions and authorizations ?
11. How problematic do we see the close ties to Kubernetes which seem to be enfored by the employment of Sidecars (and Keycloak) ?
    1. Can we think of running FOLIO (in production) in a non-Kubernetes based environment now or in the future ? If yes, what will be the implications for a switch to Eureka now ?
12. Do we still need an installation documentation for a Single Server (single VM) ?
    1. if yes, what would that look like for an Eureka-based system ? What environment would need to run on that VM ?
    2. if not, what will replace it ? Will we have a step-by-step installation documentation on a K8s-Cluster ? If yes, who will maintain that ?
13. Do we still want to have a Vagrant box for demo and development purposes  ?
14. Which other specific concerns do Sys Ops have when they think about the switch from Okapi to Eureka ?

Meeting Notes go here:

• Several responded “Yes” quickly, Ingolf explains that as he sees it - platform security is the main reason for the new platform.
• Florian and Denis mention that sidecars allow a better switch-over and easy mechanism to implement.
• Wayne explains there is a lot of new code at play in the the platform. Not necessarily bad or good, but that it needs to be taken into consideration.
• Wayne/Jason explain it’s a common architecture in K8s for a service mesh, without having to change or add a bunch of code for each backend module.

• Jason and Tod state that security requirements are becoming much more stringent in the Higher Ed space.
• Wayne in the chat states: “To be clear, IMO Keycloak integration seems like a big win for many reasons (particularly around authentication and integration with various forms of SSO), but permissions management will continue to be complex, I think.”

• Jenn: EBSCO does run multiple bugfests now they are just not full community bugfests. I would expect that would continue for at least part of the transition.
• Jenn also points out that the community might have to maintain artifacts that could otherwise be let go, and that could be costly in time and money.

Recap by Ingolf:

Question 1.  + 2. : Q. 3. should be answered first

Q 3: (also see Meeting notes)

• we understand that end-to-end encryption is a legal requirement
• We understand the motiviation, but we do not understand why this particular solution has been chosen.
  • for example: Okapi could do end-to-end encryption, it's the modules that communcate via http
  • Sidecars have been chosen in order not to have to make a code change on the modules (thus, for backward compatbility)
• Kong has been chosen because it has a large community support. Okapi commits have been lately mostly driven by Julian Ladisch (a FOLIO community developer)
• not all have the same requirements. Is the move to Eureka a benefit for the whole community ?
• General dircetion is towards more security. Consider the attacks on the British Library, recently. They are stil recovering from this.

Q 5. + 6: : We don't know enough about the platform, yet, to answer these questions. Need to rebuild the platform on our own infrastrcture. This will take month. Interesting to see what will happen in the next 6 month. The Architectural Overview that EBSCO gave is excellent. But we also need a detailed prescription how to deploy the system.

Deployment efforts of the German Sys-Ops group currently run into a dead-end: Brought up Keycloak and set env vars via the Keycloak admin surface. But credentials could not be passed to mgr-applications. Can not login in to. Wonder how EBSCO did that. Has some kind of a secret-store been used ? We need more information.

Other institutions did not yet start an effort to re-build the Eureka platform locally.

Q 7: Can not decide, yet. Sidecars introduce a greater complexity. Maybe, when everything is up and running, the maintainance effort becomes less.

Q 8: Because modules communicate directly, the number of "hops" decreases. Expect performance improvement for check-outs.

Q 9: Performance during initial migration depends on how the institution is doing that. Migration from one flower release to another (=upgrades) works differently, not so much dependent on the platform architecture.

Q 11: First of all, not sure if sidecars & keycloak really induces a dependence on K8s. Some say, EBSCO doesn't even use k8s, but Amazon ECS. Other say, they are using some Kubernetes layer on top of this. What is the truth ? It is certainly possible to spin up sidecars in docker, without K8s. But we doubt that anyone will do it that way.

Q 12: This we consider an issue. We will need a good documentation, and it's certainly going to look completely different to the current single Server prescription. Currently, the Single Server VM and the Vagrant Box deployment are the only official documentation. What will replace it ? Who will maintain this ?

Some questions that I have (probably to Index Data):

What exactly will it mean: "the support for Okapi will end" (if it will).

• Doesn't Index Data use Okapi also for other prodcuts (I remember it was there before FOLIO) ? So wouldn't they continue supporting it at least by security patches ?
• Or does it rather mean that at some point of time there will be (new? changed?) FOLIO applications that can not be deployed with Okapi anymore (if so, what could be the reason for that, technically?). What does this mean exactly ? Are we talking about the manager applications (I won't need those if I use Okapi for enablement and dependency resolution) ?

Group will re-meet in two weeks to resume this discussion