[FOLIO-2601] PoC for hosted development environment for FOLIO teams Created: 15/May/20  Updated: 26/Aug/20  Resolved: 26/Aug/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Epic Priority: P2
Reporter: John Malconian Assignee: John Malconian
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Relates
relates to FOLIO-2675 Make the Okapi URL and Tenant ID be e... Open
relates to FOLIO-2669 SPIKE: Determine method of deploying ... Closed
relates to FOLIO-2677 Add PGAdmin to hosted development env... Closed
Epic Name: Hosted development environment for FOLIO teams
Sprint: DevOps: sprint 90, DevOps: Sprint 95, DevOps: Sprint 96
Development Team: FOLIO DevOps

 Description   

Provide hosted, FOLIO sandboxes for FOLIO development teams. The idea is similar to FOLIO developer vagrant environments except hosted on AWS.

  • Each team would be provided with capped AWS compute resources in a dedicated namespace on a Kubernetes cluster managed by Rancher.
  • Each team would be provided with a base set of FOLIO components and modules in each namespace that will be deployed via Jenkins and/or existing FOLIO Ansible playbooks that can be refreshed on-demand.
  • Each team would be able to build and deploy feature branches and additional modules to their namespace utilizing Rancher pipelines.
  • Each team will have the ability to configure their own tenant environments in their namespace.
  • Each team will have the ability to managed containers and access to the container logs and metrics in each namespace through the Rancher UI.
  • Each team will be provided with access to a AWS S3 bucket that can be used for additional data storage and simple web services like hosting Stripes bundles. By default, each team will have write access to their bucket. Read-only access will be provided publicly so that data can be shared to non-team members.


 Comments   
Comment by John Malconian [ 15/May/20 ]

Considerations:

  • Rancher provides Github oauth integration. Access to the Rancher UI can therefore based on existing github/folio-org teams. Rancher supports the notion of "projects" which includes specific K8s namespaces. A Rancher project and associated namespace will be provisioned for each team. CPU, memory, storage and other resources can be reserved and capped on a per project basis. Access to container metrics and pipelines can also be specified on a per project basis as well.
  • Rancher pipelines can be stored as configuration files alongside project source code in Github similar existing Jenkins pipelines or they can be created and managed via the Rancher UI. K8s deployments templates can also be stored in Github that are used by Rancher pipelines to deploy FOLIO components into the K8s namespace.
  • Postgresql, Kafka, and Okapi containers will be pre-deployed to each team namespace. Ingress to the Okapi instance will also be provided. Beyond this, we need to determine how much of the FOLIO stack will be automatically deployed into each namespace and how that will be accomplished. Need to also determine how team environments are cleaned up.
  • Documentation will need to be provided to teams on how to use their environments.
Comment by Jakub Skoczen [ 18/May/20 ]

Anton Emelianov we are thinking about setting up the PoC rancher project for a team env based on "platform-core" rather than "platform-complete". Did you guys consider which team would be testing the setup? We want to make sure they are OK with platform-core modules only. Thanks.

Comment by jroot [ 22/May/20 ]

This is a good layout and methodology, essentially what we are doing at Tamu for some aspects. I do have a few suggestions:

It is best to set up the cloud-provider for storage classes before you create the clusters in Rancher. You can do it after the fact, but it is much better and straight forward to pass in that config at cluster creation time.

The same can be said for Project and/or namespace resource limits. It is a little cumbersome to add those to namespaces after the fact.

I have not determined what a bare minimum amount of resources are to run Folio from a Project/namespace perspective. I have set up container resource reservations and limits for each Folio back-end module - however mine are not very conservative.
For example: With memory reservations, I am using the recommended minimum suggested that is needed to run each module provided by the launch/module descriptor. For maximums, that has been trial and error to determine what will keep a module running during some process (like loading data or doing heavy queries).

There are however no cpu minimums that have been calculated. For now I have picked some arbitrarily low milli cpu amount for the Workload (like 10) and some arbitrarily high milli cpu for a limitation (like 500 or more).

I do have monitoring and metrics set up for the cluster and Projects, so I could get a good minimum for each module, I just have not done so yet.

Comment by jroot [ 22/May/20 ]

A head's up I forgot to mention: Rancher just launched a certification program that is free right now. It is a great training tool to become more familiarized with Rancher and its workings: https://academy.rancher.com/courses/course-v1:RANCHER+K101+2019/about

Comment by Marc Johnson [ 02/Jun/20 ]

Jakub Skoczen Anton Emelianov

Did you guys consider which team would be testing the setup? We want to make sure they are OK with platform-core modules only.

It was my understanding that the pilot team for this is going to be FoliJet, that makes it unlikely that the core rather than complete is appropriate as their primary area of interest is data import.

Comment by Ann-Marie Breaux (Inactive) [ 02/Jun/20 ]

Hi Marc Johnson If it's Folijet, only 1 of our modules is platform-core, though of course we interact lots with UIIN and MODINV. All the other data import apps (plus related ones like Export and quickMARC) are in platform-complete. Please let me know if that's an issue.

Comment by John Malconian [ 02/Jun/20 ]

yeah, initially we'll be looking at deployment of modules based on platform-core only. So if Folijet is not the appropriate team, perhaps core-functional is. We'll discuss on our next status call.

Comment by Ann-Marie Breaux (Inactive) [ 02/Jun/20 ]

John Malconian Hmm, that definitely limits the scope of possible teams. Not sure there would be any besides Core-fxn that would qualify.

Comment by Jakub Skoczen [ 03/Jun/20 ]

This is likley going to be implemented in two phases:

  • Phase 1 setting up infrastrucutre for the team envs and preparing platform-core for deployment into the dedicated namespaces – tickets have been created for this phase and linked to this Epic
  • Phase 2 tuning for deployment of feature branches – tickets are not yet created for this phase, John Malconian will be defining them in Sprint 90
Comment by Peter Murray [ 18/Jun/20 ]

For budgeting/billing purposes, can the resources associated with these hosted development environments for teams be assigned AWS tags? The FOLIO steering group would like to separate out the charges for these environments.

Comment by Peter Murray [ 29/Jun/20 ]

Hey, Jakub Skoczen and John Malconian: Just checking on the last comment here to see if the system setup will allow for an accounting of the costs of the hosted development environments as a separate item (e.g. with AWS tags?—I think I can create budget reports based on tags.)

Comment by John Malconian [ 30/Jun/20 ]

Peter Murray Yes, eksctl, the tool used to create AWS ec2 nodegroups for kubernetes, does support the use of tags. However, Since nodegroups are mostly immutable, I'd have to create two new nodegroups with the tags to replace the existing nodegroups which is a bit of pain. I can do this migration but not right away. If it is at all helpful, I can tell you that all ec2 resources running in us-east-2 (ohio) and us-west-2 (oregon) are used almost exclusively for this project.

Comment by Peter Murray [ 30/Jun/20 ]

Sorry, John—I wasn't clear in my initial question. The request is for the hosted development environments (the environments for the development teams) and not the hosted reference environments that the SIGs use. Are they all being mixed into the same k8 cluster?

Comment by John Malconian [ 01/Jul/20 ]

Not at this time, Peter Murray. All resources in us-east-2 and us-west-2 are specifically utilized for the hosted dev environment.

Generated at Thu Feb 08 23:21:52 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.