[FOLIO-2705] Create dedicated S3 buckets for FOLIO dev teams Created: 27/Jul/20  Updated: 14/Aug/20  Resolved: 14/Aug/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: John Malconian Assignee: John Malconian
Resolution: Done Votes: 0
Labels: dev-environment, devops-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Cloners
is cloned by FOLIO-2721 Connect rancher pipeline to dedicated... Closed
Relates
relates to MODOAIPMH-178 Create integration tests for EDGE AND... Closed
relates to MODOAIPMH-209 Run integration tests on Rancher scra... Closed
Sprint: DevOps: Sprint 95
Development Team: FOLIO DevOps

 Description   

Create a dedicated S3 bucket in us-west-2 for each dev team and provide write access and credentials to each dev team. This is primarily to support the data-import/export modules but can be used for other team tasks. Determine feasibility of public read-only access.



 Comments   
Comment by Ian Hardy [ 29/Jul/20 ]

fwiw, this is the policy we're currently using for the reference envs for data export: https://github.com/folio-org-priv/folio-infrastructure/blob/master/CI/ansible/roles/s3-data-export/templates/policy.json.j2.

mod-data-export on all of those builds (snapshot, testing etc) has the same IAM account which is granted list at the bucket root, and CRUD on sub-directories in the policy above. mod-data-export will create a directory named after the tenant it is enabled. To provide access it creates a magic link that expires in some time and shares it with the user in the UI.

Comment by John Malconian [ 12/Aug/20 ]

I've created an Ansible playbook in folio-infrastructure called 'dev-env-s3.yml' which creates an S3 bucket for each of the dev teams configured in Rancher. I've created buckets for each team already. The name of each bucket is the name of the dev team prepended with 'folio-'. For example, 'folio-firebird'.

  • Each bucket is read/write when EITHER the correct AWS credentials are specified OR the bucket is accessed from any pods on the folio-eks-2-us-west-2 K8s worker nodes.

There are one set of AWS credentials for all buckets. It may be necessary to set up these credentials as K8s secrets for each dev team's folio namespace since it's not clear to me whether the data-export module actually programmatically requires credentials be configured in order to function.

Comment by Kruthi Vuppala [ 13/Aug/20 ]

John Malconian For data-export we use the AWS sdk, so it looks up the credential chain for the access. There are multiple ways this could be done, so they could be configured either as Environment variables or Web Identity Token credentials . Here are the entire list of options: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

Comment by John Malconian [ 14/Aug/20 ]

Thanks, Kruthi Vuppala. My interpretation of that is that credentials must be provided whether they are needed or not. In the new dev environment, credentials are not needed to access write to the S3 bucket, but I can provide credentials that can be set as environment variables to the data-export pod via a Kubernetes secret.

Comment by John Malconian [ 14/Aug/20 ]

Updated documentation about S3 access here: https://github.com/folio-org/folio-org.github.io/blob/master/_faqs/how-to-get-started-with-rancher.md

Comment by John Malconian [ 14/Aug/20 ]

Created https://github.com/folio-org-priv/folio-infrastructure/pull/217 to add AWS credentials for team S3 bucket as K8s secret via Terraform to each dev team project.

Comment by Kruthi Vuppala [ 14/Aug/20 ]

Thanks much John Malconian

Generated at Thu Feb 08 23:22:38 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.