[FOLIO-2705] Create dedicated S3 buckets for FOLIO dev teams Created: 27/Jul/20 Updated: 14/Aug/20 Resolved: 14/Aug/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | John Malconian | Assignee: | John Malconian |
| Resolution: | Done | Votes: | 0 |
| Labels: | dev-environment, devops-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||
| Sprint: | DevOps: Sprint 95 | ||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||
| Description |
|
Create a dedicated S3 bucket in us-west-2 for each dev team and provide write access and credentials to each dev team. This is primarily to support the data-import/export modules but can be used for other team tasks. Determine feasibility of public read-only access. |
| Comments |
| Comment by Ian Hardy [ 29/Jul/20 ] |
|
fwiw, this is the policy we're currently using for the reference envs for data export: https://github.com/folio-org-priv/folio-infrastructure/blob/master/CI/ansible/roles/s3-data-export/templates/policy.json.j2. mod-data-export on all of those builds (snapshot, testing etc) has the same IAM account which is granted list at the bucket root, and CRUD on sub-directories in the policy above. mod-data-export will create a directory named after the tenant it is enabled. To provide access it creates a magic link that expires in some time and shares it with the user in the UI. |
| Comment by John Malconian [ 12/Aug/20 ] |
|
I've created an Ansible playbook in folio-infrastructure called 'dev-env-s3.yml' which creates an S3 bucket for each of the dev teams configured in Rancher. I've created buckets for each team already. The name of each bucket is the name of the dev team prepended with 'folio-'. For example, 'folio-firebird'.
There are one set of AWS credentials for all buckets. It may be necessary to set up these credentials as K8s secrets for each dev team's folio namespace since it's not clear to me whether the data-export module actually programmatically requires credentials be configured in order to function. |
| Comment by Kruthi Vuppala [ 13/Aug/20 ] |
|
John Malconian For data-export we use the AWS sdk, so it looks up the credential chain for the access. There are multiple ways this could be done, so they could be configured either as Environment variables or Web Identity Token credentials . Here are the entire list of options: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html |
| Comment by John Malconian [ 14/Aug/20 ] |
|
Thanks, Kruthi Vuppala. My interpretation of that is that credentials must be provided whether they are needed or not. In the new dev environment, credentials are not needed to access write to the S3 bucket, but I can provide credentials that can be set as environment variables to the data-export pod via a Kubernetes secret. |
| Comment by John Malconian [ 14/Aug/20 ] |
|
Updated documentation about S3 access here: https://github.com/folio-org/folio-org.github.io/blob/master/_faqs/how-to-get-started-with-rancher.md |
| Comment by John Malconian [ 14/Aug/20 ] |
|
Created https://github.com/folio-org-priv/folio-infrastructure/pull/217 to add AWS credentials for team S3 bucket as K8s secret via Terraform to each dev team project. |
| Comment by Kruthi Vuppala [ 14/Aug/20 ] |
|
Thanks much John Malconian |