[FOLIO-2499] Make aws credentials available to mod-data-export Created: 05/Mar/20 Updated: 03/Jun/20 Resolved: 13/Mar/20 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Story | Priority: | P3 |
| Reporter: | Kruthi Vuppala | Assignee: | Ian Hardy |
| Resolution: | Done | Votes: | 0 |
| Labels: | devops, devops-backlog, platform-backlog, security | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||
| Sprint: | DevOps: sprint 84, DevOps: sprint 82 | ||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||
| Description |
|
Overview The idea is to create a separate bucket for each tenant in the same AWS account Acceptance Criteria
|
| Comments |
| Comment by Kruthi Vuppala [ 05/Mar/20 ] |
|
Jakub Skoczen Can you please take a look at this |
| Comment by Jakub Skoczen [ 06/Mar/20 ] |
|
Kruthi Vuppala we are discussing this ticket and are ready to address this next sprint but we have concerns about security aspects of this – will get back to you early next week. |
| Comment by Kruthi Vuppala [ 06/Mar/20 ] |
|
Jakub Skoczen Thank you. We plan on testing it on the reference environments before the release. I hope it can be done early next week. |
| Comment by Oleksiy_Lemeshko [ 09/Mar/20 ] |
|
hi Jakub Skoczen dropped you direct message but since this ticket is important dependency for Concorde team I'm double checking here. Are there any updates about security aspects you mentioned? |
| Comment by Jakub Skoczen [ 09/Mar/20 ] |
|
Oleksiy_Lemeshko The team has raised concerns about the module having direct access to AWS credentials that would allow it to create S3 buckets on demand. As far as we know this would be a precedent for such behavior. We will discuss this issue more during the planning meeting today. |
| Comment by Kruthi Vuppala [ 09/Mar/20 ] |
|
Hello Jakub Skoczen, AFAIK, the credentials are already made available for edge modules. Do you anticipate anything different for this module, other than creating the buckets and storing the files? |
| Comment by Jakub Skoczen [ 09/Mar/20 ] |
|
Kruthi Vuppala is it acceptable to provide AWS credentials to this module with a read/write access to an already created S3 bucket (rather than ability to create new buckets)? And then create a folder within the bucket for each new tenant? |
| Comment by Kruthi Vuppala [ 09/Mar/20 ] |
|
Jakub Skoczen Yes that should work, that would make it easier for the module too, but if the S3 bucket is provided there are few things to keep in mind Thank you! |
| Comment by Ian Hardy [ 09/Mar/20 ] |
|
OK, sounds like each environment should create/zero out it's s3 bucket each time its built. We can work on creating a permission set that has read/write withing those buckets.
It would be just that account that needs read permissions to download the file or are the exports public? |
| Comment by Kruthi Vuppala [ 09/Mar/20 ] |
exported files are not public, just the read and write access will be sufficient |
| Comment by Kruthi Vuppala [ 10/Mar/20 ] |
|
Jakub Skoczen/Ian Hardy Just trying to understand if we can expect this story to be completed by this week? Thanks! |
| Comment by Ian Hardy [ 10/Mar/20 ] |
|
Hi Kruthi Vuppala I think that's reasonable. I think I've got the blocker (folio-2280 closed) this morning, and set up aws policies we'll need to apply to the buckets. Still need to add some ansible tasks so that a bucket is created with each reference environment build and then pass appropriate variables to your module. |
| Comment by Marc Johnson [ 10/Mar/20 ] |
|
Ian Hardy Kruthi Vuppala Is part of this work to update the installation instructions for the platform to include the need to set up an S3 bucket? Or is that part of another piece of work. |
| Comment by Ian Hardy [ 10/Mar/20 ] |
|
Hi Marc Johnson, we usually open a ticket to update the install instructions in the folio-install repository when the release comes out (for example: https://folio-org.atlassian.net/browse/FOLIO-1651), is that the type of instruction you're referring to? Seems like it would also be good to include some relevant information on s3 and what environment variables need to be passed to the module to get it working on the module's README (at least that's the first place I'd look). |
| Comment by Ian Hardy [ 13/Mar/20 ] |
|
Kruthi Vuppala the credentials should be provisioned starting with tonights builds of folio-testing and snapshot. Let me know if there are any problems. |
| Comment by Kruthi Vuppala [ 13/Mar/20 ] |
|
Ian Hardy Thank you. I will test and let you know |
| Comment by Kruthi Vuppala [ 13/Mar/20 ] |
|
Was able to successfully test in folio-testing using okapi : https://folio-testing-okapi.aws.indexdata.com |
| Comment by Ian Hardy [ 13/Mar/20 ] |
|
Thanks for testing, I spotted a mistake on folio-snapshot on my end and fixed it here: https://github.com/folio-org-priv/folio-infrastructure/pull/171/files |
| Comment by Kruthi Vuppala [ 13/Mar/20 ] |
|
Thanks Ian, Am I supposed to be able to access that link? because I cannot |
| Comment by Ian Hardy [ 13/Mar/20 ] |
|
Whoops, I forgot we moved that repo to another organization to save on private repository costs. I had got a variable name wrong and fixed it. I expect snapshot works fine next time it builds. |