[FOLIO-2499] Make aws credentials available to mod-data-export Created: 05/Mar/20  Updated: 03/Jun/20  Resolved: 13/Mar/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Story Priority: P3
Reporter: Kruthi Vuppala Assignee: Ian Hardy
Resolution: Done Votes: 0
Labels: devops, devops-backlog, platform-backlog, security
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by FOLIO-2280 tenant superuser granted excessive ok... Closed
Cloners
clones FOLIO-2444 Include mod-data-export in folio-test... Closed
Sprint: DevOps: sprint 84, DevOps: sprint 82
Development Team: FOLIO DevOps

 Description   

Overview
mod-data-export will need to store generated files, the storage currently will support only AWS S3. The module can pickup the credentials from the credential chain https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html

The idea is to create a separate bucket for each tenant in the same AWS account
Note: (Other platform agnostic implementation will be implemented in Q2 release, there may be additional changes needed at that time)

Acceptance Criteria

  • mod-data-export is able to access AWS credentials (probably in the credential file in ~/.aws/credentials similar to edge-modules)


 Comments   
Comment by Kruthi Vuppala [ 05/Mar/20 ]

Jakub Skoczen Can you please take a look at this

Comment by Jakub Skoczen [ 06/Mar/20 ]

Kruthi Vuppala we are discussing this ticket and are ready to address this next sprint but we have concerns about security aspects of this – will get back to you early next week.

Comment by Kruthi Vuppala [ 06/Mar/20 ]

Jakub Skoczen Thank you. We plan on testing it on the reference environments before the release. I hope it can be done early next week.

Comment by Oleksiy_Lemeshko [ 09/Mar/20 ]

hi Jakub Skoczen dropped you direct message but since this ticket is important dependency for Concorde team I'm double checking here. Are there any updates about security aspects you mentioned?

Comment by Jakub Skoczen [ 09/Mar/20 ]

Oleksiy_Lemeshko The team has raised concerns about the module having direct access to AWS credentials that would allow it to create S3 buckets on demand. As far as we know this would be a precedent for such behavior. We will discuss this issue more during the planning meeting today.

Comment by Kruthi Vuppala [ 09/Mar/20 ]

Hello Jakub Skoczen, AFAIK, the credentials are already made available for edge modules. Do you anticipate anything different for this module, other than creating the buckets and storing the files?

Comment by Jakub Skoczen [ 09/Mar/20 ]

Kruthi Vuppala is it acceptable to provide AWS credentials to this module with a read/write access to an already created S3 bucket (rather than ability to create new buckets)? And then create a folder within the bucket for each new tenant?

Comment by Kruthi Vuppala [ 09/Mar/20 ]

Jakub Skoczen Yes that should work, that would make it easier for the module too, but if the S3 bucket is provided there are few things to keep in mind
1) The creation/deletion of the S3 bucket and the files(during the rebuilds of reference environments) should be taken care too
2) send the bucket name as a parameter/or in config file to the module during start up
3) We also need permissions to generate a presigned URL , to be able to download the file(I think the read permissions should cover that, but just being explicit)

Thank you!

Comment by Ian Hardy [ 09/Mar/20 ]

OK, sounds like each environment should create/zero out it's s3 bucket each time its built. We can work on creating a permission set that has read/write withing those buckets.

3) We also need permissions to generate a presigned URL , to be able to download the file(I think the read permissions should cover that, but just being explicit)

It would be just that account that needs read permissions to download the file or are the exports public?

Comment by Kruthi Vuppala [ 09/Mar/20 ]

Ian Hardy

It would be just that account that needs read permissions to download the file or are the exports public?

exported files are not public, just the read and write access will be sufficient

Comment by Kruthi Vuppala [ 10/Mar/20 ]

Jakub Skoczen/Ian Hardy Just trying to understand if we can expect this story to be completed by this week? Thanks!

Comment by Ian Hardy [ 10/Mar/20 ]

Hi Kruthi Vuppala I think that's reasonable. I think I've got the blocker (folio-2280 closed) this morning, and set up aws policies we'll need to apply to the buckets. Still need to add some ansible tasks so that a bucket is created with each reference environment build and then pass appropriate variables to your module.

Comment by Marc Johnson [ 10/Mar/20 ]

Ian Hardy Kruthi Vuppala Is part of this work to update the installation instructions for the platform to include the need to set up an S3 bucket? Or is that part of another piece of work.

Comment by Ian Hardy [ 10/Mar/20 ]

Hi Marc Johnson, we usually open a ticket to update the install instructions in the folio-install repository when the release comes out (for example: https://folio-org.atlassian.net/browse/FOLIO-1651), is that the type of instruction you're referring to? Seems like it would also be good to include some relevant information on s3 and what environment variables need to be passed to the module to get it working on the module's README (at least that's the first place I'd look).

Comment by Ian Hardy [ 13/Mar/20 ]

Kruthi Vuppala the credentials should be provisioned starting with tonights builds of folio-testing and snapshot. Let me know if there are any problems.

Comment by Kruthi Vuppala [ 13/Mar/20 ]

Ian Hardy Thank you. I will test and let you know

Comment by Kruthi Vuppala [ 13/Mar/20 ]

Was able to successfully test in folio-testing using okapi : https://folio-testing-okapi.aws.indexdata.com
But running into few issues in folio-snapshot. Will keep you updated

Comment by Ian Hardy [ 13/Mar/20 ]

Thanks for testing, I spotted a mistake on folio-snapshot on my end and fixed it here: https://github.com/folio-org-priv/folio-infrastructure/pull/171/files

Comment by Kruthi Vuppala [ 13/Mar/20 ]

Thanks Ian, Am I supposed to be able to access that link? because I cannot

Comment by Ian Hardy [ 13/Mar/20 ]

Whoops, I forgot we moved that repo to another organization to save on private repository costs. I had got a variable name wrong and fixed it. I expect snapshot works fine next time it builds.

Generated at Thu Feb 08 23:21:05 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.