[FOLIO-3053] Configure mod-data-export-worker for reference Vagrant boxes Created: 03/Mar/21  Updated: 09/Apr/21  Resolved: 09/Apr/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: Wayne Schneider Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: Text File mod-data-export-worker.log    
Issue links:
Relates
relates to FOLIO-3031 Install mod-data-export-worker in the... Closed
Sprint: DevOps Sprint 109, DevOps Sprint 110, DevOps Sprint 111
Development Team: FOLIO DevOps

 Description   

mod-data-export-worker apparently requires either a MinIO environment or an S3 bucket. If neither are configured, the application crashes. Log file attached: mod-data-export-worker.log

Since we don't want to embed AWS credentials in the boxes, we'll need to look at configuring this with MinIO.



 Comments   
Comment by Wayne Schneider [ 09/Mar/21 ]

Merged a temporary fix to configuration variables that allows mod-data-export-worker to deploy without crashing.

Next up: provide MinIO environment and configuration.

Comment by Wayne Schneider [ 10/Mar/21 ]

Aleksei Prokhorov can you give me some insight on how to configure mod-data-export-worker to use a local MinIO server, please? For example, a server launched using the Docker image minio/minio.

Comment by Wayne Schneider [ 11/Mar/21 ]

Created minio-docker role in folio-ansible.

Comment by Wayne Schneider [ 17/Mar/21 ]

Aleksei Prokhorov ping – if there is no guidance on how to configure mod-data-export-worker to use a local MinIO server, we will need to close this ticket as "won't do."

Comment by Aleksei Prokhorov [ 18/Mar/21 ]

Wayne Schneider I remember this ticket and it should be done. Give me please a day or two to complete my release activities and I'll be able to address this ticket.

Comment by Wayne Schneider [ 18/Mar/21 ]

Thanks, Aleksei Prokhorov, I know this is a busy time.

Comment by Aleksei Prokhorov [ 26/Mar/21 ]

Wayne Schneider
"a server launched using the Docker image minio/minio" - yes, this is what I would use in this case.
mod-data-export-worker uses the following environment variables for connecting to a storage:
AWS_URL
AWS_REGION
AWS_BUCKET
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Examples:

When it is in Rancher then the vars are:
AWS_URL=https://s3.amazonaws.com
AWS_REGION=us-west-2
AWS_BUCKET=folio-scout
AWS_ACCESS_KEY_ID=<secret>
AWS_SECRET_ACCESS_KEY=<secret>

When it is in the testing env. then the vars are:
AWS_URL=https://s3.amazonaws.com
AWS_REGION=us-east-1
AWS_BUCKET=folio-testing-export
AWS_ACCESS_KEY_ID=<secret>
AWS_SECRET_ACCESS_KEY=<secret>

It is possible to do not set AWS_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY if they are not relevant.

Comment by Wayne Schneider [ 09/Apr/21 ]

Thanks, Aleksei Prokhorov!

For a Vagrant box using MinIO, the vars are:

AWS_URL=http://10.0.2.15:<MinIO port, default 9000>
AWS_REGION=""
AWS_BUCKET=data-export # this is arbitrary
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
Generated at Thu Feb 08 23:25:15 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.