[FOLIO-3581] Setup new Env Vars : mod-data-export-worker Created: 16/Sep/22  Updated: 15/Feb/23  Resolved: 27/Sep/22

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: Viachaslau Khandramai (Inactive) Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks MODEXPW-233 Migrate MinIO adapter changes to mast... Closed
Relates
relates to FOLIO-3588 Vagrant box MinIO implementation is b... Open
relates to FOLIO-3594 Hourly platform build failure Closed
relates to FOLIO-3597 Vagrant box: mod-data-export-worker c... Closed
Sprint: DevOps Sprint 149
Development Team: FOLIO DevOps
RCA Group: TBD

 Description   

In order to support last changes in mod-data-export-worker - migration from local file system to S3-compatible remote storage (https://folio-org.atlassian.net/browse/MODEXPW-197) one need to setup new environment variables according README https://github.com/folio-org/mod-data-export-worker/blob/v1.4.6/README.md. I.e. we need to have the following variables: LOCAL_FS_URL, LOCAL_FS_REGION, LOCAL_FS_BUCKET, LOCAL_FS_ACCESS_KEY_ID, LOCAL_FS_SECRET_ACCESS_KEY, LOCAL_FS_COMPOSE_WITH_AWS_SDK. The values LOCAL_FS_ACCESS_KEY_ID, LOCAL_FS_SECRET_ACCESS_KEY can be skipped if we use AWS IAM.



 Comments   
Comment by Wayne Schneider [ 20/Sep/22 ]

The FOLIO environment https://folio-snapshot-test.dev.folio.org has been built with these variables related to S3 in the environment for mod-data-export-worker:

LOCAL_FS_COMPOSE_WITH_AWS_SDK=true
LOCAL_FS_URL=https://s3.amazonaws.com
AWS_BUCKET=folio-snapshot-test-export
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=[xxxxx]
LOCAL_FS_BUCKET=folio-snapshot-test-export-local
AWS_SECRET_ACCESS_KEY=[xxxxx]
LOCAL_FS_REGION=us-east-1
LOCAL_FS_ACCESS_KEY_ID=[xxxxx]
LOCAL_FS_SECRET_ACCESS_KEY=[xxxxx]

Export from the circulation log works as expected, but I don't know how to test the local fs bucket. Can you please test and let me know if it is working as you expect? Thank you!

Comment by Wayne Schneider [ 22/Sep/22 ]

Viachaslau Khandramai I'm sorry, I need you to test this system (https://folio-snapshot-test.dev.folio.org) to confirm that it is working as expected. If you do not get to it by the end of next week, I will need to close this issue as "won't do". Thank you!

Comment by Viachaslau Khandramai (Inactive) [ 27/Sep/22 ]

Hello Wayne Schneider , unfortunately, env variables weren't applied for mod-data-export-worker. Typically, we add the following JAVA OPTIONS for mod-data-export to apply SYSTEM VARIABLES for module:

--ENV=FOLIO
--AWS_URL=https://s3.amazonaws.com
--AWS_REGION=...
--AWS_BUCKET=...
--AWS_ACCESS_KEY_ID=...
--AWS_SECRET_ACCESS_KEY=...
--LOCAL_FS_URL=https://s3.amazonaws.com
--LOCAL_FS_REGION=...
--LOCAL_FS_BUCKET=...
--LOCAL_FS_ACCESS_KEY_ID=...
--LOCAL_FS_SECRET_ACCESS_KEY=...
--LOCAL_FS_COMPOSE_WITH_AWS_SDK=true

I don't know how this is organized on hosted reference env but I see from logs that settled env variables aren't applied:

01:47:09 [] [] [] [] INFO  BaseFilesStorage     Creating MinIO client endpoint https://s3.amazonaws.com,region us-east-1,bucket folio-snapshot-export,accessKey <set>,secretKey <set>, isComposedWithAwsSdk false.
01:47:09 [] [] [] [] INFO  BaseFilesStorage     StaticProvider MinIO credentials provider created.
01:47:09 [] [] [] [] INFO  BaseFilesStorage     Bucket has already exist.
01:47:09 [] [] [] [] INFO  BaseFilesStorage     Creating MinIO client endpoint http://127.0.0.1:9000/,region ,bucket ,accessKey <not set>,secretKey <not set>, isComposedWithAwsSdk false. 

For second client I expect

01:47:09 [] [] [] [] INFO  BaseFilesStorage    Creating MinIO client endpoint https://s3.amazonaws.com,region us-east-1,bucket folio-snapshot-export-local,accessKey <set>,secretKey <set>, isComposedWithAwsSdk false.  

instead of

01:47:09 [] [] [] [] INFO  BaseFilesStorage     Creating MinIO client endpoint http://127.0.0.1:9000/,region ,bucket ,accessKey <not set>,secretKey <not set>, isComposedWithAwsSdk false.   

Other than that both buckets folio-snapshot-export and folio-snapshot-export-local should be pre-created.

Thanks,

Slava

Comment by Wayne Schneider [ 27/Sep/22 ]

Please see above, I asked you to test https://folio-snapshot-test.dev.folio.org. Is that the system you are looking at? The changes have not been merged to the default branches of folio-ansible/folio-infrastructure, so you would not see them in folio-snapshot or folio-snapshot-2 yet.

Comment by Viachaslau Khandramai (Inactive) [ 27/Sep/22 ]

Hello Wayne Schneider. I cannot test https://folio-snapshot-test.dev.folio.org due to proxyClient failure: mod-data-export-worker-1.5.0-SNAPSHOT.286 http://10.36.1.203:9158: finishConnect(..) failed: Connection refused: /10.36.1.203:9158: POST /bulk-edit/1e6fa84a-5422-41ca-84f9-77ce00a1d79d/upload.

Comment by Viachaslau Khandramai (Inactive) [ 27/Sep/22 ]

Wayne Schneider , I can confirm - bulk edit works fine on https://folio-snapshot-test.dev.folio.org with current env settings (users bulk edit flow is verified). So, PR can be opened and merged.

 

Thanks,

Slava

CC Magda Zacharska Mikita Siadykh Denis 

Comment by Wayne Schneider [ 27/Sep/22 ]

Thanks for the review, Viachaslau Khandramai. I've merged the changes into the build repos, and the next build of folio-snapshot/folio-snapshot-2 should include the updated configuration and new required S3 bucket.

Generated at Thu Feb 08 23:29:11 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.