[FOLIO-3581] Setup new Env Vars : mod-data-export-worker Created: 16/Sep/22 Updated: 15/Feb/23 Resolved: 27/Sep/22 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | TBD |
| Reporter: | Viachaslau Khandramai (Inactive) | Assignee: | Wayne Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||
| Sprint: | DevOps Sprint 149 | ||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||
| RCA Group: | TBD | ||||||||||||||||||||||||
| Description |
|
In order to support last changes in mod-data-export-worker - migration from local file system to S3-compatible remote storage (https://folio-org.atlassian.net/browse/MODEXPW-197) one need to setup new environment variables according README https://github.com/folio-org/mod-data-export-worker/blob/v1.4.6/README.md. I.e. we need to have the following variables: LOCAL_FS_URL, LOCAL_FS_REGION, LOCAL_FS_BUCKET, LOCAL_FS_ACCESS_KEY_ID, LOCAL_FS_SECRET_ACCESS_KEY, LOCAL_FS_COMPOSE_WITH_AWS_SDK. The values LOCAL_FS_ACCESS_KEY_ID, LOCAL_FS_SECRET_ACCESS_KEY can be skipped if we use AWS IAM. |
| Comments |
| Comment by Wayne Schneider [ 20/Sep/22 ] |
|
The FOLIO environment https://folio-snapshot-test.dev.folio.org has been built with these variables related to S3 in the environment for mod-data-export-worker: LOCAL_FS_COMPOSE_WITH_AWS_SDK=true LOCAL_FS_URL=https://s3.amazonaws.com AWS_BUCKET=folio-snapshot-test-export AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=[xxxxx] LOCAL_FS_BUCKET=folio-snapshot-test-export-local AWS_SECRET_ACCESS_KEY=[xxxxx] LOCAL_FS_REGION=us-east-1 LOCAL_FS_ACCESS_KEY_ID=[xxxxx] LOCAL_FS_SECRET_ACCESS_KEY=[xxxxx] Export from the circulation log works as expected, but I don't know how to test the local fs bucket. Can you please test and let me know if it is working as you expect? Thank you! |
| Comment by Wayne Schneider [ 22/Sep/22 ] |
|
Viachaslau Khandramai I'm sorry, I need you to test this system (https://folio-snapshot-test.dev.folio.org) to confirm that it is working as expected. If you do not get to it by the end of next week, I will need to close this issue as "won't do". Thank you! |
| Comment by Viachaslau Khandramai (Inactive) [ 27/Sep/22 ] |
|
Hello Wayne Schneider , unfortunately, env variables weren't applied for mod-data-export-worker. Typically, we add the following JAVA OPTIONS for mod-data-export to apply SYSTEM VARIABLES for module: --ENV=FOLIO I don't know how this is organized on hosted reference env but I see from logs that settled env variables aren't applied: 01:47:09 [] [] [] [] INFO BaseFilesStorage Creating MinIO client endpoint https://s3.amazonaws.com,region us-east-1,bucket folio-snapshot-export,accessKey <set>,secretKey <set>, isComposedWithAwsSdk false. 01:47:09 [] [] [] [] INFO BaseFilesStorage StaticProvider MinIO credentials provider created. 01:47:09 [] [] [] [] INFO BaseFilesStorage Bucket has already exist. 01:47:09 [] [] [] [] INFO BaseFilesStorage Creating MinIO client endpoint http://127.0.0.1:9000/,region ,bucket ,accessKey <not set>,secretKey <not set>, isComposedWithAwsSdk false. For second client I expect
01:47:09 [] [] [] [] INFO BaseFilesStorage Creating MinIO client endpoint https://s3.amazonaws.com,region us-east-1,bucket folio-snapshot-export-local,accessKey <set>,secretKey <set>, isComposedWithAwsSdk false.
instead of
01:47:09 [] [] [] [] INFO BaseFilesStorage Creating MinIO client endpoint http://127.0.0.1:9000/,region ,bucket ,accessKey <not set>,secretKey <not set>, isComposedWithAwsSdk false.
Other than that both buckets folio-snapshot-export and folio-snapshot-export-local should be pre-created. Thanks, Slava |
| Comment by Wayne Schneider [ 27/Sep/22 ] |
|
Please see above, I asked you to test https://folio-snapshot-test.dev.folio.org. Is that the system you are looking at? The changes have not been merged to the default branches of folio-ansible/folio-infrastructure, so you would not see them in folio-snapshot or folio-snapshot-2 yet. |
| Comment by Viachaslau Khandramai (Inactive) [ 27/Sep/22 ] |
|
Hello Wayne Schneider. I cannot test https://folio-snapshot-test.dev.folio.org due to proxyClient failure: mod-data-export-worker-1.5.0-SNAPSHOT.286 http://10.36.1.203:9158: finishConnect(..) failed: Connection refused: /10.36.1.203:9158: POST /bulk-edit/1e6fa84a-5422-41ca-84f9-77ce00a1d79d/upload. |
| Comment by Viachaslau Khandramai (Inactive) [ 27/Sep/22 ] |
|
Wayne Schneider , I can confirm - bulk edit works fine on https://folio-snapshot-test.dev.folio.org with current env settings (users bulk edit flow is verified). So, PR can be opened and merged.
Thanks, Slava |
| Comment by Wayne Schneider [ 27/Sep/22 ] |
|
Thanks for the review, Viachaslau Khandramai. I've merged the changes into the build repos, and the next build of folio-snapshot/folio-snapshot-2 should include the updated configuration and new required S3 bucket. |