[FOLIO-3310] Data export jobs get stuck on the folio-testing and folio-snapshot environments. Created: 11/Oct/21  Updated: 23/Dec/21  Resolved: 13/Oct/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P1
Reporter: Victoria_Smelova Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FAT-1054 Check that karate tests run correctly Closed
blocks FAT-940 mod-data-import: Implement 'Match MAR... Closed
Relates
relates to FAT-966 Fix data-export step in the mod-data-... Closed
relates to FOLIO-3313 Introduce/verify ENV vars for mod-dat... Closed
Sprint: DevOps Sprint 125
Development Team: FOLIO DevOps
Release: R3 2021

 Description   

As a result of integration of mod-data-export with min.io it is necessary to add the following JAVA_OPTIONS while mod-data-export launch on the envs:

-Daws.url={} -Daws.region={} -Daws.accessKeyId={} -Daws.secretKey={} -Dbucket.name={}

 

For reference you can review the test configuration at https://rancher.dev.folio.org/p/c-479xv:p-v4mss/workload/deployment:concorde:mod-data-export



 Comments   
Comment by Wayne Schneider [ 11/Oct/21 ]

We can certainly do this reconfiguration, but a couple of comments/questions:

  1. mod-data-export-worker is configured using the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_URL, AWS_REGION, and AWS_BUCKET. Interestingly, I can't find any documentation on this except in this JIRA ticket. Can mod-data-export use the same environment variables? That would simplify configuration for system operators who need to support both mod-data-export and mod-data-export-worker.
  2. The Storage configuration section of the README should note that these "parameters" are actually Java system properties that need to be set as above with the -D switch.
  3. The LIMITATIONS OF THE MODULE section of the README is now out of date and does not actually explain how the module needs to be configured.
Comment by Viachaslau Khandramai (Inactive) [ 12/Oct/21 ]

Hello Wayne Schneider,

we implemented this way because bucket name for this module was already used as Java options. So, we thought why we cannot introduce another parameters such way? If it is critical we can use env variables. It isn't big deal but it will be nice to have some recommendations/specifications how it is correct to handle such tasks.

And sure, we can update mentioned sections of the README.md.

Thanks,

Slava

Comment by Wayne Schneider [ 12/Oct/21 ]

Viachaslau Khandramai thanks! It is not critical, but it might be nice for you to coordinate with the team maintaining mod-data-export-worker so that the configuration is aligned.

Comment by Wayne Schneider [ 12/Oct/21 ]

It occurs to me that you could support both methods of configuration, of course.

Comment by Wayne Schneider [ 12/Oct/21 ]

The reconfiguration has been done, and data export is currently working as expected on https://folio-snapshot-load.dev.folio.org. The next rebuild of the reference environments will pull in this change.

Comment by Magda Zacharska [ 13/Oct/21 ]

Verified on folio-snapshot-load and folio-snapshot - works as expected. Thank you Wayne Schneider and Viachaslau Khandramai!

Generated at Thu Feb 08 23:27:10 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.