[FOLIO-3848] Configure MinIO/S3 for mod-oai-pmh Created: 23/Jun/23  Updated: 29/Jun/23  Resolved: 29/Jun/23

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: Wayne Schneider Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File mod-oai-pmh.log    
Sprint: DevOps Sprint 160
Development Team: FOLIO DevOps
RCA Group: TBD

 Description   

mod-oai-pmh now requires environment variables for setting up an S3-compatible storage layer. A new bucket should be provisioned for mod-oai-pmh data. This needs to be set up for Vagrant and for the hosted reference environments (folio-snapshot).

See draft PR



 Comments   
Comment by Wayne Schneider [ 23/Jun/23 ]

This unfortunately did not succeed in the hosted reference environment (which uses S3, not MinIO). In that environment, these are the environment variables that get defined (secrets redacted):

  "env" : [ {
    "name" : "S3_BUCKET",
    "value" : "folio-snapshot-test-oai-pmh"
  }, {
    "name" : "S3_REGION",
    "value" : "us-east-1"
  }, {
    "name" : "DB_PORT",
    "value" : "5432"
  }, {
    "name" : "LOG4J_FORMAT_MSG_NO_LOOKUPS",
    "value" : "true"
  }, {
    "name" : "S3_IS_AWS",
    "value" : "true"
  }, {
    "name" : "ENV",
    "value" : "FOLIO"
  }, {
    "name" : "S3_URL",
    "value" : "https://s3.amazonaws.com"
  }, {
    "name" : "DB_HOST",
    "value" : "10.36.1.168"
  }, {
    "name" : "KAFKA_PORT",
    "value" : "9092"
  }, {
    "name" : "KAFKA_HOST",
    "value" : "10.36.1.168"
  }, {
    "name" : "ELASTICSEARCH_HOST",
    "value" : "10.36.1.168"
  }, {
    "name" : "DB_USERNAME",
    "value" : "folio_admin"
  }, {
    "name" : "S3_SECRET_ACCESS_KEY",
    "value" : "***"
  }, {
    "name" : "S3_ACCESS_KEY_ID",
    "value" : "***"
  }, {
    "name" : "ELASTICSEARCH_PORT",
    "value" : "9301"
  }, {
    "name" : "DB_DATABASE",
    "value" : "okapi_modules"
  }, {
    "name" : "DB_PASSWORD",
    "value" : "***"
  } ],

The user and bucket exist and have the right access policy, AFAICT.

This is what I see in the log:

17:54:04 [] [] [] [] INFO  ?                    mod-oai-pmh-3.12.0-SNAPSHOT.311 17:54:04 [] [] [] [] WARN  ctApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'instancesDaoImpl': Unsatisfied dependency expressed through field 'folioS3Client'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'folioS3Client' defined in class path resource [org/folio/oaipmh/helpers/configuration/ErrorServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.folio.s3.client.FolioS3Client]: Factory method 'folioS3Client' threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials
17:54:04 [] [] [] [] INFO  ?                    mod-oai-pmh-3.12.0-SNAPSHOT.311 17:54:04 [] [] [] [] INFO  RestVerticle         Init hook called with implemented class named org.folio.rest.impl.InitAPIs
17:54:05 [] [] [] [] INFO  ?                    mod-oai-pmh-3.12.0-SNAPSHOT.311 17:54:04 [] [] [] [] ERROR RestVerticle         Error creating bean with name 'instancesDaoImpl': Unsatisfied dependency expressed through field 'folioS3Client'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'folioS3Client' defined in class path resource [org/folio/oaipmh/helpers/configuration/ErrorServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.folio.s3.client.FolioS3Client]: Factory method 'folioS3Client' threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials
17:54:05 [] [] [] [] INFO  ?                    mod-oai-pmh-3.12.0-SNAPSHOT.311 org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'instancesDaoImpl': Unsatisfied dependency expressed through field 'folioS3Client'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'folioS3Client' defined in class path resource [org/folio/oaipmh/helpers/configuration/ErrorServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.folio.s3.client.FolioS3Client]: Factory method 'folioS3Client' threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials

After which the container exits.

Oleksandr Bozhko you might need to take a look, I'm not sure if you've tested with S3 or just with MinIO.

Comment by Oleksandr Bozhko [ 27/Jun/23 ]

Wayne Schneider - update for mod-oai-pmh module to fix this issue is already present in the master branch.

Comment by Wayne Schneider [ 27/Jun/23 ]

We have a successful test build. Log file for mod-oai-pmh is attached. Oleksandr Bozhko please let me know if further testing is needed or if we can go ahead and merge the changes to the build playbooks. Thanks!

Comment by Wayne Schneider [ 29/Jun/23 ]

Changes are now merged (sorry for the delay), the next folio-snapshot build should contain the updated configuration for mod-oai-pmh.

Generated at Thu Feb 08 23:31:14 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.