[FOLIO-3848] Configure MinIO/S3 for mod-oai-pmh Created: 23/Jun/23 Updated: 29/Jun/23 Resolved: 29/Jun/23 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | TBD |
| Reporter: | Wayne Schneider | Assignee: | Wayne Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
| Sprint: | DevOps Sprint 160 |
| Development Team: | FOLIO DevOps |
| RCA Group: | TBD |
| Description |
|
mod-oai-pmh now requires environment variables for setting up an S3-compatible storage layer. A new bucket should be provisioned for mod-oai-pmh data. This needs to be set up for Vagrant and for the hosted reference environments (folio-snapshot). See draft PR |
| Comments |
| Comment by Wayne Schneider [ 23/Jun/23 ] |
|
This unfortunately did not succeed in the hosted reference environment (which uses S3, not MinIO). In that environment, these are the environment variables that get defined (secrets redacted): "env" : [ { "name" : "S3_BUCKET", "value" : "folio-snapshot-test-oai-pmh" }, { "name" : "S3_REGION", "value" : "us-east-1" }, { "name" : "DB_PORT", "value" : "5432" }, { "name" : "LOG4J_FORMAT_MSG_NO_LOOKUPS", "value" : "true" }, { "name" : "S3_IS_AWS", "value" : "true" }, { "name" : "ENV", "value" : "FOLIO" }, { "name" : "S3_URL", "value" : "https://s3.amazonaws.com" }, { "name" : "DB_HOST", "value" : "10.36.1.168" }, { "name" : "KAFKA_PORT", "value" : "9092" }, { "name" : "KAFKA_HOST", "value" : "10.36.1.168" }, { "name" : "ELASTICSEARCH_HOST", "value" : "10.36.1.168" }, { "name" : "DB_USERNAME", "value" : "folio_admin" }, { "name" : "S3_SECRET_ACCESS_KEY", "value" : "***" }, { "name" : "S3_ACCESS_KEY_ID", "value" : "***" }, { "name" : "ELASTICSEARCH_PORT", "value" : "9301" }, { "name" : "DB_DATABASE", "value" : "okapi_modules" }, { "name" : "DB_PASSWORD", "value" : "***" } ], The user and bucket exist and have the right access policy, AFAICT. This is what I see in the log: 17:54:04 [] [] [] [] INFO ? mod-oai-pmh-3.12.0-SNAPSHOT.311 17:54:04 [] [] [] [] WARN ctApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'instancesDaoImpl': Unsatisfied dependency expressed through field 'folioS3Client'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'folioS3Client' defined in class path resource [org/folio/oaipmh/helpers/configuration/ErrorServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.folio.s3.client.FolioS3Client]: Factory method 'folioS3Client' threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials 17:54:04 [] [] [] [] INFO ? mod-oai-pmh-3.12.0-SNAPSHOT.311 17:54:04 [] [] [] [] INFO RestVerticle Init hook called with implemented class named org.folio.rest.impl.InitAPIs 17:54:05 [] [] [] [] INFO ? mod-oai-pmh-3.12.0-SNAPSHOT.311 17:54:04 [] [] [] [] ERROR RestVerticle Error creating bean with name 'instancesDaoImpl': Unsatisfied dependency expressed through field 'folioS3Client'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'folioS3Client' defined in class path resource [org/folio/oaipmh/helpers/configuration/ErrorServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.folio.s3.client.FolioS3Client]: Factory method 'folioS3Client' threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials 17:54:05 [] [] [] [] INFO ? mod-oai-pmh-3.12.0-SNAPSHOT.311 org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'instancesDaoImpl': Unsatisfied dependency expressed through field 'folioS3Client'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'folioS3Client' defined in class path resource [org/folio/oaipmh/helpers/configuration/ErrorServiceConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.folio.s3.client.FolioS3Client]: Factory method 'folioS3Client' threw exception; nested exception is java.lang.NoClassDefFoundError: software/amazon/awssdk/auth/credentials/AwsCredentials After which the container exits. Oleksandr Bozhko you might need to take a look, I'm not sure if you've tested with S3 or just with MinIO. |
| Comment by Oleksandr Bozhko [ 27/Jun/23 ] |
|
Wayne Schneider - update for mod-oai-pmh module to fix this issue is already present in the master branch. |
| Comment by Wayne Schneider [ 27/Jun/23 ] |
|
We have a successful test build. Log file for mod-oai-pmh is attached |
| Comment by Wayne Schneider [ 29/Jun/23 ] |
|
Changes are now merged (sorry for the delay), the next folio-snapshot build should contain the updated configuration for mod-oai-pmh. |