Poppy (R2 2023) - Bugfest env preparation - Issues/preparation we needed to resolve on environment

platform-complete Issues

Issue 1: UI Part Error

Log

+ yarn build --okapi http://platform-complete-2476-2.aws.indexdata.com:9130 --tenant diku ./output
yarn run v1.22.19
$ export NODE_OPTIONS="--max-old-space-size=4096 $NODE_OPTIONS"; stripes build stripes.config.js --okapi http://platform-complete-2476-2.aws.indexdata.com:9130 --tenant diku ./output
Building...
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/babel-plugin-lodash/lib/index.js:102:44)
assets by status 64.3 MiB [cached] 359 assets
Entrypoint css = style.b526979f755b53a115af.css bundle.css31d6cfe0d16ae931b73c.js 12 auxiliary assets
Entrypoint stripesConfig = style.6234cfcfa34d7e762b2c.css bundle.stripesConfiga5fbf2bcee5950541004.js 195 auxiliary assets
Entrypoint index = bundle.index1e46ec2cd6790246f76c.js
orphan modules 16.7 MiB [orphan] 8985 modules
runtime modules 13.1 KiB 16 modules
built modules 26.3 MiB (javascript) 384 KiB (css/mini-extract) 492 KiB (asset) [built]
  javascript modules 25.2 MiB
    cacheable modules 25.2 MiB 2624 modules
    + 6 modules
  css modules 384 KiB
    modules by path ./node_modules/@folio/ 356 KiB 631 modules
    modules by path ./node_modules/@k-int/stripes-kint-components/ 9.46 KiB 10 modules
    css ./node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[7].use[1]!./node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[7].use[2]!./node_modules/flexboxgrid2/flexboxgrid2.css 18.6 KiB [built] [code generated]
  asset modules 12.8 KiB (javascript) 492 KiB (asset) 83 modules
  json modules 1.15 MiB
    ./node_modules/moment-timezone/data/packed/latest.json 756 KiB [built] [code generated]
    ./node_modules/tai-password-strength/data/common-passwords.json 97.2 KiB [built] [code generated]
    ./node_modules/tai-password-strength/data/trigraphs.json 329 KiB [built] [code generated]
LOG from webpack.FileSystemInfo
<w> Managed item /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/stripes-config.js isn't a directory or doesn't contain a package.json (see snapshot.managedPaths option)
+ 14 hidden lines
WARNING in ./node_modules/@folio/consortia-settings/src/constants.js 42:33-52
Should not import the named export 'stripes'.'route' (imported as 'stripesConfig') from default-exporting module (only default export is available soon)
 @ ./node_modules/@folio/consortia-settings/src/Root.js
 @ ./node_modules/@folio/consortia-settings/src/index.js 1:0-33 1:0-33
 @ ./node_modules/stripes-config.js
ERROR in ./node_modules/@folio/stripes-core/src/init.js 4:0-46
Module not found: Error: Can't resolve 'react-dom/client' in '/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/stripes-core/src'
 @ ./node_modules/@folio/stripes-core/index.js
 @ ./node_modules/@folio/stripes-ui/src/index.js 1:0-43 2:0-4
ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/components/ListsTable/ListsTable.tsx
./node_modules/@folio/lists/src/components/ListsTable/ListsTable.tsx 2:30-38
[tsl] ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/components/ListsTable/ListsTable.tsx(2,31)
      TS7016: Could not find a declaration file for module 'lodash'. '/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/lodash/lodash.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/lodash` if it exists or add a new declaration (.d.ts) file containing `declare module 'lodash';`
 @ ./node_modules/@folio/lists/src/components/ListsTable/index.ts 1:0-42 1:0-42
 @ ./node_modules/@folio/lists/src/components/index.ts 3:0-42 3:0-42
 @ ./node_modules/@folio/lists/src/pages/lists/ListPage.tsx 11:0-59 34:177-188 39:32-42
 @ ./node_modules/@folio/lists/src/pages/lists/index.ts 1:0-27 1:0-27
 @ ./node_modules/@folio/lists/src/pages/index.ts 1:0-35 1:0-35
 @ ./node_modules/@folio/lists/src/index.tsx 5:0-86 12:40-48 14:40-59 16:40-52 18:40-54
 @ ./node_modules/stripes-config.js
ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/pages/createlist/components/QueryBuilder/QueryBuilder.tsx
./node_modules/@folio/lists/src/pages/createlist/components/QueryBuilder/QueryBuilder.tsx 5:21-29
[tsl] ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/pages/createlist/components/QueryBuilder/QueryBuilder.tsx(5,22)
      TS7016: Could not find a declaration file for module 'lodash'. '/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/lodash/lodash.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/lodash` if it exists or add a new declaration (.d.ts) file containing `declare module 'lodash';`
 @ ./node_modules/@folio/lists/src/pages/createlist/components/QueryBuilder/index.ts 1:0-46 1:0-46
 @ ./node_modules/@folio/lists/src/pages/createlist/components/MainCreateListForm/MainCreateListForm.tsx 7:0-47 26:32-44
 @ ./node_modules/@folio/lists/src/pages/createlist/components/MainCreateListForm/index.ts 1:0-58 1:0-58
 @ ./node_modules/@folio/lists/src/pages/createlist/components/index.ts 2:0-58 2:0-58
 @ ./node_modules/@folio/lists/src/pages/createlist/CreateListPage.tsx 43:0-68 86:32-48 87:28-46
 @ ./node_modules/@folio/lists/src/pages/createlist/index.ts 1:0-50 1:0-50
 @ ./node_modules/@folio/lists/src/pages/index.ts 2:0-46 2:0-46
 @ ./node_modules/@folio/lists/src/index.tsx 5:0-86 12:40-48 14:40-59 16:40-52 18:40-54
 @ ./node_modules/stripes-config.js
ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/pages/listInformation/ListInformationPage.tsx
./node_modules/@folio/lists/src/pages/listInformation/ListInformationPage.tsx 3:25-33
[tsl] ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/pages/listInformation/ListInformationPage.tsx(3,26)
      TS7016: Could not find a declaration file for module 'lodash'. '/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/lodash/lodash.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/lodash` if it exists or add a new declaration (.d.ts) file containing `declare module 'lodash';`
 @ ./node_modules/@folio/lists/src/pages/listInformation/index.ts 1:0-38 1:0-38
 @ ./node_modules/@folio/lists/src/pages/index.ts 3:0-56 3:0-56
 @ ./node_modules/@folio/lists/src/index.tsx 5:0-86 12:40-48 14:40-59 16:40-52 18:40-54
 @ ./node_modules/stripes-config.js
ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/pages/lists/ListPage.tsx
./node_modules/@folio/lists/src/pages/lists/ListPage.tsx 4:21-29
[tsl] ERROR in /home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/@folio/lists/src/pages/lists/ListPage.tsx(4,22)
      TS7016: Could not find a declaration file for module 'lodash'. '/home/jenkins/workspace/io-org_platform-complete_PR-2476/node_modules/lodash/lodash.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/lodash` if it exists or add a new declaration (.d.ts) file containing `declare module 'lodash';`
 @ ./node_modules/@folio/lists/src/pages/lists/index.ts 1:0-27 1:0-27
 @ ./node_modules/@folio/lists/src/pages/index.ts 1:0-35 1:0-35
 @ ./node_modules/@folio/lists/src/index.tsx 5:0-86 12:40-48 14:40-59 16:40-52 18:40-54
 @ ./node_modules/stripes-config.js
5 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.
webpack 5.68.0 compiled with 5 errors and 1 warning in 229884 ms
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
script returned exit code 1"

Resolution: To resolve this issue, we performed the following actions:

Copy-pasted the devDependencies and resolutions from the snapshot branch to the R2-2023 branch.

Corrected the package.json:

{
  "name": "@folio/platform-complete",
  "version": "3.9.0",
  "license": "Apache-2.0",
  "scripts": {
    "build": "export NODE_OPTIONS=\"--max-old-space-size=4096 $NODE_OPTIONS\"; stripes build stripes.config.js",
    "stripes": "stripes",
    "start": "stripes serve stripes.config.js",
    "build-module-descriptors": "stripes mod descriptor stripes.config.js --output ./ModuleDescriptors",
    "local": "f=stripes.config.js; test -f $f.local && f=$f.local; echo Using config $f; stripes serve $f",
    "test": "echo 'No unit tests implemented'",
    "lint": "eslint test/ui-testing"
  },
  "dependencies": {
    "@folio/acquisition-units": "5.0.0",
    "@folio/agreements": "10.0.0",
    "@folio/bulk-edit": "4.0.0",
    "@folio/calendar": "10.0.0",
    "@folio/checkin": "9.0.0",
    "@folio/checkout": "10.0.0",
    "@folio/circulation": "9.0.0",
    "@folio/circulation-log": "4.0.0",
    "@folio/courses": "6.0.2",
    "@folio/consortia-settings" : "1.0.0",
    "@folio/dashboard": "5.0.0",
    "@folio/data-export": "6.0.0",
    "@folio/data-import": "7.0.0",
    "@folio/developer": "7.0.0",
    "@folio/eholdings": "9.0.0",
    "@folio/erm-comparisons": "6.0.0",
    "@folio/erm-usage": "8.0.0",
    "@folio/export-manager": "3.0.0",
    "@folio/finance": "5.0.0",
    "@folio/gobi-settings": "3.0.0",
    "@folio/handler-stripes-registry": "2.0.0",
    "@folio/inventory": "10.0.0",
    "@folio/invoice": "5.0.0",
    "@folio/ldp": "2.0.0",
    "@folio/licenses": "10.0.0",
    "@folio/lists" : "2.0.0",
    "@folio/local-kb-admin": "7.0.0",
    "@folio/marc-authorities": "4.0.0",
    "@folio/myprofile": "9.0.0",
    "@folio/notes": "9.0.0",
    "@folio/oai-pmh": "5.0.0",
    "@folio/orders": "5.0.0",
    "@folio/organizations": "5.0.0",
    "@folio/plugin-bursar-export": "2.4.0",
    "@folio/plugin-create-inventory-records": "4.0.0",
    "@folio/plugin-eusage-reports": "3.0.0",
    "@folio/plugin-find-agreement": "10.0.0",
    "@folio/plugin-find-authority": "3.0.0",
    "@folio/plugin-find-contact": "5.0.0",
    "@folio/plugin-find-eresource": "6.0.0",
    "@folio/plugin-find-erm-usage-data-provider": "6.0.0",
    "@folio/plugin-find-fund": "3.0.0",
    "@folio/plugin-find-import-profile": "7.0.0",
    "@folio/plugin-find-instance": "7.0.0",
    "@folio/plugin-find-interface": "5.0.0",
    "@folio/plugin-find-license": "10.0.0",
    "@folio/plugin-find-organization": "5.0.0",
    "@folio/plugin-find-package-title": "6.0.0",
    "@folio/plugin-find-po-line": "5.0.0",
    "@folio/plugin-find-user": "7.0.0",
    "@folio/plugin-query-builder" : "1.0.0",
    "@folio/quick-marc": "7.0.1",
    "@folio/receiving": "4.0.0",
    "@folio/remote-storage": "5.0.0",
    "@folio/requests": "9.0.0",
    "@folio/servicepoints": "7.0.0",
    "@folio/service-interaction" : "1.0.0",
    "@folio/stripes": "9.0.0",
    "@folio/stripes-authority-components": "3.0.0",
    "@folio/stripes-erm-components": "9.0.0",
    "@folio/tags": "8.0.2",
    "@folio/tenant-settings": "8.0.0",
    "@folio/users": "10.0.0",
    "final-form": "^4.20.7",
    "final-form-arrays": "^3.0.2",
    "moment": "~2.29.0",
    "react": "~18.2.0",
    "react-dom": "~18.2.0",
    "react-final-form": "^6.5.9",
    "react-final-form-arrays": "^3.1.3",
    "react-intl": "^6.4.7",
    "react-query": "^3.13.0",
    "react-redux": "^8.0.5",
    "react-router": "^5.2.0",
    "react-router-dom": "^5.2.0",
    "react-titled": "^2.0.0",
    "redux": "^4.2.1",
    "redux-observable": "^1.2.0",
    "rxjs": "^6.6.7",
    "zustand": "^4.1.1"
  },
  "devDependencies": {
    "@folio/stripes-cli": "3.0.0",
    "@types/lodash": "^4.14.197",
    "eslint": "^6.2.1",
    "lodash": "^4.17.5"
  },
  "resolutions": {
    "@rehooks/local-storage": "2.4.4",
    "colors": "1.4.0",
    "final-form": "^4.20.4",
    "minimist": "^1.2.3",
    "moment": "~2.29.0",
    "redux-form": "^8.0.0"
  }
}

Issue 2: Removal of Unreleased Modules

We removed the following modules because they were not ready for release:

@folio/plugin-bursar-export: 2.4.0
@folio/service-interaction: 1.0.0

Adding a New Module

We added edge-courses-1.3.0 as a new module.

Review of Release Notes

We reviewed the release notes for the Poppy release and adjusted our configs. Here are the changes made to specific modules:

mod-agreements

{
  "name": "JAVA_OPTS",
  "value": "$JAVA_OPTS_VERTEX_LOGGER $JAVA_OPTS_HEAP_DUMP -XX:+UseContainerSupport -XX:MaxRAMPercentage=67.0 -XX:+PrintFlagsFinal"
}

mod-lists

{
  "containerDefinitions": [
    {
      "memory": 3000,
      "memoryReservation": 2600,
      "cpu": 128,
      "environment": [
        {
          "name": "JAVA_OPTS",
          "value": "$JAVA_OPTS_VERTEX_LOGGER $JAVA_OPTS_HEAP_DUMP -XX:MetaspaceSize=384m -XX:MaxMetaspaceSize=512m -Xmx2048m"
        },
        {
          "name": "LIST_APP_S3_BUCKET",
          "value": "$CLUSTER-list-exports-us-west-2"
        },
        {
          "name": "AWS_REGION",
          "value": "us-west-2"
        },
        {
          "name": "AWS_URL",
          "value": "https://s3-us-west-2.amazonaws.com"
        }
      ]
    }
  ]
}

mod-login-saml and mod-login

{
  "name": "LOGIN_COOKIE_SAMESITE",
  "value": "Lax"
}

Deployment of Modules

We encountered various issues during module deployment. Here are the details and their resolutions:

mod-erm-usage-harvester: Environment variable 'OKAPI_URL' not set.

12:47:57 [] [] [] [] ERROR Launcher             Environment variable 'OKAPI_URL' not set

Resolution: Added the 'OKAPI_URL' variable for the taskDefenition.

{
  "name": "OKAPI_URL",
  "value": "http://pvt.lb.$CLUSTER.$DOMAIN_PREFIX.$REGION:$OKAPI_PORT"
}

mod-users: Failed to construct Kafka consumer.

12:22:58 [] [] [] [] ERROR RestVerticle         Failed to construct kafka consumer

Resolution: Added environment variables for Kafka configuration.

{
  "containerDefinitions": [
    {
      "environment": [
        {
          "name": "KAFKA_HOST",
          "value": "kafka.$CLUSTER.folio-eis.$REGION"
        },
        {
          "name": "KAFKA_PORT",
          "value": "$KAFKA_PORT"
        }
      ]
    }
  ]
}

mod-consortia: Out of Memory (OOM) error.
Resolution: Increased memory settings.
```
{
  "memory": 2048,
  "memoryReservation": 1802,
}
```

mod-fqm-manager: The web server failed to start due to a port conflict.

Web server failed to start. Port 8081 was already in use.

Resolution: Updated environment settings.

{
  "containerDefinitions": [
    {
      "environment": [
        {
          "name": "JAVA_ARGS",
          "value": "--server.port=$INTERNAL_DOCKER_PORT -Dlog.level=info"
        }
      ]
    }
  ]
}

mod-lists: The web server failed to start due to port conflict.

Web server failed to start. Port 8081 was already in use.

Resolution: Added Java arguments to resolve the port conflict.

{
  "name": "JAVA_ARGS",
  "value": "--server.port=$INTERNAL_DOCKER_PORT -Dlog.level=info"
}

edge-fqm: Invalid taskDefenition.

12:41:00 [] [] [] [] ERROR SpringApplication    Application run failed
org.springframework.context.ApplicationContextException: Unable to start web server
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:164) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:602) ~[spring-context-6.0.11.jar!/:6.0.11]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) [spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:436) [spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) [spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) [spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) [spring-boot-3.1.2.jar!/:3.1.2]
	at org.folio.fqm.edge.EdgeFqmApplication.main(EdgeFqmApplication.java:12) [classes!/:1.0.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [ms.jar:1.0.0]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) [ms.jar:1.0.0]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [ms.jar:1.0.0]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) [ms.jar:1.0.0]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:142) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:489) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:211) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:183) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-3.1.2.jar!/:3.1.2]
	... 16 more
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'defaultEdgeSecurityFilter' defined in URL [jar:file:/usr/ms/ms.jar!/BOOT-INF/lib/edge-common-spring-2.3.0.jar!/org/folio/edgecommonspring/filter/EdgeSecurityFilter.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'securityManagerService': Invocation of init method failed
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:210) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:173) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:168) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAdaptableBeans(ServletContextInitializerBeans.java:153) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:86) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:235) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:52) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4886) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:846) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:241) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:428) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:918) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.startup.Tomcat.start(Tomcat.java:485) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:489) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:211) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:183) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-3.1.2.jar!/:3.1.2]
	... 16 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityManagerService': Invocation of init method failed
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:222) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:888) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:210) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:173) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:168) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAdaptableBeans(ServletContextInitializerBeans.java:153) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:86) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:235) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:52) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4886) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:846) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:241) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:428) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:918) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.startup.Tomcat.start(Tomcat.java:485) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:489) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:211) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:183) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-3.1.2.jar!/:3.1.2]
	... 16 more
Caused by: org.folio.edge.api.utils.exception.AuthorizationException: Failed to load secure store properties
	at org.folio.edge.api.utils.util.PropertiesUtil.getProperties(PropertiesUtil.java:39) ~[edge-api-utils-1.3.0.jar!/:?]
	at org.folio.edgecommonspring.security.SecurityManagerService.init(SecurityManagerService.java:54) ~[edge-common-spring-2.3.0.jar!/:2.3.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:888) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205) ~[spring-beans-6.0.11.jar!/:6.0.11]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:210) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:173) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAsRegistrationBean(ServletContextInitializerBeans.java:168) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addAdaptableBeans(ServletContextInitializerBeans.java:153) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:86) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:261) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:235) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:52) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4886) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:846) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:241) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:428) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:918) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.apache.catalina.startup.Tomcat.start(Tomcat.java:485) ~[tomcat-embed-core-10.1.11.jar!/:?]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:489) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:211) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:183) ~[spring-boot-3.1.2.jar!/:3.1.2]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-3.1.2.jar!/:3.1.2]
	... 16 more

Resolution: Added container definitions to match other modules' deployment templates.

{
  "containerDefinitions": [
    {
      "environment": [
        {
          "name": "JAVA_OPTS",
          "value": "$JAVA_OPTS -Dokapi_url=http://pvt.lb.$CLUSTER.$DOMAIN_PREFIX.$REGION:$OKAPI_PORT -Dhttp.port=$CONTAINER_PORT -Dsecure_store=AwsSsm -Dsecure_store_props=/usr/ms/aws_ss.properties -Dlog_level=INFO"
        }
      ]
    }
  ]
}

Migrations issues

mod-entities-links without LoadR and LoadS: This issue resulted in a Liquibase exception.

java.lang.Exception: Failed to install modules POST request for mod-entities-links-2.0.0 /_/tenant failed with 500: {"errors":[{"message":"liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset db/changelog/changes/v2.0/replace_authority_data.xml::MODELINKS-2@@add-authority_data_stat-authority_id-foreign-key::Mukhiddin_Yusupov:\n     Reason: liquibase.exception.DatabaseException: ERROR: insert or update on table \"authority_data_stat\" violates foreign key constraint \"fk_authority_data_stat_authority_id\"\n  Detail: Key (authority_id)=(0ebb6490-4c8e-4366-a26c-43fb23d57728) is not present in table \"authority\". [Failed SQL: (0) ALTER TABLE fs09000000_mod_entities_links.authority_data_stat ADD CONSTRAINT fk_authority_data_stat_authority_id FOREIGN KEY (authority_id) REFERENCES fs09000000_mod_entities_links.authority (id)]","type":"TenantUpgradeException","code":"unknown"}],"total_records":1}

Resolution: Removed records from the <tenant>_mod_entities_links.authority_data_stat and instance_authority_link tables.

mod-finance-storage (with loadRef and loadSampl = true): Tenant operation failed due to API responses.

{
  "id": "mod-finance-storage-8.5.0",
  "from": "mod-finance-storage-8.4.2",
  "action": "enable",
  "message": "Tenant operation failed for module mod-finance-storage-8.5.0: GET http://mod-finance-storage-b.pbf.folio-eis.us-west-2:8051/mod-finance-storage/finance-storage/expense-classes/1bcc3247-99bf-4dca-9b0f-7bc51a2998c2 returned status 404: Not found POST http://mod-finance-storage-b.pbf.folio-eis.us-west-2:8051/mod-finance-storage/finance-storage/expense-classes returned status 400: {\"message\":\"Field Name must be unique\",\"code\":\"uniqueFieldExpenseClassNameError\",\"parameters\":[{\"key\":\"field\",\"value\":\"Name\"},{\"key\":\"entity\",\"value\":\"ExpenseClass\"}]}",
  "stage": "invoke"
}

Resolution: No action was taken as it was reported as a bug - MODFISTO-449 - Getting issue details... STATUS
It appeared when the migration for mod-entities-links failed and we tried to enable modules separately with the loadReference/loadSample flag.

mod-circulations storage: Infinity enable.

09:49:09 [728984/proxy;103164/tenant] [fs09000003] [] [mod_circulation_storage] ERROR ?                    Unhandled exception
java.lang.NullPointerException: null
	at java.util.HashMap.merge(Unknown Source) ~[?:?]
	at java.util.stream.Collectors.lambda$toMap$68(Unknown Source) ~[?:?]
	at java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source) ~[?:?]
	at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
	at org.folio.service.migration.RequestSearchFieldsMigrationService.saveCallNumbers(RequestSearchFieldsMigrationService.java:125) ~[ms.jar:?]
	at org.folio.service.migration.RequestSearchFieldsMigrationService.lambda$findCallNumbers$5(RequestSearchFieldsMigrationService.java:112) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onSuccess(FutureImpl.java:262) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[ms.jar:?]
	at io.vertx.core.impl.future.Mapping.onSuccess(Mapping.java:40) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[ms.jar:?]
	at io.vertx.core.impl.future.FixedMapping.onSuccess(FixedMapping.java:31) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[ms.jar:?]
	at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onSuccess(FutureImpl.java:262) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[ms.jar:?]
	at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[ms.jar:?]
	at io.vertx.core.impl.future.SucceededFuture.addListener(SucceededFuture.java:88) ~[ms.jar:?]
	at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:43) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onSuccess(FutureImpl.java:262) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[ms.jar:?]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[ms.jar:?]
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[ms.jar:?]
	at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49) ~[ms.jar:?]
	at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:41) ~[ms.jar:?]
	at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23) ~[ms.jar:?]
	at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:397) ~[ms.jar:?]
	at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:384) ~[ms.jar:?]
	at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362) ~[ms.jar:?]
	at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329) ~[ms.jar:?]
	at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:291) ~[ms.jar:?]
	at io.vertx.ext.web.client.impl.HttpContext.lambda$null$7(HttpContext.java:507) ~[ms.jar:?]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[ms.jar:?]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246) ~[ms.jar:?]
	at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43) ~[ms.jar:?]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[ms.jar:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[ms.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[ms.jar:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[ms.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[ms.jar:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[ms.jar:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[ms.jar:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]

Resolution: Updated the mod_inventory_storage table to set the 'effectiveShelvingOrder' field to a specific value.

update <tenant>_mod_inventory_storage.item 
set jsonb = jsonb_set(jsonb, '{effectiveShelvingOrder}', '"migration-workaround-1"')
where jsonb->>'effectiveShelvingOrder' is null;

mod-consortia: Excluded deployment for non-consortia tenants.

Other Issues

okapi:

11:33:26 [] [] [] [] ERROR ?                    Failed to initialize clustered Vert.x
java.lang.IllegalStateException: Node failed to start!
	at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126) ~[ms.jar:?]
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217) ~[ms.jar:?]
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:196) ~[ms.jar:?]
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:134) ~[ms.jar:?]
	at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61) ~[ms.jar:?]
	at io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.lambda$join$1(HazelcastClusterManager.java:135) ~[ms.jar:?]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) ~[ms.jar:?]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) ~[ms.jar:?]
	at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) ~[ms.jar:?]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[ms.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[ms.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
11:33:26 [] [] [] [] ERROR ?                    Failed to leave cluster
java.lang.NullPointerException: Cannot invoke "io.vertx.spi.cluster.hazelcast.impl.SubsMapHelper.close()" because "this.subsMapHelper" is null
	at io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.lambda$leave$5(HazelcastClusterManager.java:255) ~[ms.jar:?]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) ~[ms.jar:?]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) ~[ms.jar:?]
	at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) ~[ms.jar:?]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[ms.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[ms.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
11:33:26 [] [] [] [] ERROR ?                    Node failed to start!
java.lang.IllegalStateException: Node failed to start!
	at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126) ~[ms.jar:?]
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217) ~[ms.jar:?]
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:196) ~[ms.jar:?]
	at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:134) ~[ms.jar:?]
	at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61) ~[ms.jar:?]
	at io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.lambda$join$1(HazelcastClusterManager.java:135) ~[ms.jar:?]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:180) ~[ms.jar:?]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:277) ~[ms.jar:?]
	at io.vertx.core.impl.ContextBase.lambda$internalExecuteBlocking$2(ContextBase.java:199) ~[ms.jar:?]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[ms.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[ms.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

Resolution: Scaled down the Okapi service with the previous version of Okapi to 0 and then updated Okapi.

folio_users: Issue UIU-2971.
instance-authority.linking-rules.item.patch: Missing for the admin user.
mod-authtoken: RTR (/login-with-expiry) isn't working for current UI bundles. Waiting for the fixes from the dev side.
mod-data-export: Issue BF-573.
mod-circulations: Issue BF-574.
mod-data-export-spirng: Environment variable 'OKAPI_URL' not set.
The data-export jobs cannot start.

Resolution: Added the 'OKAPI_URL' variable for the taskDefenition.
```
{
  "name": "OKAPI_URL",
  "value": "http://pvt.lb.$CLUSTER.$DOMAIN_PREFIX.$REGION:$OKAPI_PORT"
}
```
mod-di-converter-storage (check in and check out issue)

Definition: Subsequent check in requests were taking a long time, up to 30 seconds:

Resolution:

Identified a kafka process time as bottleneck
Request a new AWS MSK cluster
Update all R53 records to point to the new AWS MSK cluster
Perform a re-install=true operation on the below kafka using modules list:

Modules list

[
    {
        "id": "mod-orders-storage-13.6.0",
        "action": "enable"
    },
    {
        "id": "mod-invoice-5.7.2",
        "action": "enable"
    },
    {
        "id": "mod-data-export-spring-3.0.2",
        "action": "enable"
    },
    {
        "id": "mod-quick-marc-5.0.1",
        "action": "enable"
    },
    {
        "id": "mod-inventory-20.1.6",
        "action": "enable"
    },
    {
        "id": "mod-search-3.0.5",
        "action": "enable"
    },
    {
        "id": "mod-data-import-3.0.7",
        "action": "enable"
    },
    {
        "id": "mod-remote-storage-3.0.1",
        "action": "enable"
    },
    {
        "id": "mod-pubsub-2.11.3",
        "action": "enable"
    },
    {
        "id": "mod-data-export-worker-3.1.2",
        "action": "enable"
    },
    {
        "id": "mod-source-record-manager-3.7.7",
        "action": "enable"
    },
    {
        "id": "mod-inventory-storage-27.0.4",
        "action": "enable"
    },
    {
        "id": "mod-source-record-storage-5.7.5",
        "action": "enable"
    },
    {
        "id": "mod-circulation-storage-17.1.7",
        "action": "enable"
    },
    {
        "id": "mod-di-converter-storage-2.1.5",
        "action": "enable"
    },
    {
        "id": "mod-invoice-storage-5.7.0",
        "action": "enable"
    }
]

instance-relationship-types reference data to DB (mod-inventory-storage module) on ECS Poppy Bugfest

Definition:

Resolution:

Collect all sample & reference data from module GitHub repo: Sample & Reference data
Update single table with collected data
Execute SQL query to propagate data across all ECS tenants:

Query

INSERT INTO tenantName_mod_inventory_storage.instance_relationship_type
(SELECT id, jsonb, creation_date, created_by
	FROM cs00000int_mod_inventory_storage.instance_relationship_type);

FAT-9196 - Getting issue details... STATUS
UIU-2971 - Getting issue details... STATUS
ERM-3136 - Getting issue details... STATUS
BF-582 - Getting issue details... STATUS
BF-593 - Getting issue details... STATUS
MODSOURCE-726 - Getting issue details... STATUS
BF-584 - Getting issue details... STATUS
BF-586 - Getting issue details... STATUS
BF-583 - Getting issue details... STATUS
MODINVSTOR-1129 - Getting issue details... STATUS
BF-608 - Getting issue details... STATUS

FTP connection issues on poppy bugfest

Definition: We passed test cases to connect to FPT server on 27 of November and were able to connect to FTP from poppy bugfest env, now we can not.

Here is connection params:

"Location type*" dropdown - "FTP" option
"Upload location" field - "ftp://ftp.ci.folio.org"
"Port" field - fill in "22"
"Directory" field - "/files/invoices"
"Format*" dropdown - "JSON" option
"Username" field - "folio"
"Password" field - "Ffx29%pu"

In module's log we see:

2023-11-02T07:50:24.292Z 07:50:24 [] [] [] [] ERROR FtpUploadService Error Connecting FTP server ftp.ci.folio.org on port 22
2023-11-02T07:50:24.292Z org.apache.commons.net.MalformedServerReplyException: Could not parse response code.
2023-11-02T07:50:24.292Z Server Reply: SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.5

Resolution: Conducted a meeting, describing that FTP port is 21, SFTP port is 22, in our case FTP is in use.