Texas A&M University Libraries experiences

Current infrastructure and deployment layout:

Duplicate instance of Folio Q3.2 on K8s infrastructure – in its own namespace using dedicated databases for Okapi and the Folio modules.
Three tenants with varying amounts and types of data:

Diku tenant - with default reference and sample data
Tamu tenant - with various data types from our migration work
Tern tenant - with various data types migrated from Voyager

Upgrade steps:

There is no current documentation outlining the exact steps or processes by which the data upgrade should happen. Used Taras’ working Google Doc as a starting point:

Automatic migrations dev and prod rollout plan

Built and deployed an upgraded Okapi v2.36.1 container.
Deployed Q4 modules on K8s infrastructure alongside the running duplicated Q3.2 containers in the same namespace, using new URLs for each.
Built the Folio Q4 Stripes front-end containers for each tenant and deployed on K8s infrastructure.
Create backup of Okapi and Folio module databases.
Built and ran update-deploy-<tenant> K8s Jobs, one for each of our tenants. The K8s Job runs a script that:

Pulls in the latest module descriptors from Index Data’s module registry to Okapi’s /_/proxy/pull/modules endpoint
Forms and Posts the new Q4 module’s deployment descriptors to Okapi’s /_/discovery/modules endpoint
Posts an install.json file with a list of the Q4 modules to be upgraded to, to the Okapi tenant endpoint with variables set to false per tenant:

/_/proxy/tenants/$TENANT_ID/install?deploy=false\&preRelease=false\&tenantParameters=loadSample%3D$SAMPLE_DATA%2CloadReference%3D$REF_DATA

Outcomes:

No tenant upgraded successfully. Each was left in an incomplete and unworkable state with no way to easily roll back without restoring a backup of the databases for Okapi and the Folio modules.
You could still log in to each tenant, but dependencies were not being met in the Settings page, and some app pages would not load without error.
Once the process errors out, no way to undo changes or have it continue.
Each tenant failed during the upgrade, logs from the create-deploy-<tenant> K8s Job erroring out differently:

Diku tenant error ( MODFISTO-83 - Getting issue details... STATUS ):

POST request for mod-finance-storage-4.1.1 /_/tenant failed with
[ " CREATE UNIQUE INDEX IF NOT EXISTS budget_name_idx_unique ON diku_mod_finance_storage.budget ( lower(f_unaccent(jsonb->>'name')) ) ;" ]
Key (lower(f_unaccent(jsonb ->> ‘name’::text)))=(gifts-one-time-fy19) is duplicated.

See workaround described in the Edelweiss Release Notes

Tern tenant error ( ERM-675 - Getting issue details... STATUS ):

POST request for mod-licenses-2.0.0 /_/tenant failed with {"error":500,"message":"liquibase.exception.ValidationFailedException: Validation Failed:\n 2 change sets check sum\n update-mod-license-1-9.groovy::1564401142529-1::sosguthorpe (generated) was: 7:63c1a0623a8be10c52f2f448c66a5586 but is now: 7:111ccc7d3f3d5941e70921eb2b1e2b59\n update-mod-license-1-9.groovy::1564401142529-2::sosguthorpe (generated) was: 7:ab6e491e0d2769cfdb7d012df7e78d3b but is now: 7:6f685298406f837ede71bd020ad2ab90\n","stackTrace":["liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:266)","liquibase.Liquibase.update(Liquibase.java:210)","liquibase.Liquibase.update(Liquibase.java:192)","liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:431)","org.grails.plugins.databasemigration.liquibase.GrailsLiquibase.performUpdate(GrailsLiquibase.groovy:83)","liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:388)","com.k_int.okapi.OkapiTenantAdminService.updateAccountSchema(OkapiTenantAdminService.groovy:228)","com.k_int.okapi.OkapiTenantAdminService.enableTenant(OkapiTenantAdminService.groovy:77)","com.k_int.okapi.TenantController.index(TenantController.groovy:40)","org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)","org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)","org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)","org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)","org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)","org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)","org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)","org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)","io.unde
rtow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)","org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)","grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter.doFilter(UpdateRequestContextHolderExceptionTranslationFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.groovy:54)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilt
er.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.SecurityRequestHolderFilter.doFilter(SecurityRequestHolderFilter.groovy:58)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)","org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.
doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)","io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)","io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65)","io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)","io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)","io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)","io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)","io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)","io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)","io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)","io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)",
"io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)","io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)","io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)","io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)","io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)","io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)","java.lang.Thread.run(Thread.java:748)"]}

Tamu tenant error ( ERM-675 - Getting issue details... STATUS ):

POST request for mod-licenses-2.0.0 /_/tenant failed with {"error":500,"message":"liquibase.exception.ValidationFailedException: Validation Failed:\n 2 change sets check sum\n update-mod-license-1-9.groovy::1564401142529-1::sosguthorpe (generated) was: 7:63c1a0623a8be10c52f2f448c66a5586 but is now: 7:111ccc7d3f3d5941e70921eb2b1e2b59\n update-mod-license-1-9.groovy::1564401142529-2::sosguthorpe (generated) was: 7:ab6e491e0d2769cfdb7d012df7e78d3b but is now: 7:6f685298406f837ede71bd020ad2ab90\n","stackTrace":["liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:266)","liquibase.Liquibase.update(Liquibase.java:210)","liquibase.Liquibase.update(Liquibase.java:192)","liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:431)","org.grails.plugins.databasemigration.liquibase.GrailsLiquibase.performUpdate(GrailsLiquibase.groovy:83)","liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:388)","com.k_int.okapi.OkapiTenantAdminService.updateAccountSchema(OkapiTenantAdminService.groovy:228)","com.k_int.okapi.OkapiTenantAdminService.enableTenant(OkapiTenantAdminService.groovy:77)","com.k_int.okapi.TenantController.index(TenantController.groovy:40)","org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)","org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)","org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)","org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)","org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)","org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)","org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)","org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)","io.unde
rtow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)","org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)","grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter.doFilter(UpdateRequestContextHolderExceptionTranslationFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.groovy:54)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilt
er.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.SecurityRequestHolderFilter.doFilter(SecurityRequestHolderFilter.groovy:58)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)","org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.
doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)","io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)","io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65)","io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)","io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)","io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)","io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)","io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)","io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)","io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)","io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)",
"io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)","io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)","io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)","io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)","io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)","io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)","java.lang.Thread.run(Thread.java:748)"]}

System Operations and Management SIG

Texas A&M University Libraries experiences

Current infrastructure and deployment layout:

Upgrade steps:

Outcomes:

Recommendations:

Related content