Texas A&M University Libraries experiences

Owned by jroot
Last updated: Jan 28, 2020 by Julian LadischVersion comment
2 min read

Current infrastructure and deployment layout:

  • Duplicate instance of Folio Q3.2 on K8s infrastructure – in its own namespace using dedicated databases for Okapi and the Folio modules.
  • Three tenants with varying amounts and types of data:
  1. Diku tenant - with default reference and sample data
  2. Tamu tenant - with various data types from our migration work
  3. Tern tenant - with various data types migrated from Voyager

Upgrade steps:

  • There is no current documentation outlining the exact steps or processes by which the data upgrade should happen. Used Taras’ working Google Doc as a starting point:

Automatic migrations dev and prod rollout plan

  • Built and deployed an upgraded Okapi v2.36.1 container.
  • Deployed Q4 modules on K8s infrastructure alongside the running duplicated Q3.2 containers in the same namespace, using new URLs for each.
  • Built the Folio Q4 Stripes front-end containers for each tenant and deployed on K8s infrastructure.
  • Create backup of Okapi and Folio module databases.
  • Built and ran update-deploy-<tenant> K8s Jobs, one for each of our tenants. The K8s Job runs a script that:
  1. Pulls in the latest module descriptors from Index Data’s module registry to Okapi’s /_/proxy/pull/modules endpoint
  2. Forms and Posts the new Q4 module’s deployment descriptors to Okapi’s /_/discovery/modules endpoint
  3. Posts an install.json file with a list of the Q4 modules to be upgraded to, to the Okapi tenant endpoint with variables set to false per tenant:
/_/proxy/tenants/$TENANT_ID/install?deploy=false\&preRelease=false\&tenantParameters=loadSample%3D$SAMPLE_DATA%2CloadReference%3D$REF_DATA

Outcomes:

  • No tenant upgraded successfully. Each was left in an incomplete and unworkable state with no way to easily roll back without restoring a backup of the databases for Okapi and the Folio modules.
  • You could still log in to each tenant, but dependencies were not being met in the Settings page, and some app pages would not load without error.
  • Once the process errors out, no way to undo changes or have it continue.
  • Each tenant failed during the upgrade, logs from the create-deploy-<tenant> K8s Job erroring out differently:


Diku tenant error ( MODFISTO-83 - Getting issue details... STATUS ):

POST request for mod-finance-storage-4.1.1 /_/tenant failed with
[ " CREATE UNIQUE INDEX IF NOT EXISTS budget_name_idx_unique ON diku_mod_finance_storage.budget ( lower(f_unaccent(jsonb->>'name')) ) ;" ]
Key (lower(f_unaccent(jsonb ->> ‘name’::text)))=(gifts-one-time-fy19) is duplicated.

See workaround described in the Edelweiss Release Notes

Tern tenant error ( ERM-675 - Getting issue details... STATUS ):

POST request for mod-licenses-2.0.0 /_/tenant failed with {"error":500,"message":"liquibase.exception.ValidationFailedException: Validation Failed:\n 2 change sets check sum\n update-mod-license-1-9.groovy::1564401142529-1::sosguthorpe (generated) was: 7:63c1a0623a8be10c52f2f448c66a5586 but is now: 7:111ccc7d3f3d5941e70921eb2b1e2b59\n update-mod-license-1-9.groovy::1564401142529-2::sosguthorpe (generated) was: 7:ab6e491e0d2769cfdb7d012df7e78d3b but is now: 7:6f685298406f837ede71bd020ad2ab90\n","stackTrace":["liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:266)","liquibase.Liquibase.update(Liquibase.java:210)","liquibase.Liquibase.update(Liquibase.java:192)","liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:431)","org.grails.plugins.databasemigration.liquibase.GrailsLiquibase.performUpdate(GrailsLiquibase.groovy:83)","liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:388)","com.k_int.okapi.OkapiTenantAdminService.updateAccountSchema(OkapiTenantAdminService.groovy:228)","com.k_int.okapi.OkapiTenantAdminService.enableTenant(OkapiTenantAdminService.groovy:77)","com.k_int.okapi.TenantController.index(TenantController.groovy:40)","org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)","org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)","org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)","org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)","org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)","org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)","org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)","org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)","io.unde
rtow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)","org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)","grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter.doFilter(UpdateRequestContextHolderExceptionTranslationFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.groovy:54)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilt
er.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.SecurityRequestHolderFilter.doFilter(SecurityRequestHolderFilter.groovy:58)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)","org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.
doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)","io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)","io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65)","io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)","io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)","io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)","io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)","io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)","io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)","io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)","io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)",
"io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)","io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)","io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)","io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)","io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)","io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)","java.lang.Thread.run(Thread.java:748)"]}

Tamu tenant error ( ERM-675 - Getting issue details... STATUS ):

POST request for mod-licenses-2.0.0 /_/tenant failed with {"error":500,"message":"liquibase.exception.ValidationFailedException: Validation Failed:\n 2 change sets check sum\n update-mod-license-1-9.groovy::1564401142529-1::sosguthorpe (generated) was: 7:63c1a0623a8be10c52f2f448c66a5586 but is now: 7:111ccc7d3f3d5941e70921eb2b1e2b59\n update-mod-license-1-9.groovy::1564401142529-2::sosguthorpe (generated) was: 7:ab6e491e0d2769cfdb7d012df7e78d3b but is now: 7:6f685298406f837ede71bd020ad2ab90\n","stackTrace":["liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:266)","liquibase.Liquibase.update(Liquibase.java:210)","liquibase.Liquibase.update(Liquibase.java:192)","liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:431)","org.grails.plugins.databasemigration.liquibase.GrailsLiquibase.performUpdate(GrailsLiquibase.groovy:83)","liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:388)","com.k_int.okapi.OkapiTenantAdminService.updateAccountSchema(OkapiTenantAdminService.groovy:228)","com.k_int.okapi.OkapiTenantAdminService.enableTenant(OkapiTenantAdminService.groovy:77)","com.k_int.okapi.TenantController.index(TenantController.groovy:40)","org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)","org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)","org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)","org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)","org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)","org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)","org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)","org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)","io.unde
rtow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)","org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)","org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)","grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter.doFilter(UpdateRequestContextHolderExceptionTranslationFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenticationFilter.groovy:54)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilt
er.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.groovy:64)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","grails.plugin.springsecurity.web.SecurityRequestHolderFilter.doFilter(SecurityRequestHolderFilter.groovy:58)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)","org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)","org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)","org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.
doFilter(FilterHandler.java:131)","io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)","io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)","io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)","io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)","io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65)","io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)","io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)","io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)","io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)","io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)","io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)","io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)","io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)",
"io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)","io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)","io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)","io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)","io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)","io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)","io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)","io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)","java.lang.Thread.run(Thread.java:748)"]}

Recommendations: