[FOLIO-2867] FOLIO Confluence wiki not offering meeting notes template Created: 06/Nov/20 Updated: 29/Apr/21 Resolved: 29/Apr/21 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | TBD |
| Reporter: | Kristin Martin | Assignee: | Peter Murray |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||
| Sprint: | DevOps: Sprint 105, DevOps: Sprint 103, DevOps: Sprint 102, DevOps Sprint 109, DevOps Sprint 107, DevOps Sprint 108, DevOps Sprint 110, DevOps Sprint 111, DevOps Sprint 112, DevOps: Sprint 104, DevOps Sprint 106, DevOps Sprint 113 | ||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||
| Description |
|
Overview: Steps to Reproduce:
Expected Results: Actual Results: Additional Information: Interested parties: |
| Comments |
| Comment by Peter Murray [ 06/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
In this 2018 answer on the Confluence community, a reply says the meeting notes blueprint is part of the "Confluence Business Blueprints - Plugin" page. I have tried disabling that system plugin and re-enabling it; no effect. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 07/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Meeting Notes Blueprint has more details about the meeting notes blueprint, but nothing that seems helpful in resolving this. I think the next step is to try rebooting the Confluence server to see if that clears up the issue. I'm going to do that now while it is off hours. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 07/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hmm...that didn't do it. Attepmpting to clear plugin cache. # systemctl start confluence.service # cd /atlassian/var/application-data/confluence # rm -rf plugins-osgi-cache plugins-temp bundled-plugins plugins-cache # systemctl start confluence.service | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 07/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hmmm...that didn't work either. John Malconian: do you have any ideas? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by John Malconian [ 18/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I see many stack trace errors in the confluence log file that look like this: 2020-11-18 20:59:02,799 ERROR [http-nio-127.0.0.1-8090-exec-93] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@1d4e865b -- url: /display/DD/Decision+log | traceId: 4f913c631079d71d | userName: malc com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.confluence.plugins.confluence-feature-discovery-plugin] ... Caused by: java.lang.RuntimeException: Could not read fields for table AO_6384AB_DISCOVERED ... Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by John Malconian [ 18/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I think setting the MySQL DB parameter 'innodb_default_row_format' to 'DYNAMIC' might resolve the error above. According to the AWS RDS console, it appears it is not currently set to anything which is weird because the default should be 'DYNAMIC'. More investigation needed. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by John Malconian [ 18/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
More info: https://dev.mysql.com/doc/refman/8.0/en/innodb-row-format.html | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 24/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I wonder if you might be onto something. I followed the suggestion to use the `utf8mb4` character set, and I found this issue for MariaDB: https://jira.mariadb.org/browse/MDEV-14904 I'll have to take the database down to make the change, so I'll wait until later today to try it out. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 24/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hmmm – good theory, but it didn't pan out. I'm going to go through and make sure that the database changes specified in the upgrade instructions actually happened. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 25/Nov/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Okay, this is kicking my butt. I've tried:
Also note that this issue may also be related to an inability to create a new space in Confluence. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 15/Dec/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Still can't figure it out. Have posted a question to the Atlassian Confluence community board seeking suggestions for solutions. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 24/Dec/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Daniel Ebers on the Atlassian Community site had two suggestions:
Verified that we are using 8.0.21; the file is located at /atlassian/opt/confluence/confluence/WEB-INF/lib/mysql-connector-java-8.0.21.jar. Ran find /confluence -name "mysql-connector*" -print to make sure there wasn't anything funny elsewhere. Lo and behold, there is a /atlassian/opt/confluence/confluence/mysql-connector-java-5.1.44-bin.jar! Could it be as easy as that? Stopped confluence, moved it out of the way, started confluence...same symptoms.
Going to try this next. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 27/Dec/20 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Was experiencing timeouts trying to access https://wiki.folio.org/admin/backup.do, so I stood up an ssh tunnel to Tomcat ( {ssh -N -L 8090:127.0.0.1:8090 wiki.folio.org}) so I could access it from the laptop as http://localhost:8090/admin/backup.do.
There are numerous log messages in "atlassian-confluence.log". Stack traces reduced to what seems like the most important lines. 2020-12-27 16:38:38,191 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Backward compatibility for export: Checking... 2020-12-27 16:38:38,192 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8401: "Upgrade to build number: 8401" - Ok 2020-12-27 16:38:38,193 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8301: "Add unmanaged unique constraints missing in CONFSERVER-38706 and CONFSERVER-58261" - Ok 2020-12-27 16:38:38,193 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok 2020-12-27 16:38:38,194 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok 2020-12-27 16:38:38,194 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok 2020-12-27 16:38:38,194 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7901: "Upgrade "synchrony.btf.off" property" - Ok 2020-12-27 16:38:38,195 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7801: "Disables backup job" - Ok 2020-12-27 16:38:38,195 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7701: "Fixes wrong spaces for drafts" - Ok 2020-12-27 16:38:38,201 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7601: "Upgrade custom colour scheme to ADGS" - Ok 2020-12-27 16:38:38,201 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7400: "Upgrade to build number: 7400" - Ok 2020-12-27 16:38:38,201 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7202: "Remove the legacy Plugin License Storage plugin from the user installed plugins" - Ok 2020-12-27 16:38:38,202 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7201: "Upgrade to build number: 7201" - Ok 2020-12-27 16:38:38,202 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7110: "Reindex all blogs." - Ok 2020-12-27 16:38:38,202 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7109: "Drop and update some low performance indexes on EVENTS table for Synchrony" - Ok 2020-12-27 16:38:38,203 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7108: "Update index sp_comp_idx when the definition is not correct" - Ok 2020-12-27 16:38:38,203 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7106: "Drop the hibernate 2 constraints in preparation for them to be re-created in the hibernate 5 naming style" - Ok 2020-12-27 16:38:38,203 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7105: "Drop or update some low performance indexes on CONTENT" - Ok 2020-12-27 16:38:38,204 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7104: "Set the mime type for csv files to text/csv" - Ok 2020-12-27 16:38:38,204 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7101: "Migrate over Documentation theme settings to Custom Page Settings in the default theme" - Ok 2020-12-27 16:38:38,209 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6439: "Reindexes all PersonalInformation objects, creating them for any users that do not have one." - Ok 2020-12-27 16:38:38,210 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6430: "Upgrade to build number: 6430" - Ok 2020-12-27 16:38:38,215 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6424: "Adds missing multi-column unique constraints to the Crowd database tables" - Ok 2020-12-27 16:38:38,221 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6423: "Cleaning data that violates constraints we want to add in the next migration task" - Ok 2020-12-27 16:38:38,221 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6422: "Upgrade to build number: 6422" - Ok 2020-12-27 16:38:38,227 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6412: "Delete all FileStore IDs" - Ok 2020-12-27 16:38:38,227 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6406: "Reindex all spaces" - Ok 2020-12-27 16:38:38,228 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6403: "Upgrade to build number: 6403" - Ok 2020-12-27 16:38:38,228 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6193: "Rebuild the Confluence favourites index" - Ok 2020-12-27 16:38:38,228 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6193: "Rebuild the Confluence favourites index" - Ok 2020-12-27 16:38:38,229 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6189: "Upgrade to build number: 6189" - Ok 2020-12-27 16:38:38,235 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6180: "Adds a multi-column index to content table that speeds up many common queries." - Ok 2020-12-27 16:38:38,235 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6175: "Upgrade to build number: 6175" - Ok 2020-12-27 16:38:38,236 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6169: "Upgrade to build number: 6169" - Not Ok 2020-12-27 16:38:38,236 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Backward compatibility for export: "Upgrade to build number: 6169" prevents compatibility before 6169 2020-12-27 16:40:34,931 ERROR [http-nio-127.0.0.1-8090-exec-5] [host.util.concurrent.LockedTaskExecutor] executeUnderLock Failed whilst getting last modified date -- url: /plugins/servlet/mobile | traceId: 741012c213b7de50 | userName: anonymous java.util.concurrent.ExecutionException: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.executeUnderLock(LockedTaskExecutor.java:96) at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getLastModified(LocalRegistrationServiceImpl.java:192) at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getCacheValue(LocalRegistrationServiceImpl.java:223) at com.atlassian.mywork.host.web.MobileNavigationContextProvider.getContextMap(MobileNavigationContextProvider.java:50) at com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel.getHtml(DefaultWebPanelModuleDescriptor.java:134) at com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebInterfaceManager$ExceptionHandlingWebPanel.getHtml(ConfluenceWebInterfaceManager.java:133) .... Caused by: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) .... 2020-12-27 16:40:34,934 ERROR [http-nio-127.0.0.1-8090-exec-5] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@6954df81 -- url: /plugins/servlet/mobile | traceId: 741012c213b7de50 | userName: anonymous org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) 2020-12-27 16:40:34,936 WARN [http-nio-127.0.0.1-8090-exec-5] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1609086934914, threadId=191, threadName='http-nio-127.0.0.1-8090-exec-5', userId='<unknown>', type='web-request', summary='/plugins/servlet/mobile?contentId=25723434'} -- url: /plugins/servlet/mobile | traceId: 741012c213b7de50 | userName: anonymous 2020-12-27 16:47:18,977 INFO [Caesium-1-1] [agent.service.check.StaleChecksCleaner] info Cleaning stale checks 2020-12-27 16:48:47,191 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1609087727050 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ; ; ; {"d irectory":"/atlassian/var/application-data/confluence","freeInMegabytes":6008,"totalInMegabytes":50268,"minimumInMegabytes":8192} 2020-12-27 16:48:47,196 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-12-27T16:48:47.050Z Component 'Operating System' alerted 'Low free disk space' (det ails: {"directory":"/atlassian/var/application-data/confluence","freeInMegabytes":6008,"totalInMegabytes":50268,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not- detected"}) 2020-12-27 16:49:02,124 WARN [http-nio-127.0.0.1-8090-exec-43] [activeobjects.confluence.backup.LoggingBackupProgressMonitor] beginBackup Begin Active objects backup, c hange log level to INFO for com.atlassian.activeobjects.confluence.backup for more detailed logging. -- referer: http://localhost:8090/admin/backup.action | url: /admin/dobackup.action | traceId: ff624f240f8ea6ca | userName: peter | action: dobackup 2020-12-27 16:49:02,422 ERROR [http-nio-127.0.0.1-8090-exec-43] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian. plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@13fcf9a6 -- referer: http://localhost:8090/admin/backup.action | url: /admin/dobackup.action | traceId: ff624f240f8ea6ca | userName: peter com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.confluence.plugins.confluence-feature-discovery-plugin] at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:95) at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:86) at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38) at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84) at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68) at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$withRequestContext$1(VCacheRequestContextOperations.java:59) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: Could not read fields for table AO_6384AB_DISCOVERED at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:141) at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126) at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110) at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62) at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:99) .... Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431) at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:271) at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:91) ... 65 more 2020-12-27 16:49:02,566 WARN [http-nio-127.0.0.1-8090-exec-43] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnap shot{startTime=1609087117761, threadId=1327, threadName='http-nio-127.0.0.1-8090-exec-43', userId='peter', type='web-request', summary='/admin/dobackup.action'} -- referer: http://localhost:8090/admin/backup.action | url: /admin/dobackup.action | traceId: ff624f240f8ea6ca | userName: peter 2020-12-27 16:49:05,493 WARN [http-nio-127.0.0.1-8090-exec-43] [common.security.jersey.XsrfResourceFilter] passesAdditionalBrowserChecks Additional XSRF checks failed for request: https://wiki.folio.org/rest/analytics/1.0/publish/bulk , origin: http://localhost:8090 , referrer: http://localhost:8090/admin/dobackup.action , credentials in request: true , allowed via CORS: false -- referer: http://localhost:8090/admin/dobackup.action | url: /rest/analytics/1.0/publish/bulk | traceId: 5536f2b38048dc18 | userName: peter 2020-12-27 16:50:00,068 INFO [Caesium-1-1] [synchrony.service.http.SynchronyRequestExecutor] execute Initiating Locking API request: SynchronyLockRequest 2020-12-27 16:50:00,172 INFO [Caesium-1-1] [synchrony.service.http.SynchronyRequestExecutor] execute Initiating Locking API request: SynchronyUnlockRequest 2020-12-27 16:54:05,187 ERROR [http-nio-127.0.0.1-8090-exec-3] [host.util.concurrent.LockedTaskExecutor] executeUnderLock Failed whilst getting last modified date -- referer: http://localhost:8090/admin/dobackup.action | url: /plugins/servlet/notifications-miniview | traceId: f2afd26796e1c155 | userName: peter java.util.concurrent.ExecutionException: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.executeUnderLock(LockedTaskExecutor.java:96) at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getLastModified(LocalRegistrationServiceImpl.java:192) at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getCacheValue(LocalRegistrationServiceImpl.java:223) at com.atlassian.mywork.host.servlet.ServletRenderer.renderWithAnchor(ServletRenderer.java:71) at com.atlassian.mywork.host.servlet.ServletRenderer.renderWithAnchor(ServletRenderer.java:43) at com.atlassian.mywork.host.servlet.NotificationsMiniviewServlet.doGet(NotificationsMiniviewServlet.java:19) .... Caused by: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) .... 2020-12-27 16:54:05,189 ERROR [http-nio-127.0.0.1-8090-exec-3] [ContainerBase.[Standalone].[localhost].[/]] log Unhandled exception occurred whilst decorating page -- referer: http://localhost:8090/admin/dobackup.action | url: /plugins/servlet/notifications-miniview | traceId: f2afd26796e1c155 | userName: peter org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) .... 2020-12-27 16:54:05,190 WARN [http-nio-127.0.0.1-8090-exec-3] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1609087745177, threadId=189, threadName='http-nio-127.0.0.1-8090-exec-3', userId='peter', type='web-request', summary='/plugins/servlet/notifications-miniview'} -- referer: http://localhost:8090/admin/dobackup.action | url: /plugins/servlet/notifications-miniview | traceId: f2afd26796e1c155 | userName: peter 2020-12-27 16:54:05,191 ERROR [http-nio-127.0.0.1-8090-exec-3] [[Standalone].[localhost].[/].[servlet-module-container-servlet]] log Servlet.service() for servlet [servlet-module-container-servlet] in context with path [] threw exception org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) .... 2020-12-27 16:54:05,194 INFO [http-nio-127.0.0.1-8090-exec-3] [atlassian.confluence.status.SystemErrorInformationLogger] writeToLog Request Unique ID : 7d65ceda-94cf-4750-a348-17ddf5b41819 -------------------------- JVM Stats -------------------------- xmx = 3221225472 usedNonHeap = 629740352 availableHeap = 1673257312 availableNonHeap = -1 allocatedHeap = 3221225472 freeAllocatedHeap = 1673257312 maxNonHeap = -1 availablePermGen = 0 maxPermGen = -1 maxHeap = 3221225472 usedHeap = 1547968160 usedPermGen = -1 xms = 3221225472 -------------------------- Request Information -------------------------- URL: https://wiki.folio.org/500page.jsp Scheme: https Server: wiki.folio.org Port: 443 URI: /500page.jsp Context Path: Servlet Path: /500page.jsp Path Info: null Query String: null -------------------------- Attributes -------------------------- javax.servlet.forward.request_uri: /plugins/servlet/notifications-miniview javax.servlet.forward.context_path: javax.servlet.forward.servlet_path: /plugins/servlet javax.servlet.forward.path_info: /notifications-miniview javax.servlet.forward.mapping: org.apache.catalina.core.ApplicationMapping$MappingImpl@6395585f javax.servlet.error.status_code: 500 com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@225cdd89 com.opensymphony.sitemesh.APPLIED_ONCE: true __cleanup_recursion_counter: 0 javax.servlet.error.message: javax.servlet.error.servlet_name: servlet-module-container-servlet com.atlassian.confluence.util.message.MessagesDecoratorFilter__already_filtered__: true com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true atlassian.core.seraph.original.url: /500page.jsp com.atlassian.labs.botkiller.BotKillerFilter: true com.atlassian.gzipfilter.GzipFilter_already_filtered: true Confluence-Request-Time: 1609087745175 B3-TraceId: fd8cf4b790dd25 loginfilter.already.filtered: true javax.servlet.error.request_uri: /plugins/servlet/notifications-miniview com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true com.atlassian.confluence.web.ConfluenceJohnsonFilter_already_filtered: true javax.servlet.error.exception: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable os_securityfilter_already_filtered: true com.atlassian.seraph.auth.LoginReason: OK com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true -------------------------- Parameters -------------------------- caused by: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404) 2020-12-27 16:54:05,216 ERROR [http-nio-127.0.0.1-8090-exec-3] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@3a29b482 -- traceId: c5224bcea37dd8cd com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.confluence.plugins.confluence-feature-discovery-plugin] at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:95) at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:86) at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38) at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84) at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68) at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$withRequestContext$1(VCacheRequestContextOperations.java:59) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: Could not read fields for table AO_6384AB_DISCOVERED at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:141) at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126) at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110) at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62) .... Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431) at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:271) at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:91) ... 65 more The on-screen error at the end of the backup/export attempt is (again condensed): com.atlassian.confluence.importexport.ImportExportException: java.lang.RuntimeException: Could not read fields for table AO_54C900_C_TEMPLATE_REF at com.atlassian.activeobjects.confluence.backup.ActiveObjectsBackupRestoreProvider.backup(ActiveObjectsBackupRestoreProvider.java:30) at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupPluginData(FileXmlExporter.java:215) at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupEverything(FileXmlExporter.java:109) at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.doExport(AbstractXmlExporter.java:94) at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExportInternal(FileXmlExporter.java:60) at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExport(FileXmlExporter.java:54) at com.atlassian.confluence.importexport.DefaultImportExportManager.doExport(DefaultImportExportManager.java:212) at com.atlassian.confluence.importexport.DefaultImportExportManager.exportAs(DefaultImportExportManager.java:178) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) .... Caused by: java.lang.RuntimeException: Could not read fields for table AO_54C900_C_TEMPLATE_REF at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:141) at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126) at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110) at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62) at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:99) at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:97) at com.google.common.collect.Iterators$6.transform(Iterators.java:785) .... Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes. at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431) at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:271) at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:91) ... 438 more Going to post back to Atlassian Community. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 28/Jan/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Atlassian support has a copy of our database and has reproduced a similar error on the same table: Caused by: java.sql.SQLException: Cannot drop index 'index_ao_54c900_c_t852152353': needed in a foreign key constraint Exception executing SQL update <DROP INDEX index_ao_54c900_c_t852152353 ON AO_54C900_C_TEMPLATE_REF> java.sql.SQLException: Cannot drop index 'index_ao_54c900_c_t852152353': needed in a foreign key constraint Atlassian support broke up the steps into migrating Confluence first from 6.15.10 to 7.8 then MySQL 5.6 to 8. In between those steps, the support person could create a new space and a new blueprint template. The error above happened after the MySQL upgrade.
The support person did note that it took around 8 hours to load in our database dump onto their test system. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 03/Feb/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Heard back from Atlassian:
Send the requested output (also attached to this issue). There are three tables set to ROW_FORMAT="Compact":
The first two are the ones seen in Confluence error logs. Also sent back the RDS equivalent of a MySQL server `my.cnf` file (generated using this answer on Stack Exchange). Waiting to hear what I assume will be three SQL commands to alter the row format to "Dynamic" for these three tables. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 04/Feb/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Shutdown Confluence, made the changes below to the MySQL database, and restarted Confluence. ALTER TABLE AO_54C900_C_TEMPLATE_REF ROW_FORMAT=DYNAMIC; ALTER TABLE AO_6384AB_DISCOVERED ROW_FORMAT=DYNAMIC; ALTER TABLE AO_6384AB_FEATURE_METADATA_AO ROW_FORMAT=DYNAMIC; For all three commands, got: ERROR: Index column size too large. The maximum column size is 767 bytes. Going to need to figure out what indexes these are, drop them, alter the tables, and re-add them. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 16/Feb/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Still stuck. My notes: SQL indexes for AO_6384AB_FEATURE_METADATA_AORunning ALTER TABLE AO_6384AB_FEATURE_METADATA_AO ROW_FORMAT=dynamic; results in ERROR: Index column size too large. The maximum column size is 767 bytes. Attempting to enumerate indexes to drop them.
Running... drop index index_ao_6384ab_fea1458039816 on AO_6384AB_FEATURE_METADATA_AO; ...results in... Query 1 ERROR: Index column size too large. The maximum column size is 767 bytes. Trying:
Possible pointers to solution
The fix appears to be in MySQL 8.0.22 and the current version of MySQL on RDS is 8.0.21. I have filed a forum request with AWS to see when this might be fixed. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 23/Feb/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Atlassian support suggests:
Unfortunately, the "ERROR: Index column size too large. The maximum column size is 767 bytes." remains. Also unfortunately, MySQL 8.0.21 remains the latest version available and there was no reply to my forum message. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 23/Feb/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I can't dump the database either with mysqldump: ``` AWS RDS offers to give me the dump in Apache Parquet format, but I don't think that is helpful/useful for editing and re-uploading to MySQL. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 16/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Just for everyone following along, I check every workday to see if AWS has adopted MySQL 8.0.22 that reportedly addresses the underlying database issue. So far no luck, and no comment on my AWS forum message about supporting this version. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 23/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Still no MySQL 8.0.22. sigh | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 25/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hey! MySQL 8.0.23 just showed up on RDS! I'll try to upgrade the Confluence database tonight and see if this issue is solved. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by John Malconian [ 25/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
woohoo! | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 27/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Still stuck. Going to ask to upgrade our AWS support plan to "Developer" so a technical support issue can be filed with AWS. Notes from today's efforts below. Steps
NotesGot stuck at step 4—getting the same error: mysqldump: Got error: 1709: Index column size too large. The maximum column size is 767 bytes. when using LOCK TABLES. See contents of “PrePatchCompatibility.log” below. Attempting stop/start of RDS instance. Same result after restart. Running the ALTER TABLE AO_54C900_C_TEMPLATE_REF ROW_FORMAT=DYNAMIC MySQL command from Tables returns the same error. The “mysql-error-running.log” (below, last line) says the engine is still 8.0.20. Confirmed in RDS console that engine is still 8.0.20. Attempting to modify instance again. Modification not successful—same entries in “PrePatchCompatibility.log”. “mysql-error-running.log” is empty. Attempting just a “reboot” of the RDS instance. Nothing helpful in the AWS Forums for how to deal with a failing MySQL engine minor upgrade problem. Stuck. Restarting Confluence. Contents of AWS RDS “PrePatchCompatibility.log”
Executing Compatibility Checks for the MySQL server at localhost:3306. Source Version: 8.0.20 - Source distribution, Target Version: 8.0.23. 1) Issues reported by 'check table x for upgrade' command foliowiki.AO_6384AB_DISCOVERED - Index column size too large. The maximum column size is 767 bytes. foliowiki.AO_6384AB_DISCOVERED - Table 'foliowiki.AO_6384AB_DISCOVERED' doesn't exist foliowiki.AO_6384AB_DISCOVERED - Corrupt foliowiki.AO_6384AB_FEATURE_METADATA_AO - Index column size too large. The maximum column size is 767 bytes. foliowiki.AO_6384AB_FEATURE_METADATA_AO - Table 'foliowiki.AO_6384AB_FEATURE_METADATA_AO' doesn't exist foliowiki.AO_6384AB_FEATURE_METADATA_AO - Corrupt foliowiki.AO_54C900_C_TEMPLATE_REF - Index column size too large. The maximum column size is 767 bytes. foliowiki.AO_54C900_C_TEMPLATE_REF - Table 'foliowiki.AO_54C900_C_TEMPLATE_REF' doesn't exist foliowiki.AO_54C900_C_TEMPLATE_REF - Corrupt 2) Column definition mismatch between InnoDB Data Dictionary and actual table definition. No issues found. 3) Tables with dangling FULLTEXT index reference No issues found. 4) MySQL pre-checks that all database, table, and trigger names are lowercase when the lower_case_table_names parameter is set to 1. No issues found. Errors: 1 Warnings: 0 Database Objects Affected: 9 ----------------------- END OF LOG ----------------------
Contents of mysql-error-running.log2021-03-27T14:41:39.958254Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-file' is deprecated and will be removed in a future release. 2021-03-27T14:41:39.958437Z 0 [System] [MY-010116] [Server] /rdsdbbin/mysql/bin/mysqld (mysqld 8.0.20) starting as process 5850 2021-03-27T14:41:39.966027Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2021-03-27T14:41:39.969348Z 1 [Warning] [MY-012191] [InnoDB] Scan path '/rdsdbdata/db/innodb' is ignored because it is a sub-directory of '/rdsdbdata/db/ 2021-03-27T14:41:41.245548Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2021-03-27T14:41:41.540259Z 0 [Warning] [MY-013414] [Server] Server SSL certificate doesn't verify: unable to get issuer certificate 2021-03-27T14:41:41.596141Z 0 [System] [MY-010931] [Server] /rdsdbbin/mysql/bin/mysqld: ready for connections. Version: '8.0.20' socket: '/tmp/mysql.sock' port: 3306 Source distribution. Links
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Julian Ladisch [ 30/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Have you tried to fix it using mysql-debug as described on https://dba.stackexchange.com/questions/234822/mysql-error-seems-unfixable-index-column-size-too-large/252255#252255 ? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 30/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Unfortunately, with RDS this isn't an option....I don't have a way to invoke MySQL-Debug using the AWS-hosted MySQL. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Julian Ladisch [ 30/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Does AWS allow you to download the MySQL data directory so that you can copy the raw data files into some other server where mysql-debug is available? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 30/Mar/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Good idea, but no. The only format AWS itself offers is the Apache Parquet format. I did some research on that format, and there doesn't seem to be a way to transform it into a MySQL dump file or MySQL directory/index format. And of course the underlying issue is preventing mysqldump from working. Thanks for thinking about this...I'll take any suggestions and try to pursue them! | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 08/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Not waiting for Technical Council approval of the AWS Support expense...I upgraded the FOLIO-Org AWS Organization to "Developer"-level support and filed an issue:
https://console.aws.amazon.com/support/home#/case/?displayId=8200441491&language=en
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 12/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Received reply back from AWS that correctly summarized the issue. The AWS tech wrote:
Replied with details:
And heard back:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 13/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
From AWS:
My reply:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 14/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
From AWS:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 19/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
From: no-reply-aws@amazon.com <no-reply-aws@amazon.com> Thank you for being patient with me. RDS MySQL team is still working on the instance. I will keep an eye on it and will inform you of any progress. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 23/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
From: no-reply-aws@amazon.com <no-reply-aws@amazon.com> Thank you for your patience. RDS MySQL team is still working on the problem you encountered and I am following it closely. I will inform you of the progress if there is any update. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 26/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
From: no-reply-aws@amazon.com <no-reply-aws@amazon.com> I am sorry for the late reply. I just got an update from RDS MySQL team and please let me share it with you. The team understand any customer database is critical and is testing the solution thoroughly. The testing is almost close to completion and the fix would be applied to the instance arn:aws:rds:us-east-1:732722833398:db:folio-wiki-202010-read-replica-for-aws-troubleshooting in two days. Thank you for your patience and understanding. I will get back to you once it is fixed. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 28/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
From: "no-reply-aws@amazon.com" <no-reply-aws@amazon.com> Thank you for being patient with me. The team informed me that the fix was applied successfully on the instance arn:aws:rds:us-east-1:732722833398:db:folio-wiki-202010-read-replica-for-aws-troubleshooting. The following is an excerpt of what the document [4] says of "Q: How do I see the status of my active read replica(s)?": Therefore, you may take checking the metric Replica Lag into consideration before the read replica is promoted. Hope it is useful. Please feel free to contact me if further assistance is needed. [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 28/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
(Fixing the formatting) From the above, I gather that they have fixed the underlying tables without upgrading the MySQL engine. That’s a pretty good trick. I confirmed that the replica is still MySQL 8.0.20.
PrePatchCompatability.logExecuting Compatibility Checks for the MySQL server at localhost:3306. Source Version: 8.0.20 - Source distribution, Target Version: 8.0.23. 1) Issues reported by 'check table x for upgrade' command No issues found. 2) Column definition mismatch between InnoDB Data Dictionary and actual table definition. No issues found. 3) Tables with dangling FULLTEXT index reference No issues found. Errors: 0 Warnings: 0 Database Objects Affected: 0 ----------------------- END OF LOG ---------------------- Confluence template notice of upgrade
!https://folio-org.atlassian.net/wiki/download/attachments/4227086/Caution.png?version=1&modificationDate=1602190749000&api=v2|align=left,alt=Caution! *{color:red}A fix was applied to address the wiki database issue.{color}* // *{color:red}Please report immediately any unusual or broken wiki features to @peter-murray on Slack or by email to peter@indexdata.com{color}.*
----
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 28/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Using Confluences Troubleshooting and Support Tools → Log analyzer, there are no relevant errors. (There is an error related to failing SMTP credentials, but that can be safely ignored for this issue.) The only symptom that remains from the original list is that the user notifications dropdown returns an error:
I'm going to add that to the issue with Atlassian, but it may be a separate problem. I'm uploading a copy of the database dump and a support Zip to Atlassian for them to look at.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Peter Murray [ 29/Apr/21 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Confluence has been running on the new database for over 24 hours and no new issues have been reported. The one remaining issue—an error and stack trace in the Notifications drop down menu—has been assigned a new issue at Atlassian (CSP-291643) and is now tracked as
From: Rodrigo Oliveira <confluence-support-system@am.atlassian.com> Date: Apr 28, 2021, 3:33 PM -0400 To: peter.murray@openlibraryfoundation.org Subject: CSP-286216 "java.sql.SQLException: Index column size too large" error after upgrading Confluence and MySQL
|