[FOLIO-2867] FOLIO Confluence wiki not offering meeting notes template Created: 06/Nov/20  Updated: 29/Apr/21  Resolved: 29/Apr/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: TBD
Reporter: Kristin Martin Assignee: Peter Murray
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: Text File aws-rds-wiki-parameter-group.txt     PNG File image-2020-11-06-15-20-51-556.png     PNG File image-2020-11-06-15-22-59-157.png     Text File select-from-information_schema.tables.txt    
Issue links:
Duplicate
is duplicated by FOLIO-2881 Error creating Confluence page from t... Closed
Relates
relates to FOLIO-2822 Upgrade FOLIO Confluence Closed
relates to FOLIO-3136 Fix Confluence notifications dropdown Closed
Sprint: DevOps: Sprint 105, DevOps: Sprint 103, DevOps: Sprint 102, DevOps Sprint 109, DevOps Sprint 107, DevOps Sprint 108, DevOps Sprint 110, DevOps Sprint 111, DevOps Sprint 112, DevOps: Sprint 104, DevOps Sprint 106, DevOps Sprint 113
Development Team: FOLIO DevOps

 Description   

Overview:
On wiki.folio.org, the macro to create a new page of meeting minutes is not offered. This is happening on at least the RM and MM SIG meeting notes pages.

Steps to Reproduce:

  1. Go to https://folio-org.atlassian.net/wiki/display/RM/RM+SIG+Meetings+and+Notes
  2. Attempt to use button that used to say {Create meeting notes}

Expected Results:
Create new meeting notes.

Actual Results:

Additional Information:
The Meeting Notes template is not offered when editing the "Create from template" macro.

Interested parties:
Reported by Kristin Martin



 Comments   
Comment by Peter Murray [ 06/Nov/20 ]

In this 2018 answer on the Confluence community, a reply says the meeting notes blueprint is part of the "Confluence Business Blueprints - Plugin" page. I have tried disabling that system plugin and re-enabling it; no effect.

Comment by Peter Murray [ 07/Nov/20 ]

Meeting Notes Blueprint has more details about the meeting notes blueprint, but nothing that seems helpful in resolving this. I think the next step is to try rebooting the Confluence server to see if that clears up the issue. I'm going to do that now while it is off hours.

Comment by Peter Murray [ 07/Nov/20 ]

Hmm...that didn't do it. Attepmpting to clear plugin cache.

# systemctl start confluence.service
# cd /atlassian/var/application-data/confluence
# rm -rf plugins-osgi-cache plugins-temp bundled-plugins plugins-cache
# systemctl start confluence.service
Comment by Peter Murray [ 07/Nov/20 ]

Hmmm...that didn't work either. John Malconian: do you have any ideas?

Comment by John Malconian [ 18/Nov/20 ]

I see many stack trace errors in the confluence log file that look like this:

2020-11-18 20:59:02,799 ERROR [http-nio-127.0.0.1-8090-exec-93] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@1d4e865b
 -- url: /display/DD/Decision+log | traceId: 4f913c631079d71d | userName: malc
com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.confluence.plugins.confluence-feature-discovery-plugin]
...
Caused by: java.lang.RuntimeException: Could not read fields for table AO_6384AB_DISCOVERED
...
Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes.
Comment by John Malconian [ 18/Nov/20 ]

I think setting the MySQL DB parameter 'innodb_default_row_format' to 'DYNAMIC' might resolve the error above. According to the AWS RDS console, it appears it is not currently set to anything which is weird because the default should be 'DYNAMIC'. More investigation needed.

Comment by John Malconian [ 18/Nov/20 ]

More info: https://dev.mysql.com/doc/refman/8.0/en/innodb-row-format.html

Comment by Peter Murray [ 24/Nov/20 ]

I wonder if you might be onto something. I followed the suggestion to use the `utf8mb4` character set, and I found this issue for MariaDB: https://jira.mariadb.org/browse/MDEV-14904

I'll have to take the database down to make the change, so I'll wait until later today to try it out.

Comment by Peter Murray [ 24/Nov/20 ]

Hmmm – good theory, but it didn't pan out. I'm going to go through and make sure that the database changes specified in the upgrade instructions actually happened.

Comment by Peter Murray [ 25/Nov/20 ]

Okay, this is kicking my butt. I've tried:

  1. Re-done all of the steps in How to Fix the Collation and Character Set of a MySQL Database manually
  2. I even put the connectionCollation=utf8mb4_bin parameter on the end of the JDBC URL as suggested at the bottom of that page.
  3. Tried using a MySQL client to delete the AO_54C900_C_TEMPLATE_REF and AO_6384AB_DISCOVERED tables, but got the same error ("ERROR: Index column size too large. The maximum column size is 767 bytes.")
  4. Noted that step 10 of Migrate MySQL database to UTF8MB4 character encoding didn't seem to work...threw an "com.mysql.cj.exceptions.WrongArgumentException: Unsupported character encoding 'utf8mb4'" error.
  5. To get around the above, tried adding properties to the {confluence.cfg.xml}

    file as suggested by Caused by: java.sql.SQLException: Unsupported character encoding 'utf8mb4' on StackOverflow.

Also note that this issue may also be related to an inability to create a new space in Confluence.

Comment by Peter Murray [ 15/Dec/20 ]

Still can't figure it out. Have posted a question to the Atlassian Confluence community board seeking suggestions for solutions.

Comment by Peter Murray [ 24/Dec/20 ]

Daniel Ebers on the Atlassian Community site had two suggestions:

1.) Could you please confirm on the version of the JDBC driver?
I was not able to identify the exact version from runbook. Although very unlikely could you please double check if the version matches against the Confluence version 7.8.0
From my understanding the version 8.0.21 of JDBC driver is to be used with that Confluence version.

Verified that we are using 8.0.21; the file is located at /atlassian/opt/confluence/confluence/WEB-INF/lib/mysql-connector-java-8.0.21.jar. Ran find /confluence -name "mysql-connector*" -print to make sure there wasn't anything funny elsewhere. Lo and behold, there is a /atlassian/opt/confluence/confluence/mysql-connector-java-5.1.44-bin.jar! Could it be as easy as that? Stopped confluence, moved it out of the way, started confluence...same symptoms.

2.) This might look like the big bang and comes to a price, even if it succeeds, but...
... have to tried to export data via XML export and restore it to a (non-production!) Confluence server.
Documentation as per: https://confluence.atlassian.com/doc/manually-backing-up-the-site-152405.html
I know, in case of a complex site this will most likely fail but it could be a pointer if something basic is broken or if it is related to database.
Sadly, even with an output this would only shed some light on where to go next.

Going to try this next.

Comment by Peter Murray [ 27/Dec/20 ]

Was experiencing timeouts trying to access https://wiki.folio.org/admin/backup.do, so I stood up an ssh tunnel to Tomcat (

{ssh -N -L 8090:127.0.0.1:8090 wiki.folio.org}

) so I could access it from the laptop as http://localhost:8090/admin/backup.do.

  1. Shutdown nginx: `systemctl stop nginx.service`
  2. Start backup at `http://localhost:8090/admin/backup.action` — 11:38:30, selected "Include attachments" and "Also save a copy to the Backups directory". Results: `Could not read fields for table AO_54C900_C_TEMPLATE_REF` error. No backup file found on filesystem. Noting `Low free disk space` error in atlassian-confluence.log.
  3. Rerunning backup/export after moving `/atlassian/var/application-data/2020_10_11-confluence-6.15.2-back.zip` out of partition to see if the disk space message was an important issue.
  4. Start another backup at 12:45:25 — errors out
  5. Start another backup at 12:57:17 without "Include attachements" — errors out
  6. Look for backup file on /atlassian partition: `find /atlassian -mmin -60 -type f -exec ls -l {} \;` — nothing there

There are numerous log messages in "atlassian-confluence.log". Stack traces reduced to what seems like the most important lines.

2020-12-27 16:38:38,191 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Backward compatibility for export: Checking...
2020-12-27 16:38:38,192 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8401: "Upgrade to build number: 8401" - Ok
2020-12-27 16:38:38,193 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8301: "Add unmanaged unique constraints missing in CONFSERVER-38706 and CONFSERVER-58261" - Ok
2020-12-27 16:38:38,193 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok
2020-12-27 16:38:38,194 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok
2020-12-27 16:38:38,194 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 8202: "Upgrade to build number: 8202" - Ok
2020-12-27 16:38:38,194 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7901: "Upgrade "synchrony.btf.off" property" - Ok
2020-12-27 16:38:38,195 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7801: "Disables backup job" - Ok
2020-12-27 16:38:38,195 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7701: "Fixes wrong spaces for drafts" - Ok
2020-12-27 16:38:38,201 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7601: "Upgrade custom colour scheme to ADGS" - Ok
2020-12-27 16:38:38,201 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7400: "Upgrade to build number: 7400" - Ok
2020-12-27 16:38:38,201 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7202: "Remove the legacy Plugin License Storage plugin from the user installed plugins" - Ok
2020-12-27 16:38:38,202 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7201: "Upgrade to build number: 7201" - Ok
2020-12-27 16:38:38,202 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7110: "Reindex all blogs." - Ok
2020-12-27 16:38:38,202 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7109: "Drop and update some low performance indexes on EVENTS table for Synchrony" - Ok
2020-12-27 16:38:38,203 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7108: "Update index sp_comp_idx when the definition is not correct" - Ok
2020-12-27 16:38:38,203 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7106: "Drop the hibernate 2 constraints in preparation for them to be re-created in the hibernate 5 naming style" - Ok
2020-12-27 16:38:38,203 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7105: "Drop or update some low performance indexes on CONTENT" - Ok
2020-12-27 16:38:38,204 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7104: "Set the mime type for csv files to text/csv" - Ok
2020-12-27 16:38:38,204 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 7101: "Migrate over Documentation theme settings to Custom Page Settings in the default theme" - Ok
2020-12-27 16:38:38,209 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6439: "Reindexes all PersonalInformation objects, creating them for any users that do not have one." - Ok
2020-12-27 16:38:38,210 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6430: "Upgrade to build number: 6430" - Ok
2020-12-27 16:38:38,215 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6424: "Adds missing multi-column unique constraints to the Crowd database tables" - Ok
2020-12-27 16:38:38,221 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6423: "Cleaning data that violates constraints we want to add in the next migration task" - Ok
2020-12-27 16:38:38,221 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6422: "Upgrade to build number: 6422" - Ok
2020-12-27 16:38:38,227 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6412: "Delete all FileStore IDs" - Ok
2020-12-27 16:38:38,227 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6406: "Reindex all spaces" - Ok
2020-12-27 16:38:38,228 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6403: "Upgrade to build number: 6403" - Ok
2020-12-27 16:38:38,228 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6193: "Rebuild the Confluence favourites index" - Ok
2020-12-27 16:38:38,228 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6193: "Rebuild the Confluence favourites index" - Ok
2020-12-27 16:38:38,229 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6189: "Upgrade to build number: 6189" - Ok
2020-12-27 16:38:38,235 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6180: "Adds a multi-column index to content table that speeds up many common queries." - Ok
2020-12-27 16:38:38,235 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6175: "Upgrade to build number: 6175" - Ok
2020-12-27 16:38:38,236 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Build number 6169: "Upgrade to build number: 6169" - Not Ok
2020-12-27 16:38:38,236 INFO [http-nio-127.0.0.1-8090-exec-43] [atlassian.confluence.upgrade.AbstractUpgradeManager] getExportBuildNumber Backward compatibility for export: "Upgrade to build number: 6169" prevents compatibility before 6169

2020-12-27 16:40:34,931 ERROR [http-nio-127.0.0.1-8090-exec-5] [host.util.concurrent.LockedTaskExecutor] executeUnderLock Failed whilst getting last modified date
 -- url: /plugins/servlet/mobile | traceId: 741012c213b7de50 | userName: anonymous
java.util.concurrent.ExecutionException: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)
	at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.executeUnderLock(LockedTaskExecutor.java:96)
	at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getLastModified(LocalRegistrationServiceImpl.java:192)
	at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getCacheValue(LocalRegistrationServiceImpl.java:223)
	at com.atlassian.mywork.host.web.MobileNavigationContextProvider.getContextMap(MobileNavigationContextProvider.java:50)
	at com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel.getHtml(DefaultWebPanelModuleDescriptor.java:134)
	at com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebInterfaceManager$ExceptionHandlingWebPanel.getHtml(ConfluenceWebInterfaceManager.java:133)
....
Caused by: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
....

2020-12-27 16:40:34,934 ERROR [http-nio-127.0.0.1-8090-exec-5] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@6954df81
 -- url: /plugins/servlet/mobile | traceId: 741012c213b7de50 | userName: anonymous
org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)

2020-12-27 16:40:34,936 WARN [http-nio-127.0.0.1-8090-exec-5] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1609086934914, threadId=191, threadName='http-nio-127.0.0.1-8090-exec-5', userId='<unknown>', type='web-request', summary='/plugins/servlet/mobile?contentId=25723434'}
 -- url: /plugins/servlet/mobile | traceId: 741012c213b7de50 | userName: anonymous

2020-12-27 16:47:18,977 INFO [Caesium-1-1] [agent.service.check.StaleChecksCleaner] info Cleaning stale checks

2020-12-27 16:48:47,191 ERROR [alert-dispatch:thread-1] [confluence.alert-log] log 1609087727050 ; ERROR ; OS ; OS-1002 ; Low free disk space ; not-detected ;  ;  ; {"d
irectory":"/atlassian/var/application-data/confluence","freeInMegabytes":6008,"totalInMegabytes":50268,"minimumInMegabytes":8192}

2020-12-27 16:48:47,196 ERROR [alert-dispatch:thread-1] [atlassian-monitor] log 2020-12-27T16:48:47.050Z Component 'Operating System' alerted 'Low free disk space' (det
ails: {"directory":"/atlassian/var/application-data/confluence","freeInMegabytes":6008,"totalInMegabytes":50268,"minimumInMegabytes":8192}, trigger: {"pluginKey": "not-
detected"})

2020-12-27 16:49:02,124 WARN [http-nio-127.0.0.1-8090-exec-43] [activeobjects.confluence.backup.LoggingBackupProgressMonitor] beginBackup Begin Active objects backup, c
hange log level to INFO for com.atlassian.activeobjects.confluence.backup for more detailed logging.
 -- referer: http://localhost:8090/admin/backup.action | url: /admin/dobackup.action | traceId: ff624f240f8ea6ca | userName: peter | action: dobackup

2020-12-27 16:49:02,422 ERROR [http-nio-127.0.0.1-8090-exec-43] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.
plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@13fcf9a6
 -- referer: http://localhost:8090/admin/backup.action | url: /admin/dobackup.action | traceId: ff624f240f8ea6ca | userName: peter
com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.confluence.plugins.confluence-feature-discovery-plugin]
        at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:95)
        at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:86)
        at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38)
        at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84)
        at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$withRequestContext$1(VCacheRequestContextOperations.java:59)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Could not read fields for table AO_6384AB_DISCOVERED
        at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:141)
        at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126)
        at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110)
        at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62)
        at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:99)
....
Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
        at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431)
        at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:271)
        at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:91)
        ... 65 more

2020-12-27 16:49:02,566 WARN [http-nio-127.0.0.1-8090-exec-43] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnap
shot{startTime=1609087117761, threadId=1327, threadName='http-nio-127.0.0.1-8090-exec-43', userId='peter', type='web-request', summary='/admin/dobackup.action'}
 -- referer: http://localhost:8090/admin/backup.action | url: /admin/dobackup.action | traceId: ff624f240f8ea6ca | userName: peter

2020-12-27 16:49:05,493 WARN [http-nio-127.0.0.1-8090-exec-43] [common.security.jersey.XsrfResourceFilter] passesAdditionalBrowserChecks Additional XSRF checks failed for request: https://wiki.folio.org/rest/analytics/1.0/publish/bulk , origin: http://localhost:8090 , referrer: http://localhost:8090/admin/dobackup.action , credentials in request: true , allowed via CORS: false
 -- referer: http://localhost:8090/admin/dobackup.action | url: /rest/analytics/1.0/publish/bulk | traceId: 5536f2b38048dc18 | userName: peter

2020-12-27 16:50:00,068 INFO [Caesium-1-1] [synchrony.service.http.SynchronyRequestExecutor] execute Initiating Locking API request: SynchronyLockRequest

2020-12-27 16:50:00,172 INFO [Caesium-1-1] [synchrony.service.http.SynchronyRequestExecutor] execute Initiating Locking API request: SynchronyUnlockRequest

2020-12-27 16:54:05,187 ERROR [http-nio-127.0.0.1-8090-exec-3] [host.util.concurrent.LockedTaskExecutor] executeUnderLock Failed whilst getting last modified date
 -- referer: http://localhost:8090/admin/dobackup.action | url: /plugins/servlet/notifications-miniview | traceId: f2afd26796e1c155 | userName: peter
java.util.concurrent.ExecutionException: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)
        at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.executeUnderLock(LockedTaskExecutor.java:96)
        at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getLastModified(LocalRegistrationServiceImpl.java:192)
        at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getCacheValue(LocalRegistrationServiceImpl.java:223)
        at com.atlassian.mywork.host.servlet.ServletRenderer.renderWithAnchor(ServletRenderer.java:71)
        at com.atlassian.mywork.host.servlet.ServletRenderer.renderWithAnchor(ServletRenderer.java:43)
        at com.atlassian.mywork.host.servlet.NotificationsMiniviewServlet.doGet(NotificationsMiniviewServlet.java:19)
....
Caused by: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
....

2020-12-27 16:54:05,189 ERROR [http-nio-127.0.0.1-8090-exec-3] [ContainerBase.[Standalone].[localhost].[/]] log Unhandled exception occurred whilst decorating page
 -- referer: http://localhost:8090/admin/dobackup.action | url: /plugins/servlet/notifications-miniview | traceId: f2afd26796e1c155 | userName: peter
org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
....

2020-12-27 16:54:05,190 WARN [http-nio-127.0.0.1-8090-exec-3] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1609087745177, threadId=189, threadName='http-nio-127.0.0.1-8090-exec-3', userId='peter', type='web-request', summary='/plugins/servlet/notifications-miniview'}
 -- referer: http://localhost:8090/admin/dobackup.action | url: /plugins/servlet/notifications-miniview | traceId: f2afd26796e1c155 | userName: peter

2020-12-27 16:54:05,191 ERROR [http-nio-127.0.0.1-8090-exec-3] [[Standalone].[localhost].[/].[servlet-module-container-servlet]] log Servlet.service() for servlet [servlet-module-container-servlet] in context with path [] threw exception
org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
....

2020-12-27 16:54:05,194 INFO [http-nio-127.0.0.1-8090-exec-3] [atlassian.confluence.status.SystemErrorInformationLogger] writeToLog
Request Unique ID : 7d65ceda-94cf-4750-a348-17ddf5b41819
--------------------------
JVM Stats
--------------------------
xmx = 3221225472
usedNonHeap = 629740352
availableHeap = 1673257312
availableNonHeap = -1
allocatedHeap = 3221225472
freeAllocatedHeap = 1673257312
maxNonHeap = -1
availablePermGen = 0
maxPermGen = -1
maxHeap = 3221225472
usedHeap = 1547968160
usedPermGen = -1
xms = 3221225472
--------------------------
Request Information
--------------------------
URL: https://wiki.folio.org/500page.jsp
Scheme: https
Server: wiki.folio.org
Port: 443
URI: /500page.jsp
Context Path:
Servlet Path: /500page.jsp
Path Info: null
Query String: null
--------------------------
Attributes
--------------------------
javax.servlet.forward.request_uri: /plugins/servlet/notifications-miniview
javax.servlet.forward.context_path:
javax.servlet.forward.servlet_path: /plugins/servlet
javax.servlet.forward.path_info: /notifications-miniview
javax.servlet.forward.mapping: org.apache.catalina.core.ApplicationMapping$MappingImpl@6395585f
javax.servlet.error.status_code: 500
com.atlassian.confluence.impl.profiling.DecoratorTimings: com.atlassian.confluence.impl.profiling.DecoratorTimings@225cdd89
com.opensymphony.sitemesh.APPLIED_ONCE: true
__cleanup_recursion_counter: 0
javax.servlet.error.message:
javax.servlet.error.servlet_name: servlet-module-container-servlet
com.atlassian.confluence.util.message.MessagesDecoratorFilter__already_filtered__: true
com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter_already_filtered: true
atlassian.core.seraph.original.url: /500page.jsp
com.atlassian.labs.botkiller.BotKillerFilter: true
com.atlassian.gzipfilter.GzipFilter_already_filtered: true
Confluence-Request-Time: 1609087745175
B3-TraceId: fd8cf4b790dd25
loginfilter.already.filtered: true
javax.servlet.error.request_uri: /plugins/servlet/notifications-miniview
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true
com.atlassian.confluence.web.ConfluenceJohnsonFilter_already_filtered: true
javax.servlet.error.exception: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
os_securityfilter_already_filtered: true
com.atlassian.seraph.auth.LoginReason: OK
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true
--------------------------
Parameters
--------------------------
caused by: org.eclipse.gemini.blueprint.service.ServiceUnavailableException: service matching filter=[(objectClass=com.atlassian.mywork.service.LocaleService)] unavailable
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:404)

2020-12-27 16:54:05,216 ERROR [http-nio-127.0.0.1-8090-exec-3] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@3a29b482
 -- traceId: c5224bcea37dd8cd
com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.confluence.plugins.confluence-feature-discovery-plugin]
	at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:95)
	at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:86)
	at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38)
	at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:84)
	at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:68)
	at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$withRequestContext$1(VCacheRequestContextOperations.java:59)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Could not read fields for table AO_6384AB_DISCOVERED
	at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:141)
	at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126)
	at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110)
	at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62)
....
Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes.
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431)
	at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:271)
	at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:91)
	... 65 more

The on-screen error at the end of the backup/export attempt is (again condensed):

com.atlassian.confluence.importexport.ImportExportException: java.lang.RuntimeException: Could not read fields for table AO_54C900_C_TEMPLATE_REF
	at com.atlassian.activeobjects.confluence.backup.ActiveObjectsBackupRestoreProvider.backup(ActiveObjectsBackupRestoreProvider.java:30)
	at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupPluginData(FileXmlExporter.java:215)
	at com.atlassian.confluence.importexport.impl.FileXmlExporter.backupEverything(FileXmlExporter.java:109)
	at com.atlassian.confluence.importexport.impl.AbstractXmlExporter.doExport(AbstractXmlExporter.java:94)
	at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExportInternal(FileXmlExporter.java:60)
	at com.atlassian.confluence.importexport.impl.FileXmlExporter.doExport(FileXmlExporter.java:54)
	at com.atlassian.confluence.importexport.DefaultImportExportManager.doExport(DefaultImportExportManager.java:212)
	at com.atlassian.confluence.importexport.DefaultImportExportManager.exportAs(DefaultImportExportManager.java:178)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
....
Caused by: java.lang.RuntimeException: Could not read fields for table AO_54C900_C_TEMPLATE_REF
	at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:141)
	at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126)
	at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110)
	at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62)
	at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:99)
	at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:97)
	at com.google.common.collect.Iterators$6.transform(Iterators.java:785)
....
Caused by: java.sql.SQLException: Index column size too large. The maximum column size is 767 bytes.
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1003)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:431)
	at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:271)
	at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:91) 
... 438 more 

Going to post back to Atlassian Community.

Comment by Peter Murray [ 28/Jan/21 ]

Atlassian support has a copy of our database and has reproduced a similar error on the same table:

Caused by: java.sql.SQLException: Cannot drop index 'index_ao_54c900_c_t852152353': needed in a foreign key constraint
Exception executing SQL update <DROP INDEX index_ao_54c900_c_t852152353 ON AO_54C900_C_TEMPLATE_REF>
java.sql.SQLException: Cannot drop index 'index_ao_54c900_c_t852152353': needed in a foreign key constraint

Atlassian support broke up the steps into migrating Confluence first from 6.15.10 to 7.8 then MySQL 5.6 to 8. In between those steps, the support person could create a new space and a new blueprint template. The error above happened after the MySQL upgrade.

Next Steps

I will engage our senior engineers to help us out here on how to fix it.

The support person did note that it took around 8 hours to load in our database dump onto their test system.

Comment by Peter Murray [ 03/Feb/21 ]

Heard back from Atlassian:

I just got word back from our senior engineer with our next steps.

“Error 1709: Index column size too large. The maximum column size is 767 bytes”.
This happens if you create a ROW_FORMAT=COMPRESSED table with utf8 encoding; it makes the table’s row too large.
You can avoid this problem by using ROW_FORMAT=DYNAMIC for that table instead.
The default row format for InnoDB tables is defined by innodb_default_row_format variable, which has a default value of DYNAMIC. The default row format is used when the ROW_FORMAT table option is not defined explicitly or when ROW_FORMAT=DEFAULT is specified.

So, let's check it out!
Next Steps

Please share the output of the following database queries:

SELECT table_name, row_format FROM information_schema.tables WHERE table_schema='foliowiki' order by table_name; --table_schema=<database_name>

SHOW GLOBAL VARIABLES where variable_name like 'innodb_default_row_format';

Also share your MySQL my.cnf file, please.

Send the requested output (also attached to this issue). There are three tables set to ROW_FORMAT="Compact":

  • AO_54C900_C_TEMPLATE_REF
  • AO_6384AB_DISCOVERED
  • AO_6384AB_FEATURE_METADATA_AO

The first two are the ones seen in Confluence error logs. Also sent back the RDS equivalent of a MySQL server `my.cnf` file (generated using this answer on Stack Exchange). Waiting to hear what I assume will be three SQL commands to alter the row format to "Dynamic" for these three tables.

Comment by Peter Murray [ 04/Feb/21 ]

Shutdown Confluence, made the changes below to the MySQL database, and restarted Confluence.

ALTER TABLE AO_54C900_C_TEMPLATE_REF ROW_FORMAT=DYNAMIC;
ALTER TABLE AO_6384AB_DISCOVERED ROW_FORMAT=DYNAMIC;
ALTER TABLE AO_6384AB_FEATURE_METADATA_AO ROW_FORMAT=DYNAMIC;

For all three commands, got:

ERROR: Index column size too large. The maximum column size is 767 bytes.

Going to need to figure out what indexes these are, drop them, alter the tables, and re-add them.

Comment by Peter Murray [ 16/Feb/21 ]

Still stuck.  My notes:

SQL indexes for AO_6384AB_FEATURE_METADATA_AO

Running ALTER TABLE AO_6384AB_FEATURE_METADATA_AO ROW_FORMAT=dynamic; results in ERROR: Index column size too large. The maximum column size is 767 bytes.

Attempting to enumerate indexes to drop them.
{{  show index from AO_6384AB_FEATURE_METADATA_AO;}}

 

Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment Visible Expression
AO_6384AB_FEATURE_METADATA_AO 0 PRIMARY 1 ID A 6961       BTREE     YES  
AO_6384AB_FEATURE_METADATA_AO 1 index_ao_6384ab_fea1458039816 1 CONTEXT A 262     YES BTREE     YES  
AO_6384AB_FEATURE_METADATA_AO 1 index_ao_6384ab_fea1458039816 2 KEY A 7089     YES BTREE     YES  

Running... drop index index_ao_6384ab_fea1458039816 on AO_6384AB_FEATURE_METADATA_AO; ...results in... Query 1 ERROR: Index column size too large. The maximum column size is 767 bytes.

Trying:
{{  mysqldump -h <rds-hostname> -u root -p foliowiki AO_6384AB_FEATURE_METADATA_AO > AO_6384AB_FEATURE_METADATA_AO.sql}}
... results in

Warning: A partial dump from a server that has GTIDs will by default include the GTIDs of all transactions, even those that changed suppressed parts of the database. If you don't want to restore GTIDs, pass --set-gtid-purged=OFF. To make a complete dump, pass --all-databases --triggers --routines --events.

mysqldump: Got error: 1709: Index column size too large. The maximum column size is 767 bytes. when doing LOCK TABLES

Possible pointers to solution

The fix appears to be in MySQL 8.0.22 and the current version of MySQL on RDS is 8.0.21. I have filed a forum request with AWS to see when this might be fixed.

Comment by Peter Murray [ 23/Feb/21 ]

Atlassian support suggests:

I can see that not even trying to drop it directly works.
If you give it a shot running this command first:

SET FOREIGN_KEY_CHECKS=0

Does it work using the

show create table

?

I will escalate this again internally, see if anybody else has any inputs on this one.
Thanks again for all your patience and assistance on this case. You are doing a phenomenal job.

I will keep you posted. Keep me posted too if the foreign key checks helped.

Unfortunately, the "ERROR: Index column size too large. The maximum column size is 767 bytes." remains.

Also unfortunately, MySQL 8.0.21 remains the latest version available and there was no reply to my forum message.

Comment by Peter Murray [ 23/Feb/21 ]

I can't dump the database either with mysqldump:

```
mysqldump -h folio-wiki-202010.<identifier>.us-east-1.rds.amazonaws.com -u root -p foliowiki > ~ec2-user/20210223-Confluence-dump.sql
Enter password:
Warning: A partial dump from a server that has GTIDs will by default include the GTIDs of all transactions, even those that changed suppressed parts of the database. If you don't want to restore GTIDs, pass --set-gtid-purged=OFF. To make a complete dump, pass --all-databases --triggers --routines --events.
mysqldump: Got error: 1709: Index column size too large. The maximum column size is 767 bytes. when using LOCK TABLES
```

AWS RDS offers to give me the dump in Apache Parquet format, but I don't think that is helpful/useful for editing and re-uploading to MySQL.

Comment by Peter Murray [ 16/Mar/21 ]

Just for everyone following along, I check every workday to see if AWS has adopted MySQL 8.0.22 that reportedly addresses the underlying database issue.  So far no luck, and no comment on my AWS forum message about supporting this version.

Comment by Peter Murray [ 23/Mar/21 ]

Still no MySQL 8.0.22.  sigh

Comment by Peter Murray [ 25/Mar/21 ]

Hey!  MySQL 8.0.23 just showed up on RDS!  I'll try to upgrade the Confluence database tonight and see if this issue is solved.

Comment by John Malconian [ 25/Mar/21 ]

woohoo!

Comment by Peter Murray [ 27/Mar/21 ]

Still stuck.  Going to ask to upgrade our AWS support plan to "Developer" so a technical support issue can be filed with AWS.  Notes from today's efforts below.


Steps

  1. Shutdown Confluence: systemctl stop confluence.service
  2. Snapshot RDS database instance. RDS snapshot
  3. Modify Confluence DB instance: RDS console—started 10:08:50, ended approx. 10:15
  4. Snapshot Confluence Data EC2 volume (Link to snapshot)
  5. Attempt “mysqldump”: mysqldump -h folio-wiki-202010.[dbidentifier].us-east-1.rds.amazonaws.com -u root -p foliowiki > ~ec2-user/20210327-Confluence-dump.sql
  6. Start Confluence: systemctl start confluence.service
  7. Test meeting notes macro
  8. Test Design Decision space macro
  9. Create new space (Product Owners)

Notes

Got stuck at step 4—getting the same error: mysqldump: Got error: 1709: Index column size too large. The maximum column size is 767 bytes. when using LOCK TABLES. See contents of “PrePatchCompatibility.log” below. Attempting stop/start of RDS instance.

Same result after restart. Running the ALTER TABLE AO_54C900_C_TEMPLATE_REF ROW_FORMAT=DYNAMIC MySQL command from Tables returns the same error. The “mysql-error-running.log” (below, last line) says the engine is still 8.0.20. Confirmed in RDS console that engine is still 8.0.20. Attempting to modify instance again.

Modification not successful—same entries in “PrePatchCompatibility.log”. “mysql-error-running.log” is empty. Attempting just a “reboot” of the RDS instance.

Nothing helpful in the AWS Forums for how to deal with a failing MySQL engine minor upgrade problem. Stuck. Restarting Confluence.

Contents of AWS RDS “PrePatchCompatibility.log”

 

Executing Compatibility Checks for the MySQL server at localhost:3306.
 Source Version: 8.0.20 - Source distribution, Target Version: 8.0.23.
1) Issues reported by 'check table x for upgrade' command
 foliowiki.AO_6384AB_DISCOVERED - Index column size too large. The maximum column size is 767 bytes.
 foliowiki.AO_6384AB_DISCOVERED - Table 'foliowiki.AO_6384AB_DISCOVERED' doesn't exist
 foliowiki.AO_6384AB_DISCOVERED - Corrupt
 foliowiki.AO_6384AB_FEATURE_METADATA_AO - Index column size too large. The maximum column size is 767 bytes.
 foliowiki.AO_6384AB_FEATURE_METADATA_AO - Table 'foliowiki.AO_6384AB_FEATURE_METADATA_AO' doesn't exist
 foliowiki.AO_6384AB_FEATURE_METADATA_AO - Corrupt
 foliowiki.AO_54C900_C_TEMPLATE_REF - Index column size too large. The maximum column size is 767 bytes.
 foliowiki.AO_54C900_C_TEMPLATE_REF - Table 'foliowiki.AO_54C900_C_TEMPLATE_REF' doesn't exist
 foliowiki.AO_54C900_C_TEMPLATE_REF - Corrupt
2) Column definition mismatch between InnoDB Data Dictionary and actual table definition.
 No issues found.
3) Tables with dangling FULLTEXT index reference
 No issues found.
4) MySQL pre-checks that all database, table, and trigger names are lowercase when the lower_case_table_names parameter is set to 1.
 No issues found.
Errors: 1
 Warnings: 0
 Database Objects Affected: 9
 ----------------------- END OF LOG ----------------------

 

Contents of mysql-error-running.log

2021-03-27T14:41:39.958254Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-file' is deprecated and will be removed in a future release.
 2021-03-27T14:41:39.958437Z 0 [System] [MY-010116] [Server] /rdsdbbin/mysql/bin/mysqld (mysqld 8.0.20) starting as process 5850
 2021-03-27T14:41:39.966027Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
 2021-03-27T14:41:39.969348Z 1 [Warning] [MY-012191] [InnoDB] Scan path '/rdsdbdata/db/innodb' is ignored because it is a sub-directory of '/rdsdbdata/db/
 2021-03-27T14:41:41.245548Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
 2021-03-27T14:41:41.540259Z 0 [Warning] [MY-013414] [Server] Server SSL certificate doesn't verify: unable to get issuer certificate
 2021-03-27T14:41:41.596141Z 0 [System] [MY-010931] [Server] /rdsdbbin/mysql/bin/mysqld: ready for connections. Version: '8.0.20' socket: '/tmp/mysql.sock' port: 3306 Source distribution.

Links

Comment by Julian Ladisch [ 30/Mar/21 ]

Have you tried to fix it using mysql-debug as described on https://dba.stackexchange.com/questions/234822/mysql-error-seems-unfixable-index-column-size-too-large/252255#252255 ?

Comment by Peter Murray [ 30/Mar/21 ]

Unfortunately, with RDS this isn't an option....I don't have a way to invoke MySQL-Debug using the AWS-hosted MySQL.

Comment by Julian Ladisch [ 30/Mar/21 ]

Does AWS allow you to download the MySQL data directory so that you can copy the raw data files into some other server where mysql-debug is available?

Comment by Peter Murray [ 30/Mar/21 ]

Good idea, but no.  The only format AWS itself offers is the Apache Parquet format.  I did some research on that format, and there doesn't seem to be a way to transform it into a MySQL dump file or MySQL directory/index format.  And of course the underlying issue is preventing mysqldump from working.

Thanks for thinking about this...I'll take any suggestions and try to pursue them!

Comment by Peter Murray [ 08/Apr/21 ]

Not waiting for Technical Council approval of the AWS Support expense...I upgraded the FOLIO-Org AWS Organization to "Developer"-level support and filed an issue:

We have an RDS MySQL database that has tables in a state where a fix in MySQL engine 8.0.22 is required, but the PrePatchCompatibility tests are failing and won't allow the upgrade through the AWS console. Rather than to repeat the details here, may I point you to this link in our public issue tracker: https://folio-org.atlassian.net/browse/FOLIO-2867?focusedCommentId=197579&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel

That points to a comment February 15, 2021, where a root cause was identified, upgrading to MySQL engine 8.0.22 was suggested as the fix, and our subsequent attempts to upgrade the MySQL engine in RDS. DB Instance(s): folio-wiki-202010

https://console.aws.amazon.com/support/home#/case/?displayId=8200441491&language=en

 

Comment by Peter Murray [ 12/Apr/21 ]

Received reply back from AWS that correctly summarized the issue. The AWS tech wrote:

Last but not the least, in order to expedite the investigation, could you please share the following information with me?
1. Please execute the following command on the instance in question as the master user and share the result with me.
SELECT distinct a.*
FROM
(SELECT ic.table_schema, ic.table_name, ist.index_name, if(ist.SUB_PART IS NULL, (ic.CHARACTER_MAXIMUM_LENGTH * ics.MAXLEN), (ist.SUB_PART * ics.MAXLEN)) as INDEX_LEN
FROM
information_schema.columns ic
JOIN
information_schema.tables it
ON it.table_schema=ic.table_schema and it.table_name=ic.table_name
JOIN
information_schema.STATISTICS ist
ON ist.table_schema=ic.table_schema and ist.table_name=ic.table_name and ist.column_name=ic.column_name
JOIN
information_schema.CHARACTER_SETS ics
ON ics.CHARACTER_SET_NAME = ic.CHARACTER_SET_NAME
WHERE
it.table_schema not in ('information_schema','mysql','performance_schema') and it.row_format in ('Redundant','Compact','Compact or Redundant')
) AS a
WHERE a.INDEX_LEN > 767;

2. Since AWS highly values customers' privacy, we as AWS engineers do not have access to customers' data. Could you please share the following log files with me? If the contents posted at [1] were all the contents in the files, please feel free to let me know.
a. PrePatchCompatibility.log
b. mysql-error-running.log

3. Was the instance upgraded from a version prior to MySQL 8? If it was, may I know what the previous major version or the upgrade path history was?

Replied with details:

Your summary of the issues is correct. The attached screen capture of the results of the MySQL command (attached as "Case-8200441491-MySQL-results.jpg") confirm those three tables are the issue. The contents in the comment at the Jira issue link in the previous message are the complete contents of those two logs.

You also asked: "Was the instance upgraded from a version prior to MySQL 8? If it was, may I know what the previous major version or the upgrade path history was?" — Yes, this instance was upgraded from MySQL engine 5.6 using the RDS-supplied upgrade path. (In other words, this database has always been on RDS, so RDS performed the 5.6→8 upgrade.) If RDS wrote logs for that upgrade, I did not keep them.

And heard back:

I have escalated the problem you encountered to RDS MySQL team for further investigation. In general, the process will take several days, which depends on the complexity of the case. Please rest assured and I will keep you informed. During the investigation, please feel free to contact us if you have any question.

Comment by Peter Murray [ 13/Apr/21 ]

From AWS:

I just got an update from RDS MySQL team and please let me share it with you.

Based on the preliminary findings, the indexes prevented the upgrade from running successfully. RDS MySQL team would like to know if it is feasible for you to perform PITR [1] on the instance to a time which the indexes in question could be dropped and then try to perform the upgrade again.

Having read the thread, I understand the problem have persists for some time so PITR might not be a useful solution. If PITR is not feasible for you, could you please help create a read replica [2] of folio-wiki-202010 and the read replica is dedicated (no workload on it) for RDS MySQL team to investigate? If creating a read replica is convenient for you, please share the Amazon Resource Name (ARN) of the read replica instance and your permission with us. RDS MySQL team will continue to work on the read replica so the instance folio-wiki-202010 would not be affected.

I anticipate your reply. Please feel free to let me know your preference.

[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html
[2] https://docs.aws.amazon.com/AmazonRDS/latest/RDSonVMwareUserGuide/read-replica.html#creating-read-replica

My reply:

Unfortunately, PITR is not going to be an option for us...time has marched on and the users have continued to add data to the wiki. (Although functionality was impaired in Confluence, it wasn't so bad as to have us restore the system to the state prior to the upgrade.)

The read replica of `folio-wiki-202010` is ready for the RDS MySQL team. The ARN is:

arn:aws:rds:us-east-1:732722833398:db:folio-wiki-202010-read-replica-for-aws-troubleshooting

...and the RDS MySQL team has permission to use this replica for troubleshooting. We will not be making connections to it, so it is dedicated to your use.

 

Comment by Peter Murray [ 14/Apr/21 ]

From AWS:

I understand that PITR is not feasible for you and a read replica was created from the instance in question for RDS MySQL team to investigate. Please correct me if I am wrong.

The information of the read replica arn:aws:rds:us-east-1:732722833398:db:folio-wiki-202010-read-replica-for-aws-troubleshooting and your permission were shared with RDS MySQL team. I will follow the progress closely and keep you informed.

Please feel free to contact me in the interim if there is any question.

Comment by Peter Murray [ 19/Apr/21 ]

From: no-reply-aws@amazon.com <no-reply-aws@amazon.com>
Date: Apr 18, 2021, 7:44 PM -0400
To: folio-aws@openlibraryfoundation.org
Subject: RE:[CASE 8200441491] Cannot upgrade engine to 8.0.23—fails PrePatchCompatibility tests

Thank you for being patient with me.

RDS MySQL team is still working on the instance. I will keep an eye on it and will inform you of any progress.

Comment by Peter Murray [ 23/Apr/21 ]

From: no-reply-aws@amazon.com <no-reply-aws@amazon.com>
Date: Apr 23, 2021, 4:50 AM -0400
To: folio-aws@openlibraryfoundation.org
Cc: peter.murray@openlibraryfoundation.org, olf-infra@ole-lists.openlibraryfoundation.org
Subject: RE:[CASE 8200441491] Cannot upgrade engine to 8.0.23—fails PrePatchCompatibility tests

Thank you for your patience.

RDS MySQL team is still working on the problem you encountered and I am following it closely. I will inform you of the progress if there is any update.

Comment by Peter Murray [ 26/Apr/21 ]

From: no-reply-aws@amazon.com <no-reply-aws@amazon.com>
Date: Apr 25, 2021, 9:24 PM -0400
To: folio-aws@openlibraryfoundation.org
Cc: peter.murray@openlibraryfoundation.org, olf-infra@ole-lists.openlibraryfoundation.org
Subject: RE:[CASE 8200441491] Cannot upgrade engine to 8.0.23—fails PrePatchCompatibility tests

I am sorry for the late reply.

I just got an update from RDS MySQL team and please let me share it with you.

The team understand any customer database is critical and is testing the solution thoroughly. The testing is almost close to completion and the fix would be applied to the instance arn:aws:rds:us-east-1:732722833398:db:folio-wiki-202010-read-replica-for-aws-troubleshooting in two days.

Thank you for your patience and understanding. I will get back to you once it is fixed.

Comment by Peter Murray [ 28/Apr/21 ]

From: "no-reply-aws@amazon.com" <no-reply-aws@amazon.com>
Subject: RE:[CASE 8200441491] Cannot upgrade engine to 8.0.23—fails PrePatchCompatibility tests
Date: April 27, 2021 at 8:55:19 PM EDT
To: folio-aws@openlibraryfoundation.org
Cc: peter.murray@openlibraryfoundation.org, olf-infra@ole-lists.openlibraryfoundation.org

Thank you for being patient with me.

The team informed me that the fix was applied successfully on the instance arn:aws:rds:us-east-1:732722833398:db:folio-wiki-202010-read-replica-for-aws-troubleshooting.
Could you please take a snapshot [1] of it first to keep the current state and check if the problem is fixed on your side? If the problem is gone, you may consider:
1. Restoring another instance from the snapshot [2] taken from the read replica and testing the upgrade.
2. If it could be upgraded successfully, you could upgrade the read replica first. Then, please schedule a maintenance window to promote the read replica [3] and redirect the traffic from applications to it. Please make sure the read replica is in sync with the primary before the promotion to prevent data loss.

The following is an excerpt of what the document [4] says of "Q: How do I see the status of my active read replica(s)?":
"The number of seconds that the read replica is behind the master is published as an Amazon CloudWatch metric ("Replica Lag") available via the AWS Management Console or Amazon CloudWatch APIs. For Amazon RDS for MySQL, the source of this information is the same as that displayed by issuing a standard "Show Slave Status" MySQL command against the read replica."

Therefore, you may take checking the metric Replica Lag into consideration before the read replica is promoted.

Hope it is useful. Please feel free to contact me if further assistance is needed.

[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateSnapshot.html
[2] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html
[3] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote
[4] https://aws.amazon.com/rds/faqs/

Comment by Peter Murray [ 28/Apr/21 ]

(Fixing the formatting)

From the above, I gather that they have fixed the underlying tables without upgrading the MySQL engine. That’s a pretty good trick. I confirmed that the replica is still MySQL 8.0.20.

  1. Shutdown Confluence: systemctl stop confluence.service
  2. Snapshot the replica RDS database instance called folio-wiki-202010-read-replica-for-aws-troubleshooting.
  3. Restore snapshot to new MySQL RDS instance called folio-wiki-2021-04 – started 10:47, ended 10:57
  4. Modify restored snapshot configuration to 8.0.23: RDS console—started 11:01, ended approx. 11:28
  5. Check “PrePatchCompatibility.log” of folio-wiki-2021-04
  6. Check output of MySQL command to see if all of the tables are dynaminc: SELECT table_name, row_format FROM information_schema.tables WHERE table_schema='foliowiki' order by row_format;
    1. If not, run ALTER TABLE AO_54C900_C_TEMPLATE_REF ROW_FORMAT=DYNAMIC; ALTER TABLE AO_6384AB_DISCOVERED ROW_FORMAT=DYNAMIC; ALTER TABLE AO_6384AB_FEATURE_METADATA_AO ROW_FORMAT=DYNAMIC;
  7. Attempt “mysqldump”: mysqldump -h folio-wiki-2021-04.c4levujzofum.us-east-1.rds.amazonaws.com -u root -p foliowiki > ~ec2-user/20210427-Confluence-dump.sql
  8. Update Confluence configuration hibernate.connection.url parameter to point to new database (/atlassian/var/application-data/confluence/confluence.cfg.xml)
  9. Start Confluence: systemctl start confluence.service
  10. Test meeting notes macro
  11. Test Design Decision space macro
  12. Create new space (Product Owners)

PrePatchCompatability.log

Executing Compatibility Checks for the MySQL server at localhost:3306.
 Source Version: 8.0.20 - Source distribution, Target Version: 8.0.23.
1) Issues reported by 'check table x for upgrade' command
 No issues found.
2) Column definition mismatch between InnoDB Data Dictionary and actual table definition.
 No issues found.
3) Tables with dangling FULLTEXT index reference
 No issues found.
Errors: 0
 Warnings: 0
 Database Objects Affected: 0
 ----------------------- END OF LOG ----------------------

Confluence template notice of upgrade

 

!https://folio-org.atlassian.net/wiki/download/attachments/4227086/Caution.png?version=1&modificationDate=1602190749000&api=v2|align=left,alt=Caution! *{color:red}A fix was applied to address the wiki database issue.{color}* // *{color:red}Please report immediately any unusual or broken wiki features to @peter-murray on Slack or by email to peter@indexdata.com{color}.*
----

 

Comment by Peter Murray [ 28/Apr/21 ]

Using Confluences Troubleshooting and Support Tools → Log analyzer, there are no relevant errors.  (There is an error related to failing SMTP credentials, but that can be safely ignored for this issue.)  The only symptom that remains from the original list is that the user notifications dropdown returns an error:

Cause

java.lang.RuntimeException: java.util.concurrent.TimeoutException
    at com.google.common.base.Throwables.propagate(Throwables.java:241)

caused by: java.util.concurrent.TimeoutException
    at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.tryExecuteUnderLock(LockedTaskExecutor.java:55)

Stack Trace:[hide]

java.lang.RuntimeException: java.util.concurrent.TimeoutException at com.google.common.base.Throwables.propagate(Throwables.java:241) at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.executeUnderLock(LockedTaskExecutor.java:106) at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getLastModified(LocalRegistrationServiceImpl.java:192) at com.atlassian.mywork.host.service.LocalRegistrationServiceImpl.getCacheValue(LocalRegistrationServiceImpl.java:223)
...
Caused by: java.util.concurrent.TimeoutException
at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.tryExecuteUnderLock(LockedTaskExecutor.java:55)
at com.atlassian.mywork.host.util.concurrent.LockedTaskExecutor.executeUnderLock(LockedTaskExecutor.java:96)
... 318 more

I'm going to add that to the issue with Atlassian, but it may be a separate problem.  I'm uploading a copy of the database dump and a support Zip to Atlassian for them to look at.

 

 

Comment by Peter Murray [ 29/Apr/21 ]

Confluence has been running on the new database for over 24 hours and no new issues have been reported.  The one remaining issue—an error and stack trace in the Notifications drop down menu—has been assigned a new issue at Atlassian (CSP-291643) and is now tracked as FOLIO-3136 Closed .  Here is the final message from Atlassian about the indexing problem.

 

From: Rodrigo Oliveira <confluence-support-system@am.atlassian.com>

Date: Apr 28, 2021, 3:33 PM -0400

To: peter.murray@openlibraryfoundation.org

Subject: CSP-286216 "java.sql.SQLException: Index column size too large" error after upgrading Confluence and MySQL

 

That's fantastic! It took us a while, but you finally fixed it! Congratulations on chasing AWS to fix this one. Amazing news, indeed!

I will be looking at the new set of data you just shared with us! Hopefully, everything else will stay okay Unable to render embedded object: File (jira-generated-image-static-smile-f7911317-98aa-421b-910a-001a253684c3) not found.

I also went ahead and created a new ticket on your behalf so we can keep track of this other issue: The Notification drop-down returns an error with a stack trace. . The support zip is already attached there too.

I will keep you posted. Great news today!

Generated at Thu Feb 08 23:23:50 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.