[LIQUTIL-25] Liquibase Tenant operation Validation Failed: 5 change sets check sum Created: 28/Jun/22  Updated: 03/Nov/23  Resolved: 29/Aug/22

Status: Closed
Project: liquibase-util
Components: None
Affects versions: None
Fix versions: 1.5.2

Type: Bug Priority: P2
Reporter: Julian Ladisch Assignee: Ruslan Lavrov
Resolution: Done Votes: 0
Labels: epam-folijet, folijet-support, no-epic-needed, security, security-reviewed
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Defines
defines UXPROD-3576 NFR: Data Import Support Bug work (No... Closed
Relates
relates to MODKBEKBJ-656 mod-kb-ebsco-java failing to migrate ... Closed
Sprint: EPAM-Veg Sprint 145
Story Points: 3
Development Team: Other dev
Release: Nolana (R3 2022)
RCA Group: Implementation coding issue

 Description   

Upgrading mod-pubsub fails with this error in a multi-tenant environment:

Tenant operation failed for module mod-pubsub-2.5.1: liquibase.exception.ValidationFailedException: Validation Failed:
     5 change sets check sum
          liquibase/tenant/scripts/v-0.0.1/2019-09-06--15-00-create-audit_message-table.xml::2019-09-06--15-00-create-message_state-type::KaterynaSenchenko was: 8:fd0d822c7d8aba9ebb82717a6dcdc080 but is now: 8:0677f5b7f9c5d8c8417286d9256d5702
          liquibase/tenant/scripts/v-0.0.1/2019-09-06--15-00-create-audit_message-table.xml::2019-09-06--15-00-create-audit_message-table::KaterynaSenchenko was: 8:668b7c6a54e50398c827aeb2a3ce2668 but is now: 8:23275735d59fcce794dc0014766bf9ec
          liquibase/tenant/scripts/v-0.0.1/2019-11-08--15-00-change-audit_message-table.xml::2019-11-08--15-00-add-rejected-message_state::KaterynaSenchenko was: 8:9a0377e6e9a932d18421939736ab59ce but is now: 8:4b3d5e142bfa4e8675911c05639fa630
          liquibase/tenant/scripts/v-0.0.1/2019-11-08--15-00-change-audit_message-table.xml::2019-11-08--15-00-update-audit_message-table::KaterynaSenchenko was: 8:cb80ed99ac6688a68aa692489350458f but is now: 8:eb01c205f94be6e775d70f215f61bbdc
          liquibase/tenant/scripts/v-0.0.1/2020-06-20--15-00-update-audit-message-payload-data.xml::2020-06-20--13-00-update-audit-message-payload-data::KaterynaSenchenko was: 8:190718a406903987d6985a9c5ae9e203 but is now: 8:60eea3ce275d7f32dc840547ee8b1ded

Workaround: Manually run

set search_path to minerva4_mod_pubsub ;
update databasechangelog set md5sum='8:0677f5b7f9c5d8c8417286d9256d5702' where md5sum='8:fd0d822c7d8aba9ebb82717a6dcdc080';
update databasechangelog set md5sum='8:23275735d59fcce794dc0014766bf9ec' where md5sum='8:668b7c6a54e50398c827aeb2a3ce2668';
update databasechangelog set md5sum='8:4b3d5e142bfa4e8675911c05639fa630' where md5sum='8:9a0377e6e9a932d18421939736ab59ce';
update databasechangelog set md5sum='8:eb01c205f94be6e775d70f215f61bbdc' where md5sum='8:cb80ed99ac6688a68aa692489350458f';
update databasechangelog set md5sum='8:60eea3ce275d7f32dc840547ee8b1ded' where md5sum='8:190718a406903987d6985a9c5ae9e203';

It seems that mod-pubsub used a wrong tenant when calculating the md5sum. This only happens in a multi-tenant environment.



 Comments   
Comment by Julian Ladisch [ 05/Jul/22 ]

Bumping priority from P3 to P2 because the root cause is critical: Liquibase has used the tenant id of a different tenant when module was enabled in a multi-tenant environment.

You can compare the md5sum values on a test server after you enable the module for multiple tenants and restart the module before you enable it for a single tenant.

Comment by Craig McNally [ 07/Jul/22 ]

Stephanie Buck the Security team wanted to make sure this was on your radar as it's a P2 and fairly serious.

Comment by Stephanie Buck [ 07/Jul/22 ]

Thanks, Craig McNally. It's towards the top of our backlog and I've moved the release up to MG bugfix. 

Comment by Oleksandr Vidinieiev [ 21/Jul/22 ]

Julian Ladisch I am struggling to reproduce this issue locally. Is there a specific set of steps I can follow to see this error?

Comment by Alexander Kurash [ 27/Jul/22 ]

Julian Ladisch Kateryna Senchenko We can't reproduce this issue yet, neither manually nor with a test, but we have found a potential issue in folio-liquibase-util - it doesn't specify the schema for liquibase tables explicitly (potential fix is here https://github.com/folio-org/folio-liquibase-util/pull/22). We also noticed that check sums that you have in the minerva4_mod_pubsub schema (like 8:fd0d822c7d8aba9ebb82717a6dcdc080, 8:668b7c6a54e50398c827aeb2a3ce2668) are the same check sums we have on our scratch env for tenant diku which also suggests that liquibase has chosen the wrong schema for its own tables.

Comment by Ann-Marie Breaux (Inactive) [ 29/Jul/22 ]

Alexander Kurash will Vega release this bugfix, or does Folijet need to do it? If Folijet, please coordinate with Kateryna Senchenko . Thank you!

Comment by Ann-Marie Breaux (Inactive) [ 10/Aug/22 ]

Hi Kateryna Senchenko and Ruslan Lavrov Do we need to move this to Folijet, or leave as Vega?

Comment by Ann-Marie Breaux (Inactive) [ 11/Aug/22 ]

Switched to Folijet dev team and Folijet MG Bug feature

Comment by Ruslan Lavrov [ 17/Aug/22 ]

Hello Julian Ladisch, could you please describe the steps on how to reproduce this issue? I set up a second tenant on my local env and then performed mod-pub-sub deployment and upgrade modules but didn't manage to reproduce the issue.

Comment by Ruslan Lavrov [ 19/Aug/22 ]

Hello Julian Ladisch, please attach the content of the "diku_mod_pubsub.databasechangelog" and "minerva4_mod_pubsub.databasechangelog" tables from the environment where the issue has been reproduced.

Comment by Ann-Marie Breaux (Inactive) [ 19/Aug/22 ]

Hi Julian Ladisch Good to talk with you just now - thank you for trying to dig up the old logs and/or reproducing and including the repro steps and new logs!

Ruslan Lavrov We should have more info by Monday. Let's review at standup Monday and figure out next steps.

Comment by Julian Ladisch [ 19/Aug/22 ]
# select * from minerva4_mod_pubsub.databasechangelog order by orderexecuted;
id|2019-09-06--13-00-create-audit_message_payload-table
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-09-06--13-00-create-audit_message_payload-table.xml
dateexecuted|2020-05-04 15:05:20.514288
orderexecuted|1
exectype|EXECUTED
md5sum|8:a50b24b5c7a5a0d64c3d0fc87647fa12
description|createTable tableName=audit_message_payload
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2019-09-06--15-00-create-message_state-type
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-09-06--15-00-create-audit_message-table.xml
dateexecuted|2020-05-04 15:05:20.51809
orderexecuted|2
exectype|EXECUTED
md5sum|8:0677f5b7f9c5d8c8417286d9256d5702
description|sql
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501
         
id|2019-09-06--15-00-create-audit_message-table
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-09-06--15-00-create-audit_message-table.xml
dateexecuted|2020-05-04 15:05:20.522376
orderexecuted|3
exectype|EXECUTED
md5sum|8:23275735d59fcce794dc0014766bf9ec
description|createTable tableName=audit_message
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2019-11-08--15-00-add-rejected-message_state
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-11-08--15-00-change-audit_message-table.xml
dateexecuted|2020-05-04 15:05:20.525878
orderexecuted|4
exectype|EXECUTED
md5sum|8:4b3d5e142bfa4e8675911c05639fa630
description|sql
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2019-11-08--15-00-update-audit_message-table
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-11-08--15-00-change-audit_message-table.xml
dateexecuted|2020-05-04 15:05:20.530737
orderexecuted|5
exectype|EXECUTED
md5sum|8:eb01c205f94be6e775d70f215f61bbdc
description|addColumn tableName=audit_message; addColumn tableName=audit_message; dropNotNullConstraint columnName=correlation_id, tableName=audit_message; dropNotNullConstraint columnName=created_by, tableName=audit_message
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2019-11-12--17-00-drop-foreign-key-constraint-on-message-payload
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-11-12--17-00-drop-foreign-key-constraint-on-message-payload.xml
dateexecuted|2020-05-04 15:05:20.533186
orderexecuted|6
exectype|EXECUTED
md5sum|8:a18313efdc5cd2e8bb887a324b8dbf0a
description|dropForeignKeyConstraint baseTableName=audit_message, constraintName=fk_event_id
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2019-11-19--15-00-create-user-table
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2019-11-19--15-00-create-user-table.xml
dateexecuted|2020-05-04 15:05:20.537242
orderexecuted|7
exectype|EXECUTED
md5sum|8:c77c51520e63f82d11240b6e5b7207c2
description|createTable tableName=user; insert tableName=user
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2020-04-20--15-00-update-audit_message-table
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2020-04-20-15-00-change-audit_message-table.xml
dateexecuted|2020-05-04 15:05:20.53963
orderexecuted|8
exectype|EXECUTED
md5sum|8:fe78c1405824c7b0c1d8cc6ee95b1e04
description|addColumn tableName=audit_message
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|8604720501

id|2020-06-20--13-00-update-audit-message-payload-data
author|KaterynaSenchenko
filename|liquibase/tenant/scripts/v-0.0.1/2020-06-20--15-00-update-audit-message-payload-data.xml
dateexecuted|2020-07-24 10:12:26.764879
orderexecuted|9
exectype|EXECUTED
md5sum|8:60eea3ce275d7f32dc840547ee8b1ded
description|sql
comments|
tag|
liquibase|3.8.9
contexts|
labels|
deployment_id|5585546749

id|2021-04-26--15-00-drop-user-table
author|AlexanderKurash
filename|liquibase/tenant/scripts/v-0.0.1/2021-04-26--15-00-drop-user-table.xml
dateexecuted|2022-06-28 14:43:03.853728
orderexecuted|10
exectype|EXECUTED
md5sum|8:b2f2e0038e73df6dbdb9dccc19b8698f
description|dropTable tableName=user
comments|
tag|     
liquibase|4.9.0
contexts|
labels|
deployment_id|6427383622

The environment doesn't have a diku tenant and doesn't have diku_mod_pubsub schema.

Comment by Julian Ladisch [ 19/Aug/22 ]

I cannot reproduce the issue with the current version of mod-pubsub, it might have been caused by an old version and a specific sequence of multiple tenant installs/upgrades.

Comment by Ann-Marie Breaux (Inactive) [ 19/Aug/22 ]

Thank you very much, Julian Ladisch.

Ruslan Lavrov looks like we should be able to resolve this on Monday, and maybe a MG BF will not be needed after all. Please discuss with Kateryna Senchenko or Julian Ladisch if any additional questions.

cc: Ivan Kryzhanovskyi

Comment by Ruslan Lavrov [ 22/Aug/22 ]

Thank you Julian Ladisch! It looks like "databasechangelog" table for some reason was copied manually from "diku" tenant from some environment to the "minerva4" tenant.
During investigation found that check sums values which mentioned like "was: 8:fd0d822c7d8aba9ebb82717a6dcdc080" are relevant for "diku" tenant. And values mentioned like
"but is now: 8:0677f5b7f9c5d8c8417286d9256d5702" relevant for the "minerva4" tenant (when minerva4 tenantId is specified during enabling/upgrade). I got the such values for "diku" and "minerva4" tenants on my local environment correspondingly. The fact that the environment does not have "diku" tenant also can say that "databasechangelog" table for "minerva4" could be replaced manually from "diku" (possibly from vagrant env) for some reason (maybe during some troubleshooting).
Julian, tell please the described workaround from the ticket description has been applied to those check sums, right?

Comment by Julian Ladisch [ 22/Aug/22 ]

Yes, the workaround has been applied to those check sums.

From the entries you see that the installation was made in May 2020, a long time ago. We know that we did NOT copy the databasechangelog table. What might have happened is an installation for the diku tenant caused by an unintended copy+paste error, then a deletion of that tenant, and then an installation of the minerva4 tenant.

I don't know which mod-pubsub version had been used at that time.

The pull request is a good idea. It should be merged even if we don't know and cannot easily test whether it actually fixes the problem.

Comment by Ruslan Lavrov [ 22/Aug/22 ]

Julian Ladisch, is it an assumption, or do we know for sure that we had "diku" tenant in this environment previously?

Comment by Julian Ladisch [ 23/Aug/22 ]

It's an assumption. It was May 2020, a long time ago.

Comment by Ann-Marie Breaux (Inactive) [ 23/Aug/22 ]

Hi Olamide Kolawole and Kateryna Senchenko Could you review this together, and possibly discuss with Julian Ladisch. To me, it's unclear if these changes actually need to be merged, or if the situation may have already been resolved. It feels risky to me for us to merge it if it cannot be properly tested.

Comment by Julian Ladisch [ 25/Aug/22 ]

I agree that it is unlikely that the pull request with setLiquibaseSchemaName fixes the issue. It doesn't harm, though.

I suggest to close this issue as "Cannot reproduce".

Comment by Kateryna Senchenko [ 25/Aug/22 ]

Hi Ann-Marie Breaux, we decided to merge this PR, but for Nolana. We won't release it as a MG bugfix

Comment by Ann-Marie Breaux (Inactive) [ 25/Aug/22 ]

OK, and I see that the release field and linked feature have been switched to Nolana. So we'll just let is proceed through the regular non-bugfix Jira flow. Thank you Kateryna Senchenko and Julian Ladisch

Comment by Ann-Marie Breaux (Inactive) [ 28/Aug/22 ]

Julian Ladisch and Kateryna Senchenko Anything that needs to be manually tested on Snapshot before this Jira can be closed?

Comment by Julian Ladisch [ 28/Aug/22 ]

No. Jira can be closed as done.

Generated at Thu Feb 08 22:13:41 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.