[MSEARCH-74] Outage due to permissions error with mod-search Created: 08/Mar/21  Updated: 09/Mar/21  Resolved: 09/Mar/21

Status: Closed
Project: mod-search
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Marc Johnson Assignee: Hongwei Ji
Resolution: Won't Do Votes: 0
Labels: environments, outage
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by OKAPI-996 Reload permissions immediately on ena... Closed
Sprint:
Development Team: Falcon

 Description   

The folio-snapshot build fails with the following error:

fatal: [10.36.1.88]: FAILED! => {"changed": false, "connection": "close", "content": "POST request for mod-search-1.0.0-SNAPSHOT.33 /_/tenant failed with 500: {\"errors\":[{\"message\":\"[422 Unprocessable Entity] during [POST] to [http://perms/users] [PermissionsClient#assignPermissionsToUser(Permissions)]: [{\\n  \\\"errors\\\" : [ {\\n    \\\"message\\\" : \\\"Unable to update derived fields: Attempting to add non-existent permissions inventory-storage.inventory-view.instances.collection.get to permission user with id 14a02f1f-9fbc-4ccc-9523-4cd9f474abf3\\\",\\n    \\\"type\\\" : \\\"1\\\",\\n    \\\"code\\\" : \\\"-1\\\",\\n    \\\"parameters\\\" : [ {\\n      \\\"key\\\" : \\\"id\\\",\\n      \\\"value\\\" : \\\"14a02f1f-9fbc-4ccc-9523-4cd9f474abf3\\\"\\n    } ]\\n  } ]\\n}]\",\"type\":\"UnprocessableEntity\",\"code\":\"unknown_error\"}],\"total_records\":1}", "content_length": "715", "content_type": "text/plain", "elapsed": 157, "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://10.36.1.88:9130/_/proxy/tenants/diku/install?deploy=true&tenantParameters=loadSample%3Dtrue%2CloadReference%3Dtrue", "vary": "origin"}


 Comments   
Comment by Marc Johnson [ 08/Mar/21 ]

Craig McNally

I thought that mod-permissions created dummy permissions for nested permissions that didn't exist?

Comment by Ian Hardy [ 08/Mar/21 ]

looks like mod-search is trying to build a system user of some type https://github.com/folio-org/mod-search/commit/095cb98e462e89cf5468aeb4a3f54725cf4f6ff9 and perhaps failing there. Looks like the error okapi is throwing is saying this perm doesn't exist yet https://github.com/folio-org/mod-search/blob/095cb98e462e89cf5468aeb4a3f54725cf4f6ff9/src/main/resources/permissions/mod-search.csv. 

 

I also notice thie weird address which seems like an undefined variable for an okapi url.
http://perms/users

Comment by Marc Johnson [ 08/Mar/21 ]

What is slightly odd about this error is that the permission is defined by mod-inventory-storage which should be enabled before mod-search based upon it depending upon on the instance-storage interface.

Craig McNally Does mod-permissions apply any rules about which permissions can be granted to a user?

Comment by Marc Johnson [ 08/Mar/21 ]

Ian Hardy

I also notice this weird address which seems like an undefined variable for an okapi url.

That is odd and could easily be the cause of an error like this. However, the text Attempting to add non-existent permissions is triggered by mod-permissions which suggests the actual module is being invoked.

The check for this in mod-permissions doesn't seem to include any other criteria other than the permission name.

Comment by Marc Johnson [ 08/Mar/21 ]

I think we are left with the same two options:

  • Revert these changes
  • Remove mod-search and ui-inventory-search from the snapshot definition of platform complete
Comment by Ian Hardy [ 08/Mar/21 ]

Marc Johnson I think you're right I'm inclined to revert the changes I guess, not sure if others have an opinion.

 

fwiw, mod-inventory-storage is enabled on diku on the broken build (as expected) but there is no entry in the mod-permissions db for the perm in question:

okapi_modules=# select jsonb->>'permissionName' from diku_mod_permissions.permissions where jsonb->>'permissionName' like 'inventory-storage.inventory%';
 ?column? 
 ----------
(0 rows)
Comment by Marc Johnson [ 08/Mar/21 ]

Ian Hardy

fwiw, mod-inventory-storage is enabled on diku on the broken build (as expected) but there is no entry in the mod-permissions db for the perm in question

Then it might be worth getting the Core Platform team involved to understand why that permission isn't loaded.

Comment by Adam Dickmeiss [ 08/Mar/21 ]

I think this is a side effect of OKAPI-985 Closed . OKAPI-985 Closed solved an issue with Honeysuckle modules. I think Okapi should only refresh "last" if mod-permissions is "upgraded".

Dummy permissions are for modules that refer to yet-to-be defined permissions. I'm not sure users can be assigned those.

It's really a problem that we are going to have a gazilion users and passwords floating around.

The better approach would be that tenant init provided a token that the module could use (forever). Then there would be no login. Since the the token is handed out at tenant init, Okapi and mod-authtoken would essentially know that the request (due to the token) originates from a backend module. So it is more trustworthy that something coming from the "outside".

It would also makes the code more elegant for each module.

Comment by Hongwei Ji [ 08/Mar/21 ]

Adam Dickmeiss , to be on the same page, so it is not mod-search's fault at this moment. We are going to fix Okapi, correct?

Comment by Marc Johnson [ 08/Mar/21 ]

Adam Dickmeiss

The better approach would be that tenant init provided a token that the module could use (forever). Then there would be no login. Since the the token is handed out at tenant init, Okapi and mod-authtoken would essentially know that the request (due to the token) originates from a backend module. So it is more trustworthy that something coming from the "outside".

A long lived access token is a reasonable alternative to consider. My understanding is that expiring tokens are part of the roadmap for the platform in 2021. These two ideas might not be aligned with each other.

Comment by Adam Dickmeiss [ 08/Mar/21 ]

Hongwei Ji. I think it's better to fix OKAPI. I don't think mod-search is doing anything wrong WRT this. So I'd suggest no revert in mod-search

Comment by Ian Hardy [ 08/Mar/21 ]

Thanks Adam and Hongwei, I removed mod-search/ui-inventory-elasticsearch from the build for now so we could get a build. Happy to work w/mod-search devs to test enabling this module again when ready.

Comment by Magda Zacharska [ 08/Mar/21 ]

Ian Hardy - from the comments above it seems the issue wasn't mod-search issue. Could you please re-enable both module in the reference environments?

Comment by Oleksii Petrenko [ 09/Mar/21 ]

Please verify and close

Comment by Marc Johnson [ 09/Mar/21 ]

Oleksii Petrenko

Why has this been assigned to me, because I was the reporter?

I do not think I am appropriately placed to verify that this is resolved. Adam Dickmeiss made the changes necessary. Ian Hardy rebuilt the environment yesterday and confirmed the resolution of this. I am not able to offer anything more than he did.

I suggest this be owned by the Falcon team to check that mod-search is present and working as expected.

Comment by Hongwei Ji [ 09/Mar/21 ]

Technically this is not a mod-search bug. I just verified testing/snapshot env and can see the system user mod-search was created and the perm
inventory-storage.inventory-view.instances.collection.get was assigned to the system user as well. So let's mark this ticket as closed and be done with it.

Comment by Hongwei Ji [ 09/Mar/21 ]

Not a mod-search bug.

Generated at Thu Feb 08 23:25:21 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.