Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Introduce filter request-log to avoid buffering HTTP content in memory

Description

Observation: Okapi uses a lot of memory when uploading...

wrote on Slack:

Hello all! I have the question about okapi and his redirecting mechanism.
As I understand we call okapi from UI and it's looking module and redirect request with the body to him. But when we upload files okapi try to read buffer and write new request's body for a particular module. And here we get OutOfMemmory. Also, mod-audit-filter in prePostFilter trying to read ANY post request and create JsonObject from request's body without checking ContentType header. I created a bug for mod-audit-filter https://folio-org.atlassian.net/browse/MODAFILT-5#icft=MODAFILT-5. But what I should do with Okapi? @dickmeiss @julianladisch

curl -w '\n' -X POST -D - \ -H "Content-type: application/octet-stream" \ -d @marc.mrc \ http://localhost:9130/data-import/upload/file?uploadDefinitionId=67dfac11-1caf-4470-9ad1-d533f6360bdd&fileId=88dfac11-1caf-4470-9ad1-d533f6360bdd

As I understand Okapi handle this request and redirect to mod-data-import which provide this interface but -d marc.mrc this file can be 1GB And okapi die with OutOfMemmory.

Environment

None

Potential Workaround

None

Checklist

hide

TestRail: Results

Activity

Show:

Adam Dickmeiss January 9, 2019 at 1:14 PM

Problem occurs because modules for the user are using the type=request-only filter. That reads the whole incoming request before passing the full request to the handler.. It is buffered, because it has to be posted to filters folloing it .. In particular, the handler (which is of type request-response).

Solution: make a new filter type request-log which does not buffer the incoming request into memory but posts to multiple filters at the same time.. The response (HTTP error code), will have no impact on the pipeline.. so response is essentially ignored.. (But Okapi could log it).. This if fine for logging purposes of sending data to remote systems etc.. The filter is otherwise identical to request-only ..

Done

Details

Assignee

Reporter

Labels

Priority

Development Team

Core: Platform

Fix versions

TestRail: Cases

Open TestRail: Cases

TestRail: Runs

Open TestRail: Runs

Created January 4, 2019 at 12:26 PM
Updated February 7, 2019 at 2:59 PM
Resolved January 18, 2019 at 12:14 PM
TestRail: Cases
TestRail: Runs

Flag notifications