<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:21:34 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[FOLIO-2563] SPIKE: propose prevention of DoS via CQL query</title>
                <link>https://folio-org.atlassian.net/browse/FOLIO-2563</link>
                <project id="10290" key="FOLIO">FOLIO</project>
                    <description>&lt;p&gt;&lt;b&gt;Description&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Several application endpoints accessible by normal users perform CQL (Common Query Language) queries leveraging CQL-Java. CQL is a formal language for representing queries to information retrieval systems such as web indexes, bibliographic catalogs and museum collection information.1 The application can be forced to perform an OR expression over objects that match a certain patterns. As a result, the CQL engine is forced to do very expensive queries to search and parse the result of the operations. Although it appears that the full result of the operation is filtered downstream when returned to the user, It was noticed that each request to the target endpoint would take around 29 seconds to respond back. An attacker can exploit this by performing a denial of service attack by requesting several GET requests simultaneously or in short intervals, resulting in the exhaustion of the server&#8217;s memory resources.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Steps to reproduce&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Perform the following request with the thread count set to 20 and throttle time set to 1 milliseconds. Substitute the value of X-Okapi-Token header with a token belonging to a user that has inventory.instances.collection.get permission.&lt;/p&gt;

&lt;p&gt;GET /inventory/instances?limit=1000&amp;amp;query=%28keyword%20all%20%22ncc%22%29%20or%201=1&lt;br/&gt;
urldecoded query: &lt;tt&gt;(keyword all &quot;ncc&quot;) or 1=1&lt;/tt&gt;&lt;br/&gt;
Users can use the Inventory front-end &quot;Query search&quot; slot to enter any CQL: &lt;a href=&quot;https://folio-snapshot.aws.indexdata.com/inventory?qindex=querySearch&amp;amp;sort=Title&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://folio-snapshot.aws.indexdata.com/inventory?qindex=querySearch&amp;amp;sort=Title&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Observe that all inventory related endpoints will become unresponsive&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Acceptance criteria&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Propose an approach that will prevent heavy queries to cause a module to become unresponsive. Provide a PoC implementation for review.&lt;/p&gt;

&lt;p&gt;Potential solutions:&lt;/p&gt;

&lt;p&gt;Query validation based:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;RMB: validate query length&lt;/li&gt;
	&lt;li&gt;RMB: validate query structure (max number of OR expression)&lt;/li&gt;
	&lt;li&gt;RMB: validate if query field and relation is backed up by an index (in schema.json, reject if not)&lt;/li&gt;
	&lt;li&gt;RMB: validate known problematic searches (e.g cql.allRecords)&lt;/li&gt;
	&lt;li&gt;RMB: wide truncation queries&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Query execution based:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Can Postgres analyze query runtime?&lt;/li&gt;
	&lt;li&gt;Can Postgres limit the runtime for a SELECT&lt;/li&gt;
	&lt;li&gt;If not, can we timeout a connection for long running queries&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="81661">FOLIO-2563</key>
            <summary>SPIKE: propose prevention of DoS via CQL query</summary>
                <type id="10003" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium">Task</type>
                                            <priority id="10001" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p2.svg">P2</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="70121:84bca0b8-4380-4d87-8a90-53e9133584f6">Mikhail Fokanov</assignee>
                                                                <reporter accountid="557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d">Jakub Skoczen</reporter>
                                    <labels>
                            <label>platform-backlog</label>
                            <label>security</label>
                    </labels>
                <created>Tue, 14 Apr 2020 13:00:20 +0000</created>
                <updated>Tue, 13 Jul 2021 08:49:47 +0000</updated>
                            <resolved>Thu, 4 Jun 2020 13:14:23 +0000</resolved>
                                                                        <due></due>
                            <votes>0</votes>
                                    <watches>5</watches>
                                                                <comments>
                                                            <comment id="198231" author="5cf6c546b87c300f36eb7b9a" created="Thu, 16 Apr 2020 13:56:55 +0000"  >&lt;p&gt;Which ever approach we take, we need to be very careful not to break existing queries throughout the system.  For instance, if you start limiting query length or the number of OR&apos;d terms, queries that ask for a bunch of specific records via UUID might break&lt;/p&gt;</comment>
                                                            <comment id="198233" author="5ee89462f7aa140abd82d11d" created="Thu, 16 Apr 2020 15:33:30 +0000"  >&lt;p&gt;We may use pg_cancel_backend to cancel a long-running query. (Don&apos;t use pg_terminate_backend as it may result in an inconsistent state.)&lt;br/&gt;
&lt;a href=&quot;https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SIGNAL&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SIGNAL&lt;/a&gt;&lt;/p&gt;</comment>
                                                            <comment id="198235" author="70121:84bca0b8-4380-4d87-8a90-53e9133584f6" created="Thu, 7 May 2020 12:54:13 +0000"  >&lt;p&gt;queryTimeout parameter can be specified (&lt;a href=&quot;https://vertx.io/docs/vertx-mysql-postgresql-client/java/&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://vertx.io/docs/vertx-mysql-postgresql-client/java/&lt;/a&gt;) for postgres connection in getPostgreSQLClientConfig. In such case we can have 2 pools of connections: without execution time restriction and without it.&lt;br/&gt;
If the client breaks the connection PostgreSQL server will set the interrupt flag. Next time the query checks for interrupts as it executes it will see the flag and abort. Sometimes a query might be doing CPU-heavy work within code that doesn&apos;t check for interrupts.&lt;/p&gt;</comment>
                                                            <comment id="198236" author="63e2a2771b13d42998e4e706" created="Thu, 7 May 2020 13:13:26 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=70121%3A84bca0b8-4380-4d87-8a90-53e9133584f6&quot; class=&quot;user-hover&quot; rel=&quot;70121:84bca0b8-4380-4d87-8a90-53e9133584f6&quot; data-account-id=&quot;70121:84bca0b8-4380-4d87-8a90-53e9133584f6&quot; accountid=&quot;70121:84bca0b8-4380-4d87-8a90-53e9133584f6&quot; rel=&quot;noreferrer&quot;&gt;Mikhail Fokanov&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;In such case we can have 2 pools of connections: without execution time restriction and without it.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;What would the two different pools be used for? I&apos;m guessing the time-limited pool would be used for queries. What would the non-time-limited pool be used for?&lt;/p&gt;</comment>
                                                            <comment id="198238" author="70121:84bca0b8-4380-4d87-8a90-53e9133584f6" created="Thu, 7 May 2020 13:21:59 +0000"  >&lt;p&gt;There could be still some requests from another modules, that should take more than 5 sec by design. For them it could be used. I&apos;m not aware is it the case for now.&lt;/p&gt;</comment>
                                                            <comment id="198240" author="5ee89462f7aa140abd82d11d" created="Thu, 7 May 2020 14:00:10 +0000"  >&lt;p&gt;The vertx-mysql-postgresql-client has the &lt;tt&gt;queryTimeout&lt;/tt&gt; connection parameter: &lt;a href=&quot;https://vertx.io/docs/vertx-mysql-postgresql-client/java/#_configuration&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://vertx.io/docs/vertx-mysql-postgresql-client/java/#_configuration&lt;/a&gt;&lt;br/&gt;

    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;RMB-246&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/RMB-246&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;migrate to reactive postgres client (vertx-pg-client)&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            RMB-246
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 switches to the vertx-pg-client: &lt;a href=&quot;https://vertx.io/docs/vertx-pg-client/java/#_configuration&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://vertx.io/docs/vertx-pg-client/java/#_configuration&lt;/a&gt;&lt;br/&gt;
Has vertx-pg-client a query timeout parameter?&lt;/p&gt;

&lt;p&gt;A vertx-pg-client connection can cancel the query using the cancelRequest method ( &lt;a href=&quot;https://vertx.io/docs/vertx-pg-client/java/#_cancelling_request&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://vertx.io/docs/vertx-pg-client/java/#_cancelling_request&lt;/a&gt; ):&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
connection
  .query(&lt;span class=&quot;code-quote&quot;&gt;&quot;SELECT pg_sleep(20)&quot;&lt;/span&gt;)
  .execute(ar -&amp;gt; { ... });
connection
  .cancelRequest(ar -&amp;gt; { ... });
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                                                            <comment id="198242" author="70121:84bca0b8-4380-4d87-8a90-53e9133584f6" created="Mon, 11 May 2020 11:50:40 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5ee89462f7aa140abd82d11d&quot; class=&quot;user-hover&quot; rel=&quot;5ee89462f7aa140abd82d11d&quot; data-account-id=&quot;5ee89462f7aa140abd82d11d&quot; accountid=&quot;5ee89462f7aa140abd82d11d&quot; rel=&quot;noreferrer&quot;&gt;Julian Ladisch&lt;/a&gt;, I see. In such case, would it be efficient to implement thread in PostgresClient, which will once per 5 seconds verify whether there are requests that live for more than 5 seconds and kill them? In such case any request will be terminated within range of &lt;span class=&quot;error&quot;&gt;&amp;#91;5,10&amp;#93;&lt;/span&gt; seconds. Long running requests (by design) would be added to that list.&lt;/p&gt;

&lt;p&gt;There also could be separate url prefix for internal REST calls, which is not visible outside of okapi. Requests for such urls can have long-running queries. Also commands, that use not standard RMB CQL query processing would be not added to the list.&lt;/p&gt;</comment>
                                                            <comment id="198244" author="5ee89462f7aa140abd82d11d" created="Mon, 11 May 2020 12:54:00 +0000"  >&lt;p&gt;&lt;a href=&quot;https://github.com/vert-x3/vertx-mysql-postgresql-client&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/vert-x3/vertx-mysql-postgresql-client&lt;/a&gt; says: &quot;This client is deprecated - use instead &lt;a href=&quot;https://github.com/eclipse-vertx/vertx-sql-client&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/eclipse-vertx/vertx-sql-client&lt;/a&gt;&quot;, vertx-sql-client contains &lt;a href=&quot;https://github.com/eclipse-vertx/vertx-sql-client/tree/master/vertx-pg-client&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/eclipse-vertx/vertx-sql-client/tree/master/vertx-pg-client&lt;/a&gt;&lt;/p&gt;</comment>
                                                            <comment id="198246" author="63e2a2771b13d42998e4e706" created="Fri, 3 Jul 2020 15:20:18 +0000"  >&lt;p&gt;This issue is closed, could someone please summarise what decision was made.&lt;/p&gt;</comment>
                                                            <comment id="198248" author="5ee89462f7aa140abd82d11d" created="Tue, 11 Aug 2020 10:08:09 +0000"  >&lt;p&gt;Solution: 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;RMB-615&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/RMB-615&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;cancel queries that take longer than limit&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            RMB-615
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 gives RMB based modules the option to set a time limit, RMB cancels a query that exceeds the time limit.&lt;br/&gt;
For each RMB based module that should cancel long running queries a new issue should be created. Example: &lt;a href=&quot;https://folio-org.atlassian.net/browse/MODINVSTOR-496&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://folio-org.atlassian.net/browse/MODINVSTOR-496&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This CQL based denial of service attack has low severity: It requires a valid staff login.&lt;br/&gt;
We also know that there will always be some legitimate queries that take long. Any member of staff can cause a denial of service by running multiple long running queries, however, this is unlikely. If we get reports that this actually happens we may consider other possibilities, for example implementing a DoS detection that automatically disables the login that causes the DoS. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                            <outwardlinks description="blocks">
                                        <issuelink>
            <issuekey id="79617">FOLIO-2524</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10003">
                    <name>Relates</name>
                                            <outwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="56658">RMB-505</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57209">RMB-615</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="56666">RMB-534</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60358">MODINV-283</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="56068">MODINVSTOR-510</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57215">RMB-677</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57257">RMB-638</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10144"><![CDATA[Core: Platform]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|hzx6me:v1</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1858">CP: sprint 87</customfieldvalue>
    <customfieldvalue id="1108">CP: sprint 90</customfieldvalue>
    <customfieldvalue id="1421">CP: sprint 88</customfieldvalue>
    <customfieldvalue id="1422">CP: sprint 89</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10044" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 16 Apr 2020 13:56:55 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>