<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:16:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[FOLIO-1812] SPIKE: verify and analyze checkout performance</title>
                <link>https://folio-org.atlassian.net/browse/FOLIO-1812</link>
                <project id="10290" key="FOLIO">FOLIO</project>
                    <description>&lt;p&gt;EBSCO did checkin/checkout workflow performance testing on q3 and q4 environment. One of the slowest API call was checkout by barcode &lt;tt&gt;/circulation/check-out-by-barcode&lt;/tt&gt;. See attached JMeter result file for 5 concurrent users on a small tenant for 5 minutes:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;64146_thumb&quot; href=&quot;/rest/api/3/attachment/content/64146&quot; title=&quot;ebsco_checkinout_perf_result.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;64146&quot; file-preview-title=&quot;ebsco_checkinout_perf_result.png&quot;&gt;&lt;jira-attachment-thumbnail url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64146?default=false&quot; jira-url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64146&quot; filename=&quot;ebsco_checkinout_perf_result.png&quot;&gt;&lt;img src=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64146&quot; data-attachment-name=&quot;ebsco_checkinout_perf_result.png&quot; data-attachment-type=&quot;thumbnail&quot; data-media-services-id=&quot;42d14c42-1847-4410-b734-b64a1a41e4a5&quot; data-media-services-type=&quot;file&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/jira-attachment-thumbnail&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;We should verify the result using isolated test on nightly perf environment to see if the issue still exists and what are the possible causes.&lt;/p&gt;

&lt;p&gt;Acceptance criteria: if performance issue is confirmed, create new JIRA ticket to address it.&lt;/p&gt;</description>
                <environment></environment>
        <key id="81093">FOLIO-1812</key>
            <summary>SPIKE: verify and analyze checkout performance</summary>
                <type id="10003" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium">Task</type>
                                            <priority id="10002" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p3.svg">P3</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c">Hongwei Ji</assignee>
                                                                <reporter accountid="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c">Hongwei Ji</reporter>
                                    <labels>
                            <label>platform-backlog</label>
                            <label>q1-performance</label>
                    </labels>
                <created>Mon, 18 Feb 2019 11:47:11 +0000</created>
                <updated>Wed, 3 Jun 2020 16:39:31 +0000</updated>
                            <resolved>Wed, 20 Feb 2019 21:46:27 +0000</resolved>
                                                                        <due></due>
                            <votes>0</votes>
                                    <watches>2</watches>
                                                                <comments>
                                                            <comment id="195587" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 22:26:39 +0000"  >&lt;p&gt;To verify the result, I used the same JMeter checkin/out script but disabled all API calls in the workflow except the checkout-by-barcode one. The test was done on a standard perf env with the code from folio stable site on 2018-02-15. The test sends 60 requests in 60 seconds ramp-up window. Each request looks like  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;/rest/api/3/attachment/content/64140&quot; title=&quot;checkout.txt attached to FOLIO-1812&quot; data-attachment-type=&quot;file&quot; data-attachment-name=&quot;checkout.txt&quot; data-media-services-type=&quot;file&quot; data-media-services-id=&quot;286ece07-e839-45f3-9517-022dea5f887b&quot; rel=&quot;noreferrer&quot;&gt;checkout.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; and the barcode is selected from a pool and is different for each request. The JMeter log is in  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;/rest/api/3/attachment/content/64142&quot; title=&quot;checkout_60_60.log attached to FOLIO-1812&quot; data-attachment-type=&quot;file&quot; data-attachment-name=&quot;checkout_60_60.log&quot; data-media-services-type=&quot;file&quot; data-media-services-id=&quot;6effe4ff-5226-4fef-888e-c160dd9a553c&quot; rel=&quot;noreferrer&quot;&gt;checkout_60_60.log&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; and the result is in  &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;64144_thumb&quot; href=&quot;/rest/api/3/attachment/content/64144&quot; title=&quot;checkout_60_60.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;64144&quot; file-preview-title=&quot;checkout_60_60.png&quot;&gt;&lt;jira-attachment-thumbnail url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64144?default=false&quot; jira-url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64144&quot; filename=&quot;checkout_60_60.png&quot;&gt;&lt;img src=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64144&quot; data-attachment-name=&quot;checkout_60_60.png&quot; data-attachment-type=&quot;thumbnail&quot; data-media-services-id=&quot;d1f0736b-2968-42e1-96f1-906b9e4e62c6&quot; data-media-services-type=&quot;file&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/jira-attachment-thumbnail&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;In summary: checkout-bar-code has similar performance pattern to checkin-by-barcode, even though a single call on an idle system is not slow (under 2 seconds). When there are multiple concurrent checkout API calls (60 requests in 60s ramp-up window), each individual API call suffer some latency but better than checkin (average is 20 seconds, between min 3s to max 36s). Also the throughput is still good (49 requests per minute). Behind the scene, checkout-by-code API also makes multiple other API calls, see attached log files for  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;/rest/api/3/attachment/content/64148&quot; title=&quot;mod-circulation.log attached to FOLIO-1812&quot; data-attachment-type=&quot;file&quot; data-attachment-name=&quot;mod-circulation.log&quot; data-media-services-type=&quot;file&quot; data-media-services-id=&quot;c947f287-6d2e-47cc-84bb-266dc5867a8f&quot; rel=&quot;noreferrer&quot;&gt;mod-circulation.log&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;,  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;/rest/api/3/attachment/content/64147&quot; title=&quot;mod-circulation-storage.log attached to FOLIO-1812&quot; data-attachment-type=&quot;file&quot; data-attachment-name=&quot;mod-circulation-storage.log&quot; data-media-services-type=&quot;file&quot; data-media-services-id=&quot;ea0fb417-5d46-4ffd-82b6-abb87ff2b746&quot; rel=&quot;noreferrer&quot;&gt;mod-circulation-storage.log&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;, and  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;/rest/api/3/attachment/content/64149&quot; title=&quot;mod-inventory-storage.log attached to FOLIO-1812&quot; data-attachment-type=&quot;file&quot; data-attachment-name=&quot;mod-inventory-storage.log&quot; data-media-services-type=&quot;file&quot; data-media-services-id=&quot;4b55bcc3-2a15-4d73-ab6a-07eb3f4d3cbb&quot; rel=&quot;noreferrer&quot;&gt;mod-inventory-storage.log&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;. Furthermore, I noticed that mod-circulation module will stop working if sending too many requests too fast. For example, initially I was using the same 100 requests in 60 seconds (as in checkin test case), mod-circulation stopped working and could not recover by itself after waiting long time. I had to restart it. It is possible that some resources (socket for example) were not released quickly/properly in in mod-circulation. &lt;/p&gt;</comment>
                                                            <comment id="195593" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Wed, 20 Feb 2019 21:46:27 +0000"  >&lt;p&gt;Created 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;FOLIO-1815&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/FOLIO-1815&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;SPIKE: &amp;quot;profile&amp;quot; checkin/out-by-barcode API response time&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium&quot; /&gt;
            FOLIO-1815
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 to address checkout-by-barcode performance issue.&lt;br/&gt;
Created 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;CIRC-225&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/CIRC-225&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;mod-circulation hangs when checkout-by-barcode requests come in too fast&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10303?size=medium&quot; /&gt;
            CIRC-225
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 to address the mod-circulation hangs.&lt;/p&gt;</comment>
                                                            <comment id="195596" author="5d8a51e1bcc5480dd6ddfec7" created="Wed, 27 Feb 2019 14:55:02 +0000"  >&lt;p&gt;In my tests last night I saw the following error in 2/10 calls to circulation/loans.  This may be related.&lt;/p&gt;

&lt;p&gt;Failed to fetch loans from storage (500:ErrorMessage(fields=Map(Line -&amp;gt; 784, File -&amp;gt; postinit.c, SQLSTATE -&amp;gt; 53300, Routine -&amp;gt; InitPostgres, V -&amp;gt; FATAL, Message -&amp;gt; remaining connection slots are reserved for non-replication superuser connections, Severity -&amp;gt; FATAL)))&lt;/p&gt;</comment>
                                                            <comment id="195599" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Tue, 5 Mar 2019 16:20:11 +0000"  >&lt;p&gt;Update: above db error is not related. As discovered in 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;CIRC-225&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/CIRC-225&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;mod-circulation hangs when checkout-by-barcode requests come in too fast&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10303?size=medium&quot; /&gt;
            CIRC-225
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
, it is due to mod-circulation makes http connection to itself through Okapi, so the default http connection pool of 5 is exhausted. To make reproducing the problem easier, I also added a separate &lt;a href=&quot;https://github.com/folio-org/folio-perf-test/blob/master/Folio-Test-Plans/core-platform/FOLIO-1812&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;JMeter test&lt;/a&gt; for it.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10003">
                    <name>Relates</name>
                                            <outwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="47642">CIRC-225</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="81100">FOLIO-1815</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="81090">FOLIO-1811</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="64140" name="checkout.txt" size="241" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 11:46:10 +0000"/>
                            <attachment id="64142" name="checkout_60_60.log" size="783" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 11:46:10 +0000"/>
                            <attachment id="64144" name="checkout_60_60.png" size="51671" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 11:46:10 +0000"/>
                            <attachment id="64146" name="ebsco_checkinout_perf_result.png" size="247965" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 21:39:32 +0000"/>
                            <attachment id="64147" name="mod-circulation-storage.log" size="2636" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 11:46:10 +0000"/>
                            <attachment id="64148" name="mod-circulation.log" size="13269" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 11:46:10 +0000"/>
                            <attachment id="64149" name="mod-inventory-storage.log" size="3960" author="557058:4f6bed01-40a6-48d5-8471-7ef21f5ea97c" created="Mon, 18 Feb 2019 11:46:10 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10144"><![CDATA[Core: Platform]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|hzzfuf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1661">Core: Platform - Sprint 57</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10044" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 27 Feb 2019 14:55:02 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>