<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Fri Feb 09 00:30:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[UXPROD-3191] NFR: R3 2021 Kiwi Data import performance work</title>
                <link>https://folio-org.atlassian.net/browse/UXPROD-3191</link>
                <project id="10000" key="UXPROD">UX Product</project>
                    <description>&lt;p&gt;&lt;font color=&quot;#de350b&quot;&gt;&lt;b&gt;Team estimation - 45 days&lt;/b&gt;&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;&lt;font color=&quot;#FF0000&quot;&gt;
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3135&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3135&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data Import Stabilization and Reliability work &quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3135
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 was split into 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3193&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3193&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data import Stability/Reliability work &quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3193
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 for stability and reliability and 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3191&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3191&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data import performance work&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3191
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 for performance; &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5af5e627525ba96b58654f12&quot; class=&quot;user-hover&quot; rel=&quot;5af5e627525ba96b58654f12&quot; data-account-id=&quot;5af5e627525ba96b58654f12&quot; accountid=&quot;5af5e627525ba96b58654f12&quot; rel=&quot;noreferrer&quot;&gt;Ann-Marie Breaux&lt;/a&gt; to close 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3135&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3135&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data Import Stabilization and Reliability work &quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3135
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 once all issues moved from it to the new features&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Current situation or problem&lt;/b&gt;:&lt;/p&gt;

&lt;p&gt;1.Kafka producer closed after sending&lt;/p&gt;

&lt;p&gt;2.WARN message when no handler found&lt;/p&gt;

&lt;p&gt;3.&#160;Kafka cache resource consumption&lt;br/&gt;
 4. Data import impacts other processes&lt;/p&gt;

&lt;p&gt;5.&#160;High resource consumption to get job(s) status/progress&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;In scope&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Out of scope&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Use case(s)&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Proposed solution/stories&lt;/b&gt;&lt;br/&gt;
 *1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.&lt;br/&gt;
 2. Do not subscribe to messages you&apos;re not going to process OR&#160;&lt;b&gt;Lower log lever for this type of messages.&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;*3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) &lt;font color=&quot;#ff0000&quot;&gt;(blocked by 6 PUT LINK TO FEATURE in p.6)&lt;br/&gt;
 &lt;b&gt;4. SPIKE REQ&lt;/b&gt;.:&#160;&lt;b&gt;Need investigation (possible solution - configure rate limiter). Relates to&lt;/b&gt;&#160;&lt;em&gt;High CPU/Memory consumption on modules&lt;/em&gt;&lt;br/&gt;
 &lt;font color=&quot;#172b4d&quot;&gt;&lt;/font&gt;5.&#160;Add some kind of caching for progress tracking (database or in-memory)&lt;/font&gt;&lt;font color=&quot;&quot;&gt;&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;Links to additional information:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;&lt;a href=&quot;https://wiki.folio.org/pages/resumedraft.action?draftId=65115261&amp;amp;draftShareId=5061165d-96d0-4c00-a9f9-3a18b80833b2&amp;amp;&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki&lt;/a&gt;&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Questions&lt;/b&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="12139">UXPROD-3191</key>
            <summary>NFR: R3 2021 Kiwi Data import performance work</summary>
                <type id="10002" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium">New Feature</type>
                            <parent id="13571">UXPROD-47</parent>
                                    <priority id="10001" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p2.svg">P2</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="5af5e627525ba96b58654f12">Ann-Marie Breaux</assignee>
                                                                <reporter accountid="60a5273475e875006f2e05a0">Taisiya Trunova</reporter>
                                    <labels>
                            <label>NFR</label>
                            <label>data-import</label>
                            <label>epam-folijet</label>
                            <label>performance</label>
                    </labels>
                <created>Thu, 22 Jul 2021 13:04:38 +0000</created>
                <updated>Mon, 15 Nov 2021 14:21:17 +0000</updated>
                            <resolved>Fri, 5 Nov 2021 15:15:10 +0000</resolved>
                                                    <fixVersion>Kiwi (R3 2021)</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>3</watches>
                                                                <comments>
                                                            <comment id="17399" author="5af5e627525ba96b58654f12" created="Wed, 25 Aug 2021 13:05:18 +0000"  >&lt;p&gt;Grooming today - have most of the spikes, but likely will have more stories/tasks once the spikes are completed&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Continues</name>
                                                                <inwardlinks description="is continued by">
                                        <issuelink>
            <issuekey id="12192">UXPROD-3261</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10008">
                    <name>Defines</name>
                                            <outwardlinks description="defines">
                                        <issuelink>
            <issuekey id="13571">UXPROD-47</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is defined by ">
                                        <issuelink>
            <issuekey id="62636">MODDATAIMP-499</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60410">MODINV-427</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="56374">MODINVSTOR-792</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63218">MODSOURCE-307</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63249">MODSOURCE-388</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58120">MODSOURMAN-537</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58055">MODSOURMAN-548</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58056">MODSOURMAN-550</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60493">MODINV-508</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60564">MODINV-572</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63185">MODSOURCE-340</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10003">
                    <name>Relates</name>
                                            <outwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="12045">UXPROD-3023</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="12076">UXPROD-3135</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Back End Estimate</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10125"><![CDATA[XXXL: 30-45 days]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10153"><![CDATA[Folijet]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10014" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue key="$xmlutils.escape($text)">Batch Importer (Bib/Acq)</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Front-End Confidence factor</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10106"><![CDATA[Medium]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10063" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>PO Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>116.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i01ra7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 25 Aug 2021 13:05:18 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>