<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Fri Feb 09 00:30:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[UXPROD-3193] NFR: R3 2021 Kiwi Data import Stability/Reliability work </title>
                <link>https://folio-org.atlassian.net/browse/UXPROD-3193</link>
                <project id="10000" key="UXPROD">UX Product</project>
                    <description>&lt;p&gt;&lt;font color=&quot;#de350b&quot;&gt;&lt;b&gt;Team estimation - 90 days&lt;/b&gt;&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;&lt;font color=&quot;#ff0000&quot;&gt;
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3135&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3135&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data Import Stabilization and Reliability work &quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3135
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 was split into 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3193&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3193&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data import Stability/Reliability work &quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3193
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 for stability and reliability and 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3191&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3191&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data import performance work&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3191
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 for performance; &lt;/font&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5af5e627525ba96b58654f12&quot; class=&quot;user-hover&quot; rel=&quot;5af5e627525ba96b58654f12&quot; data-account-id=&quot;5af5e627525ba96b58654f12&quot; accountid=&quot;5af5e627525ba96b58654f12&quot; rel=&quot;noreferrer&quot;&gt;Ann-Marie Breaux&lt;/a&gt;&lt;font color=&quot;#ff0000&quot;&gt; to close 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-3135&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-3135&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;NFR: R3 2021 Kiwi Data Import Stabilization and Reliability work &quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-3135
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 once all issues moved from it to the new features&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Current situation or problem&lt;/b&gt;:&lt;br/&gt;
1.High CPU/Memory consumption on modules&lt;/p&gt;

&lt;p&gt;2.Duplicates created upon import&lt;br/&gt;
3.&#160;SRS can fail when processing &lt;a href=&quot;https://folio-org.atlassian.net/wiki/display/~mage.air/1.+Create?focusedCommentId=65115740#comment-65115740&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;message &lt;/a&gt;during import&lt;/p&gt;

&lt;p&gt;4.&#160;If we have infrastructure issue (like DB not available, module being restarted or network failure), we are sending DI_ERROR instead of retrying&lt;br/&gt;
5.&#160;De-duplication of status messages for progress bar&lt;/p&gt;

&lt;p&gt;Investigation required for:&lt;/p&gt;

&lt;p&gt;&lt;font color=&quot;#00875a&quot;&gt;&lt;b&gt;&lt;em&gt;6.&#160;Race condition on start (Kafka consumers start working before DB is configured) OR Periodical DB shutdown after SRS restart. Jobs get stuck if not able to update status in DB (messages ACKed even if we could not process them)&lt;/em&gt;&lt;/b&gt;&lt;/font&gt;&lt;br/&gt;
&lt;font color=&quot;#00875a&quot;&gt;&lt;b&gt;&lt;em&gt;7.Kafka consumers stop reading messages eventually, breaking job progress until module restart.&lt;/em&gt;&lt;/b&gt;&lt;/font&gt;&lt;br/&gt;
&lt;font color=&quot;#00875a&quot;&gt;&lt;b&gt;&lt;em&gt;8.mod-data-import stores input file in memory, limiting size of uploaded file and possibly having oom&lt;/em&gt;&lt;/b&gt;&lt;/font&gt;&lt;br/&gt;
&lt;font color=&quot;#00875a&quot;&gt;&lt;b&gt;&lt;em&gt;9.Consumer gets disconnected from Kafka cluster&lt;/em&gt;&lt;/b&gt;&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;In scope&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Out of scope&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Use case(s)&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Proposed solution/stories&lt;/b&gt;&lt;br/&gt;
&lt;em&gt;1*.*Significantly decrease size of payload:&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;&lt;em&gt;Remove immutable parts. Instead fetch them on demand and cache locally for reuse.&lt;/em&gt;&lt;/li&gt;
	&lt;li&gt;&lt;em&gt;Change message handling mechanism (currently relies on pt1 - profile) (optional)&lt;/em&gt;&lt;/li&gt;
	&lt;li&gt;&lt;em&gt;Move archiving to Kafka instead of module level&lt;/em&gt;&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;em&gt;2.Make consumers behave idempotent. Add pass-through identifier to de-duplicate messages.&lt;/em&gt;&#160;&lt;br/&gt;
&lt;em&gt;3.Generate &quot;INSTANCE CREATED&quot; from mod-inventory. Consume in SRS to update HRID in BIB and in INVENTORY to continue processing.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;4.Do not ACK messages in Kafka if there&apos;s not a logic, but infrastructure error/exception. Split failed processing results into 2 categories:&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;&lt;em&gt;IO errors - do not ack. retry until fixed&lt;/em&gt;&lt;/li&gt;
	&lt;li&gt;&lt;em&gt;Business logic - DI_ERROR and Ack current message&lt;/em&gt;&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;em&gt;Remove unnecessary topics (* ready for post processing and hrid set)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;5.De-duplicate status messages per-record while tracking progress&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;font color=&quot;#de350b&quot;&gt;&lt;b&gt;Problems 6,7,8 and 9 require investigation&lt;/b&gt;&lt;/font&gt;&lt;br/&gt;
&lt;font color=&quot;#de350b&quot;&gt;Possible solution for problem 8 -&#160; Split to chunks, put to database, work with database/temp storage. Partially done&#160;(to be investigated)&lt;/font&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Links to additional info:&lt;/b&gt;&lt;br/&gt;
&lt;a href=&quot;https://wiki.folio.org/pages/resumedraft.action?draftId=65115261&amp;amp;draftShareId=5061165d-96d0-4c00-a9f9-3a18b80833b2&amp;amp;&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;&lt;b&gt;Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki&lt;/b&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Questions&lt;/b&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="12140">UXPROD-3193</key>
            <summary>NFR: R3 2021 Kiwi Data import Stability/Reliability work </summary>
                <type id="10002" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium">New Feature</type>
                            <parent id="13571">UXPROD-47</parent>
                                    <priority id="10001" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p2.svg">P2</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="5af5e627525ba96b58654f12">Ann-Marie Breaux</assignee>
                                                                <reporter accountid="60a5273475e875006f2e05a0">Taisiya Trunova</reporter>
                                    <labels>
                            <label>data-import</label>
                            <label>epam-folijet</label>
                            <label>split</label>
                    </labels>
                <created>Thu, 22 Jul 2021 13:20:57 +0000</created>
                <updated>Tue, 4 Jan 2022 14:11:57 +0000</updated>
                            <resolved>Fri, 5 Nov 2021 15:14:45 +0000</resolved>
                                                    <fixVersion>Kiwi (R3 2021)</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>2</watches>
                                                                    <issuelinks>
                            <issuelinktype id="10010">
                    <name>Continues</name>
                                                                <inwardlinks description="is continued by">
                                        <issuelink>
            <issuekey id="12087">UXPROD-3210</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10008">
                    <name>Defines</name>
                                            <outwardlinks description="defines">
                                        <issuelink>
            <issuekey id="12076">UXPROD-3135</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="13571">UXPROD-47</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is defined by ">
                                        <issuelink>
            <issuekey id="62528">MODDATAIMP-440</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62568">MODDATAIMP-474</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62688">MODDATAIMP-558</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="39458">KAFKAWRAP-10</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62554">MODDATAIMP-390</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62562">MODDATAIMP-465</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62679">MODDATAIMP-544</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62683">MODDATAIMP-548</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="32842">MODDICORE-198</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60402">MODINV-417</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60556">MODINV-493</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60557">MODINV-494</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60475">MODINV-553</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="66667">MODINVOICE-314</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="56341">MODINVSTOR-794</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63184">MODSOURCE-339</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63248">MODSOURCE-390</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57951">MODSOURMAN-522</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58076">MODSOURMAN-579</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="39459">KAFKAWRAP-11</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="62608">MODDATAIMP-430</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="68428">MODDICONV-200</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="60396">MODINV-405</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="66567">MODINVOICE-251</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63207">MODSOURCE-286</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57998">MODSOURMAN-463</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57999">MODSOURMAN-464</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58000">MODSOURMAN-465</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58001">MODSOURMAN-466</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58006">MODSOURMAN-481</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57949">MODSOURMAN-521</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58122">MODSOURMAN-575</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Back End Estimate</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10126"><![CDATA[Jumbo: > 45 days]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10153"><![CDATA[Folijet]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10014" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue key="$xmlutils.escape($text)">Batch Importer (Bib/Acq)</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Front-End Confidence factor</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10107"><![CDATA[Low]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10063" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>PO Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>117.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i01ra5:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10069" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Rank: Cornell (Full Sum 2021)</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10213"><![CDATA[R1]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10091" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Rank: U of AL (MVP Oct 2020)</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10324"><![CDATA[R1]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>