<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Fri Feb 09 00:40:39 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[UXPROD-4533] Long term solution for creates/updates of authority records (entire dataset)</title>
                <link>https://folio-org.atlassian.net/browse/UXPROD-4533</link>
                <project id="10000" key="UXPROD">UX Product</project>
                    <description>&lt;p&gt;&lt;b&gt;Problem:&lt;/b&gt; Tremendous difficulty to a.) migrate authority records who are moving to FOLIO from another system, OR b.) update all authority records to support FOLIO authority control.&#160; With the Nolana release, a short term solution was applied &lt;a href=&quot;https://folio-org.atlassian.net/wiki/display/DD/ARCH-36+Provide+a+way+to+update+MARC+authority+records+when+mapping+rules+have+changed&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://folio-org.atlassian.net/wiki/display/DD/ARCH-36+Provide+a+way+to+update+MARC+authority+records+when+mapping+rules+have+changed&lt;/a&gt; but has proven unreliable and not scalable. In addition, folks migrating authorities from one system of another that use &lt;b&gt;(POST /authority-storage/authorities)&lt;/b&gt; report the following issue: posting records one-by-one seems very slow. about 40 seconds per 50 records.&lt;/p&gt;

&lt;p&gt;Until this issue is addressed, many libraries will not be able to use the MARC authority app and authority control effectively. Note Library of Congress has over 10 million authority records.&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;This feature covers:&lt;/b&gt;&#160;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Create new authority records (over 500,000+ records)&#160;&lt;/li&gt;
	&lt;li&gt;Update authority records (over 500,000+ records)&#160;&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The proposed solution should&lt;/b&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;consider a solution outside of data import because the problem we are trying to solve is really with migrating a very large set of records (in many cases the entire data set) and data import is not meant to migrate millions of records.&lt;/li&gt;
	&lt;li&gt;be discussed with Folijet or developed in collaboration with this team because ideally it should be the solution (serve as the pattern) for bib/holdings/item records of any format.&#160;&lt;/li&gt;
	&lt;li&gt;consider concurrency level change to minimize performance and/or reliability degradation&lt;/li&gt;
	&lt;li&gt;support the following environment setups
	&lt;ul&gt;
		&lt;li&gt;Self-hosted?&lt;/li&gt;
		&lt;li&gt;Single tenant + single cluster&#160;&#160;&lt;/li&gt;
		&lt;li&gt;Enhanced consortia support&#160;&lt;/li&gt;
		&lt;li&gt;Multi-tenant + single cluster &amp;gt; if one or more tenants is processing a very large dataset or entire data set then other tenants should continue to operate without significant latency or delays. &lt;font color=&quot;#de350b&quot;&gt;&lt;em&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;Since stand-alone solution &amp;gt; shared database. Performance degradation will occur but unsure of impact because it is a database update. No use of folio modules in this operation. Will need to measure via PTF testing.&amp;#93;&lt;/span&gt;&lt;/em&gt;&lt;/font&gt;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;no impact significantly deteriorate a.) Check-in Check out (CICO), b.) data import (for example can a library still run data import jobs for orders? bibs? holdings? items? c.) access to Inventory and MARC authority workflows&#160;&#160;&lt;/li&gt;
	&lt;li&gt;&lt;b&gt;Stats&lt;/b&gt; Action: Create authority records&#160;
	&lt;ul&gt;
		&lt;li&gt;
		&lt;ul&gt;
			&lt;li&gt;
			&lt;ul&gt;
				&lt;li&gt;500,000 records are updated []&lt;/li&gt;
				&lt;li&gt;1 million records are updated []&lt;/li&gt;
				&lt;li&gt;5 million records are updated []&lt;/li&gt;
				&lt;li&gt;10 million records are updated []&lt;/li&gt;
				&lt;li&gt;20 million records can updated []&lt;/li&gt;
			&lt;/ul&gt;
			&lt;/li&gt;
			&lt;li&gt;Action:&#160; Update authority records
			&lt;ul&gt;
				&lt;li&gt;500,000 records are updated []&lt;/li&gt;
				&lt;li&gt;1 million records are updated []&lt;/li&gt;
				&lt;li&gt;5 million records are updated []&lt;/li&gt;
				&lt;li&gt;10 million records are updated []&lt;/li&gt;
				&lt;li&gt;20 million records can updated []&lt;/li&gt;
			&lt;/ul&gt;
			&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Ability to have these migrations executed in the background with minimal impact to FOLIO usage.&#160;&lt;/li&gt;
	&lt;li&gt;Implement
	&lt;ul&gt;
		&lt;li&gt;slicing solution similar to data import implementation 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;UXPROD-4337&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/UXPROD-4337&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Reliably process large files in DI by automatically splitting and processing source files&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            UXPROD-4337
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
&lt;/li&gt;
		&lt;li&gt;Report on progress and status&#160;&lt;/li&gt;
		&lt;li&gt;Users should have a simple solution to get/receive files that contain records with errors (The records that were not processed successfully).&lt;/li&gt;
		&lt;li&gt;Log and report errors &amp;gt; Provide a response that includes details on records that failed to create or update in SRS and Inventory.&lt;/li&gt;
		&lt;li&gt;Possible: Batch POST/PUT Authority API support to support create and update over 500,000+ records&lt;/li&gt;
		&lt;li&gt;Support UTF-8 encoding.&lt;/li&gt;
		&lt;li&gt;Support optimistic locking mechanism.&lt;/li&gt;
		&lt;li&gt;Enforce data validation rules during the creation of MARC authority records to prevent duplicates.&lt;/li&gt;
		&lt;li&gt;Apply the same data validation rules enforced during the updating of MARC authority records.&lt;/li&gt;
		&lt;li&gt;User should be able to load a file and system just processes it until loading is complete. Highly reliable.&#160;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Applies to MARC 21 format&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;b&gt;View migration numbers for several libraries&lt;/b&gt; : &lt;a href=&quot;https://docs.google.com/spreadsheets/d/10GiFrfZee8aY8PcE0JJxf-lWtMkddFWnOYo_tiKYXrs/edit#gid=0&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://docs.google.com/spreadsheets/d/10GiFrfZee8aY8PcE0JJxf-lWtMkddFWnOYo_tiKYXrs/edit#gid=0&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Definition of done&lt;/b&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Load testing by development team&#160;
	&lt;ul&gt;
		&lt;li&gt;Scenario: New customer: Migration from one system to FOLIO&lt;/li&gt;
		&lt;li&gt;Scenario: Existing customer never used MARC authority app before&#160;&#160;&lt;/li&gt;
		&lt;li&gt;Scenario: Existing customer that already uses MARC authority app: Updating entire record set to support FOLIO authority control&#160;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Verify handling of errors&#160;&lt;/li&gt;
	&lt;li&gt;PTF testing
	&lt;ul&gt;
		&lt;li&gt;Scenario: New customer: Migration from one system to FOLIO&lt;/li&gt;
		&lt;li&gt;Scenario: Existing customer never used MARC authority app before&#160;&lt;/li&gt;
		&lt;li&gt;Scenario: Existing customer that already uses MARC authority app: Updating entire record set to support FOLIO authority control&#160;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Production-like testing
	&lt;ul&gt;
		&lt;li&gt;Scenario: New customer: Migration from one system to FOLIO&lt;/li&gt;
		&lt;li&gt;Scenario: Existing customer never used MARC authority app before&#160;&#160;&lt;/li&gt;
		&lt;li&gt;Scenario: Existing customer that already uses MARC authority app: Updating entire record set to support FOLIO authority control&#160;&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Release notes and documentation&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="12962">UXPROD-4533</key>
            <summary>Long term solution for creates/updates of authority records (entire dataset)</summary>
                <type id="10002" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium">New Feature</type>
                            <parent id="11927">UXPROD-2944</parent>
                                    <priority id="10001" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p2.svg">P2</priority>
                        <status id="1" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/open.png" description="The issue is open and ready for the assignee to start work on it.">Open</status>
                    <statusCategory id="2" key="new" colorName="blue-gray"/>
                                    <resolution id="-1">Unresolved</resolution>
                                                        <assignee accountid="62a96ae7192edb006f9f1bf9">Khalilah Gambrell</assignee>
                                                                <reporter accountid="62a96ae7192edb006f9f1bf9">Khalilah Gambrell</reporter>
                                    <labels>
                            <label>NFR</label>
                            <label>SolutionArchitecture</label>
                            <label>arlef-di</label>
                            <label>authority</label>
                            <label>back-end</label>
                            <label>cataloging</label>
                            <label>data-import</label>
                            <label>di-swat</label>
                            <label>loc</label>
                            <label>marc-authority</label>
                            <label>metadatamanagement</label>
                    </labels>
                <created>Mon, 30 Oct 2023 14:49:28 +0000</created>
                <updated>Thu, 14 Dec 2023 18:52:47 +0000</updated>
                                                            <fixVersion>Trillium (R1 2025)</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>9</watches>
                                                                <comments>
                                                            <comment id="14733" author="712020:04e6a696-31fc-4eb7-ba03-b3bb97374090" created="Tue, 31 Oct 2023 11:19:08 +0000"  >&lt;p&gt;Good to know that this is being thought about.&#160; Presumably once done solution will be applied to libraries with existing authority records but less than 500,000.&lt;/p&gt;</comment>
                                                            <comment id="14736" author="62a96ae7192edb006f9f1bf9" created="Tue, 31 Oct 2023 11:24:12 +0000"  >&lt;p&gt;Hey &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=712020%3A04e6a696-31fc-4eb7-ba03-b3bb97374090&quot; class=&quot;user-hover&quot; rel=&quot;712020:04e6a696-31fc-4eb7-ba03-b3bb97374090&quot; data-account-id=&quot;712020:04e6a696-31fc-4eb7-ba03-b3bb97374090&quot; accountid=&quot;712020:04e6a696-31fc-4eb7-ba03-b3bb97374090&quot; rel=&quot;noreferrer&quot;&gt;Ben Taylor&lt;/a&gt;&#160; - We are hoping to release a data import update that allows libraries to more reliably import less than 500,000 records in Poppy or &quot;Q&quot;. How many authority records does Trinity have?&#160;&lt;/p&gt;</comment>
                                                            <comment id="14739" author="712020:04e6a696-31fc-4eb7-ba03-b3bb97374090" created="Tue, 31 Oct 2023 11:35:06 +0000"  >&lt;p&gt;I&apos;ve surprised myself by discovering that we actually have just over 500,000 authority records.&#160; I&apos;ve never considered that the number of authority records would be far more than any other sort of record!&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10001">
                    <name>Cloners</name>
                                            <outwardlinks description="clones">
                                        <issuelink>
            <issuekey id="13764">UXPROD-4082</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10007">
                    <name>Requires</name>
                                            <outwardlinks description="requires">
                                        <issuelink>
            <issuekey id="34850">MODELINKS-84</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="34857">MODELINKS-92</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="56556">MODINVSTOR-1057</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="56561">MODINVSTOR-1068</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Back End Estimate</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10125"><![CDATA[XXXL: 30-45 days]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10054" key="com.atlassian.jira.plugin.system.customfieldtypes:userpicker">
                        <customfieldname>Back End Estimator</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>62a96ae7192edb006f9f1bf9</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10055" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Back-End Confidence factor</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10137"><![CDATA[100%]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10180"><![CDATA[Spitfire]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10014" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue key="$xmlutils.escape($text)">MARC Authority - Phase 1 Features</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10050" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Front End Estimate</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10094"><![CDATA[Out of scope]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10051" key="com.atlassian.jira.plugin.system.customfieldtypes:userpicker">
                        <customfieldname>Front End Estimator</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>62a96ae7192edb006f9f1bf9</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10063" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>PO Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i08elk:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10046" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Release</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10076"><![CDATA[Poppy (R2 2023)]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 31 Oct 2023 11:19:08 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                </customfields>
    </item>
</channel>
</rss>