<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:12:16 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[MODUIMP-4] Bulk import of users needs performance improvements</title>
                <link>https://folio-org.atlassian.net/browse/MODUIMP-4</link>
                <project id="10211" key="MODUIMP">mod-user-import</project>
                    <description>&lt;p&gt;We need performant bulk import of users for both migration and ongoing operations. &lt;/p&gt;

&lt;p&gt;The current user import module submits users one-by-one, which is rather slow at a scale of, say, 90K users, especially when trying to test data loading at scale. SysOps foresees similar issues with loading data into other modules.&lt;/p&gt;

&lt;p&gt;There are two general scenarios:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;At migration time, sites will need to do an initial bulk import from their existing systems, and new UUIDs will somehow need to be minted as part of the migration process.&lt;/li&gt;
	&lt;li&gt;In ongoing operations, sites will do regular bulk updates from their campus identity management (IdM) infrastructure to refresh their campus users. In this case, existing UUIDs and some internal FOLIO fields will need to be preserved.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The migration scenario here seems like a special case of the bulk import to support operations.&lt;/p&gt;

&lt;p&gt;There will be a need for bulk imports across FOLIO modules for both migration and operations. Users seems like a good starting point to consider some common strategies or techniques for bulk import that could inform best practices for project as a whole.&lt;/p&gt;

&lt;p&gt;This is related to 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;MODUSERS-3&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/MODUSERS-3&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Add bulk-loading functionality to mod-users&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium&quot; /&gt;
            MODUSERS-3
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
.&lt;/p&gt;</description>
                <environment></environment>
        <key id="76962">MODUIMP-4</key>
            <summary>Bulk import of users needs performance improvements</summary>
                <type id="10008" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10300?size=medium">Tech Debt</type>
                                            <priority id="10002" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p3.svg">P3</priority>
                        <status id="1" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/open.png" description="The issue is open and ready for the assignee to start work on it.">Open</status>
                    <statusCategory id="2" key="new" colorName="blue-gray"/>
                                    <resolution id="-1">Unresolved</resolution>
                                                        <assignee accountid="-1">Unassigned</assignee>
                                                                <reporter accountid="557058:b787c0f1-34df-41c2-8585-c4aed346caec">Tod Olson</reporter>
                                    <labels>
                            <label>back-end</label>
                            <label>migration-load</label>
                            <label>qulto</label>
                            <label>sprint46</label>
                            <label>sprint47</label>
                            <label>sprint48</label>
                    </labels>
                <created>Wed, 20 Jun 2018 16:14:20 +0000</created>
                <updated>Tue, 19 Jan 2021 20:52:19 +0000</updated>
                                                                                <due></due>
                            <votes>0</votes>
                                    <watches>10</watches>
                                                                <comments>
                                                            <comment id="184498" author="5af5ed55244bc90a106063c7" created="Thu, 21 Jun 2018 09:10:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Ab8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; class=&quot;user-hover&quot; rel=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; data-account-id=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; accountid=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; rel=&quot;noreferrer&quot;&gt;Jakub Skoczen&lt;/a&gt;, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5e17508d5361330daaeb6d63&quot; class=&quot;user-hover&quot; rel=&quot;5e17508d5361330daaeb6d63&quot; data-account-id=&quot;5e17508d5361330daaeb6d63&quot; accountid=&quot;5e17508d5361330daaeb6d63&quot; rel=&quot;noreferrer&quot;&gt;Istv&#225;n Bender&lt;/a&gt;, thoughts on how we can improve performance here?  Is it time to add bulk loading functionality per 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;MODUSERS-3&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/MODUSERS-3&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Add bulk-loading functionality to mod-users&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium&quot; /&gt;
            MODUSERS-3
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
?  FYI &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=62a96ae7192edb006f9f1bf9&quot; class=&quot;user-hover&quot; rel=&quot;62a96ae7192edb006f9f1bf9&quot; data-account-id=&quot;62a96ae7192edb006f9f1bf9&quot; accountid=&quot;62a96ae7192edb006f9f1bf9&quot; rel=&quot;noreferrer&quot;&gt;Khalilah Gambrell&lt;/a&gt;, as well.&lt;/p&gt;</comment>
                                                            <comment id="184499" author="5e17508d5361330daaeb6d63" created="Mon, 25 Jun 2018 09:32:19 +0000"  >&lt;p&gt;I can profile which part takes long time but my gut feeling is that calling the mod-users API twice per user (create/update user and adding empty permission set) must be the bottleneck. Bulk user creation would definitely improve the performance of user import.&lt;/p&gt;</comment>
                                                            <comment id="184500" author="5e17508d5361330daaeb6d63" created="Tue, 28 Aug 2018 12:10:51 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Ab787c0f1-34df-41c2-8585-c4aed346caec&quot; class=&quot;user-hover&quot; rel=&quot;557058:b787c0f1-34df-41c2-8585-c4aed346caec&quot; data-account-id=&quot;557058:b787c0f1-34df-41c2-8585-c4aed346caec&quot; accountid=&quot;557058:b787c0f1-34df-41c2-8585-c4aed346caec&quot; rel=&quot;noreferrer&quot;&gt;Tod Olson&lt;/a&gt; could you send me a sample user import JSON file which contains 90k users to import. I will do some profiling and performance tests.&lt;/p&gt;</comment>
                                                            <comment id="184501" author="557058:c2706f9e-f281-4999-8937-b5c5a8ff211f" created="Wed, 29 Aug 2018 15:20:17 +0000"  >&lt;p&gt;I generated a sample file of 100,000 users and attached it to this issue. The file just contains generated data and a subset of the JSON attributes. I&apos;m thinking this will get you going. I can create a more realistic file if needed.&lt;/p&gt;</comment>
                                                            <comment id="184502" author="5e17508d5361330daaeb6d63" created="Thu, 30 Aug 2018 12:03:54 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Ac2706f9e-f281-4999-8937-b5c5a8ff211f&quot; class=&quot;user-hover&quot; rel=&quot;557058:c2706f9e-f281-4999-8937-b5c5a8ff211f&quot; data-account-id=&quot;557058:c2706f9e-f281-4999-8937-b5c5a8ff211f&quot; accountid=&quot;557058:c2706f9e-f281-4999-8937-b5c5a8ff211f&quot; rel=&quot;noreferrer&quot;&gt;Jon Miller&lt;/a&gt; I generated a new one (using &lt;a href=&quot;http://www.mockaroo.com/&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;http://www.mockaroo.com/&lt;/a&gt;) because some attributes (patronGroup, addresses, type, dateOfBirth) were missing from your sample file. I cannot upload it here because the compressed size is 15M and Jira allows only max 10M file size to upload.&lt;/p&gt;

&lt;p&gt;An example user in my samle JSON:&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
    {
        &lt;span class=&quot;code-quote&quot;&gt;&quot;active&quot;&lt;/span&gt;: &lt;span class=&quot;code-keyword&quot;&gt;true&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;barcode&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;19-971-1678&quot;&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;externalSystemId&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;8294b291-f227-4cb8-8635-762fadc8ce7f&quot;&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;patronGroup&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;graduate&quot;&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;personal&quot;&lt;/span&gt;: {
            &lt;span class=&quot;code-quote&quot;&gt;&quot;addresses&quot;&lt;/span&gt;: [
                {
                    &lt;span class=&quot;code-quote&quot;&gt;&quot;addressLine1&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;74 Fuller Point&quot;&lt;/span&gt;,
                    &lt;span class=&quot;code-quote&quot;&gt;&quot;addressTypeId&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;Payment&quot;&lt;/span&gt;,
                    &lt;span class=&quot;code-quote&quot;&gt;&quot;city&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;San Francisco&quot;&lt;/span&gt;,
                    &lt;span class=&quot;code-quote&quot;&gt;&quot;postalCode&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;94132&quot;&lt;/span&gt;,
                    &lt;span class=&quot;code-quote&quot;&gt;&quot;primaryAddress&quot;&lt;/span&gt;: &lt;span class=&quot;code-keyword&quot;&gt;true&lt;/span&gt;,
                    &lt;span class=&quot;code-quote&quot;&gt;&quot;region&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;California&quot;&lt;/span&gt;
                }
            ],
            &lt;span class=&quot;code-quote&quot;&gt;&quot;dateOfBirth&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;1968-04-23&quot;&lt;/span&gt;,
            &lt;span class=&quot;code-quote&quot;&gt;&quot;email&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;aguppie0@vinaora.com&quot;&lt;/span&gt;,
            &lt;span class=&quot;code-quote&quot;&gt;&quot;firstName&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;Aubrey&quot;&lt;/span&gt;,
            &lt;span class=&quot;code-quote&quot;&gt;&quot;lastName&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;Guppie&quot;&lt;/span&gt;,
            &lt;span class=&quot;code-quote&quot;&gt;&quot;mobilePhone&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;235(862)784-4412&quot;&lt;/span&gt;,
            &lt;span class=&quot;code-quote&quot;&gt;&quot;phone&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;48(210)504-6148&quot;&lt;/span&gt;,
            &lt;span class=&quot;code-quote&quot;&gt;&quot;preferredContactTypeId&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;mail&quot;&lt;/span&gt;
        },
        &lt;span class=&quot;code-quote&quot;&gt;&quot;type&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;patron&quot;&lt;/span&gt;,
        &lt;span class=&quot;code-quote&quot;&gt;&quot;username&quot;&lt;/span&gt;: &lt;span class=&quot;code-quote&quot;&gt;&quot;969402e4-cf2d-4255-be18-8673c58939f3&quot;&lt;/span&gt;
    }
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Feel free to use my schema: &lt;a href=&quot;https://www.mockaroo.com/c238b770&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://www.mockaroo.com/c238b770&lt;/a&gt;&lt;/p&gt;</comment>
                                                            <comment id="184503" author="557058:c2706f9e-f281-4999-8937-b5c5a8ff211f" created="Thu, 30 Aug 2018 12:27:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5e17508d5361330daaeb6d63&quot; class=&quot;user-hover&quot; rel=&quot;5e17508d5361330daaeb6d63&quot; data-account-id=&quot;5e17508d5361330daaeb6d63&quot; accountid=&quot;5e17508d5361330daaeb6d63&quot; rel=&quot;noreferrer&quot;&gt;Istv&#225;n Bender&lt;/a&gt; Thanks&lt;/p&gt;</comment>
                                                            <comment id="184504" author="5af5ed55244bc90a106063c7" created="Wed, 5 Sep 2018 09:01:13 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5e17508d5361330daaeb6d63&quot; class=&quot;user-hover&quot; rel=&quot;5e17508d5361330daaeb6d63&quot; data-account-id=&quot;5e17508d5361330daaeb6d63&quot; accountid=&quot;5e17508d5361330daaeb6d63&quot; rel=&quot;noreferrer&quot;&gt;Istv&#225;n Bender&lt;/a&gt;.  Any test results yet?  &lt;/p&gt;</comment>
                                                            <comment id="184505" author="5e17508d5361330daaeb6d63" created="Wed, 5 Sep 2018 09:10:32 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=63f928fafb3ac4003fa4257d&quot; class=&quot;user-hover&quot; rel=&quot;63f928fafb3ac4003fa4257d&quot; data-account-id=&quot;63f928fafb3ac4003fa4257d&quot; accountid=&quot;63f928fafb3ac4003fa4257d&quot; rel=&quot;noreferrer&quot;&gt;Zoltan Erdos&lt;/a&gt; will do it in the current sprint. He will also give an estimate how many effort to integrate mod-user-import in mod-users. I hope we will be able to answer these questions at the end of this week.&lt;/p&gt;</comment>
                                                            <comment id="184506" author="63f928fafb3ac4003fa4257d" created="Tue, 11 Sep 2018 09:24:13 +0000"  >&lt;p&gt;Test environment.&lt;/p&gt;

&lt;p&gt;16GB DDR4 RAM, i7-7820HQ CPU, 240GB SSD&lt;/p&gt;

&lt;p&gt;base vagrant box : folio/testing&lt;br/&gt;
Replaced/Redeployed modules, from locally source code with embedded postgres:&lt;br/&gt;
mod-users, mod-permissions, mod-user-import, mod-users-bl&lt;/p&gt;

&lt;p&gt;Used ram for these 4 module (run from intelliJ IDEA) was 2.5GB(max value) when I imported 100.000 user at once. The json file with the users was 92MB.&lt;/p&gt;

&lt;p&gt;Importing time test results:&lt;/p&gt;

&lt;p&gt;5k insert - 1.0 min;0.78 minutes; 0.85 min, 0.72 min&lt;br/&gt;
5k update - 0.5 min; 0.47 min; 0.46 min, 0.47 min, 0.5 min&lt;br/&gt;
100k insert: 24.5 min, 25min&lt;/p&gt;

&lt;p&gt;The 40% of the import time is spent in mod users to get users by external id-s.&lt;br/&gt;
The 20% is for the POST or PUT the users into mod-users.&lt;br/&gt;
The rest 40% is for POST permissions info into mod-permission module.&lt;/p&gt;

&lt;p&gt;It is important to add temporarily more RAM for mod-user-import, mod-user, mod-permission if they want to import a lot of user (more than 10k).&lt;/p&gt;

&lt;p&gt;The slowest parts of the user import are:&lt;br/&gt;
Querying mod-users by external-id (bulk GET with 10-30 user at once). &lt;br/&gt;
Insert permissions to mod-permissions.&lt;/p&gt;

&lt;p&gt;Recommendation for mod-users developers:&lt;br/&gt;
The user import would be faster, if there will be an index on the users table on the externalSystemId field.&lt;/p&gt;

&lt;p&gt;Recommendation for mod-permissions developers:&lt;br/&gt;
Review postPermsUsers implementation is necessary, because POST permessionUser it takes a little longer than we expected. (10 times longer(3-5ms vs 30-50ms)) than post new user into mod-users)&lt;/p&gt;</comment>
                                                            <comment id="184507" author="5e17508d5361330daaeb6d63" created="Fri, 28 Sep 2018 13:18:56 +0000"  >&lt;p&gt;We performed new bulk import test in the following environment:&lt;/p&gt;

&lt;p&gt;16GB DDR4 RAM, i7-7820HQ CPU, 240GB SSD&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;base vagrant box : folio/testing version: 5.0.0-20180925.1100&lt;/li&gt;
	&lt;li&gt;mod_users, version: 15.3.0&lt;/li&gt;
	&lt;li&gt;mod_user_import, version: 3.1.1&lt;/li&gt;
	&lt;li&gt;mod_permissions, version: 5.4.0&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;All modules are running in Vagrant box none of them hosted locally.&lt;/p&gt;

&lt;p&gt;Experiences:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Vagrant box and services are starting much slower than usual&lt;/li&gt;
	&lt;li&gt;The performance of user  import is much lower than it was during our last measure. 1000 user&apos;s import took more then 5 minutes!&lt;/li&gt;
	&lt;li&gt;We experienced a lot of postgres process running at the same time in the box consuming huge CPU (see attached image)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;62649_thumb&quot; href=&quot;/rest/api/3/attachment/content/62649&quot; title=&quot;vagrant_top.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;62649&quot; file-preview-title=&quot;vagrant_top.png&quot;&gt;&lt;jira-attachment-thumbnail url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/62649?default=false&quot; jira-url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/62649&quot; filename=&quot;vagrant_top.png&quot;&gt;&lt;img src=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/62649&quot; data-attachment-name=&quot;vagrant_top.png&quot; data-attachment-type=&quot;thumbnail&quot; data-media-services-id=&quot;2b3b56b3-d115-4dab-aa9e-538ea8842d46&quot; data-media-services-type=&quot;file&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/jira-attachment-thumbnail&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;We didn&apos;t have time to dig deeper and identify the root cause of poor performance but something may changed in a wrong way since our last benchmark. I cannot state that the cause of slow-down is mod-users or mod-permission modules. Perhaps there is something completely different reason.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Ab8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; class=&quot;user-hover&quot; rel=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; data-account-id=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; accountid=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; rel=&quot;noreferrer&quot;&gt;Jakub Skoczen&lt;/a&gt; What should we do now? What do you suggest considering we don&apos;t have too much time until our contract termination.&lt;/p&gt;</comment>
                                                            <comment id="184508" author="5f5241b9fcaf93003b5bee25" created="Mon, 24 Feb 2020 20:16:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5af5ed55244bc90a106063c7&quot; class=&quot;user-hover&quot; rel=&quot;5af5ed55244bc90a106063c7&quot; data-account-id=&quot;5af5ed55244bc90a106063c7&quot; accountid=&quot;5af5ed55244bc90a106063c7&quot; rel=&quot;noreferrer&quot;&gt;Cate Boerema&lt;/a&gt;, Ian and I are reviewing this ticket - would this be superceded by Modusers-3?&lt;/p&gt;

&lt;p&gt;Still looking for a benchamrk of 70 records per second.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                            <outwardlinks description="blocks">
                                        <issuelink>
            <issuekey id="80821">FOLIO-1470</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is blocked by">
                                        <issuelink>
            <issuekey id="67627">MODUSERS-85</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="34252">MODPERMS-46</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10008">
                    <name>Defines</name>
                                            <outwardlinks description="defines">
                                        <issuelink>
            <issuekey id="11817">UXPROD-2873</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10003">
                    <name>Relates</name>
                                            <outwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="67606">MODUSERS-3</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="10132">UXPROD-850</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="62648" name="users.zip" size="4634726" author="557058:c2706f9e-f281-4999-8937-b5c5a8ff211f" created="Wed, 29 Aug 2018 15:16:21 +0000"/>
                            <attachment id="62649" name="vagrant_top.png" size="97547" author="5e17508d5361330daaeb6d63" created="Fri, 28 Sep 2018 13:11:34 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|hzyo2n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 21 Jun 2018 09:10:36 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                </customfields>
    </item>
</channel>
</rss>