<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:26:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[FOLIO-3275] run Karate  tests against scratch env with OL enabled</title>
                <link>https://folio-org.atlassian.net/browse/FOLIO-3275</link>
                <project id="10290" key="FOLIO">FOLIO</project>
                    <description>&lt;p&gt;Run Karate tests against scratch env with optimistic locking enabled.&lt;/p&gt;

&lt;p&gt;Currently the FAT modules that have overlap with optimistic locking are as follows:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;mod-audit&lt;/li&gt;
	&lt;li&gt;mod-data-export&lt;/li&gt;
	&lt;li&gt;mod-feesfines&lt;/li&gt;
	&lt;li&gt;mod-patron-blocks (maybe bc mod-patron was only one in scope)&lt;/li&gt;
	&lt;li&gt;mod-search&lt;/li&gt;
	&lt;li&gt;mod-data-import&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This means that these module&apos;s tests can be run separately to save time since running the full suite of tests for FAT takes quite a bit of time.&lt;/p&gt;

&lt;p&gt;There is a branch in FAT that has the okapi ingress for the CP team&apos;s rancher scratch added to each sub-project&apos;s karate config: &lt;a href=&quot;https://github.com/folio-org/folio-integration-tests/tree/core-platform-team-optimistic-locking&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/folio-org/folio-integration-tests/tree/core-platform-team-optimistic-locking&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;To run the test for each of the relevant modules listed above do:&lt;/p&gt;

&lt;p&gt;./runtests.sh mod-audit scratch&lt;/p&gt;

&lt;p&gt;and so on for each module. Most of the tests in mod-feesfines are not implemented (they are marked as @Undefined).&#160; Same for mod-patron-blocks. But there are still a few implemented tests in each. But the other modules (mod-audit, mod-data-export, mod-search) have implemented lots of tests.&lt;/p&gt;</description>
                <environment></environment>
        <key id="82126">FOLIO-3275</key>
            <summary>run Karate  tests against scratch env with OL enabled</summary>
                <type id="10005" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10309?size=medium">Story</type>
                                            <priority id="10005" iconUrl="https://dev.folio.org/assets/jira-priority/tbd.svg">TBD</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="62e181430b4bf7ad924b3732">Steve Ellis</assignee>
                                                                <reporter accountid="557058:f8c71e3c-04a0-49c1-bae9-f8f88e253821">Hanna Hulevich</reporter>
                                    <labels>
                    </labels>
                <created>Mon, 30 Aug 2021 12:21:58 +0000</created>
                <updated>Mon, 27 Sep 2021 12:07:34 +0000</updated>
                            <resolved>Mon, 27 Sep 2021 12:07:34 +0000</resolved>
                                                                        <due></due>
                            <votes>0</votes>
                                    <watches>3</watches>
                                                                <comments>
                                                            <comment id="196587" author="62e181430b4bf7ad924b3732" created="Thu, 16 Sep 2021 22:06:40 +0000"  >&lt;p&gt;&lt;b&gt;Overview&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Karate tests are not passing consistently in the scratch env for modules I&apos;ve been able to test.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Details&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Summary of progress configuring the rancher environment and karate tests to run:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Have spent the day tweaking workload memory. This has helped test reliability somewhat, but it hasn&apos;t been completely effective.&lt;/li&gt;
	&lt;li&gt;A handful of modules were restarting a lot. Adding more memory has fixed that.&lt;/li&gt;
	&lt;li&gt;Okapi logs are only available on stdout. Need to redirect to a file to try to get a better handle on them.&lt;/li&gt;
	&lt;li&gt;Increased request timeouts in karate tests and this has solved any issues related to performance and failures due to this.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Summary of FAT karate module tests that are in scope (see above) so far:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;mod-feesfines - No problems. All tests pass.&lt;/li&gt;
	&lt;li&gt;mod-audit - Tests fail inconsistently (tests mostly pass but different tests fail at different times). Thought it was performance related but now I&apos;m now I&apos;m not so sure. This project&apos;s tests however rely heavily on programmatic pauses between requests, which to me is a code smell.&lt;/li&gt;
	&lt;li&gt;mod-patron-blocks - Was failing with Incompatible version for module &lt;tt&gt;mod-inventory-17.1.0-coreplatform.1 interface instance-storage. Need 7.8. Have 8&lt;/tt&gt;.&#160;Tried reinstalling the module via helm chart thinking it wasn&apos;t current. But it won&apos;t start. Need to investigate the startup logs via kubectl.&lt;/li&gt;
	&lt;li&gt;mod-data-export - Tests can&apos;t connect to module. Need to troubleshoot.&lt;/li&gt;
	&lt;li&gt;mod-search - Haven&apos;t tested.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Summary of spot checks of other modules that are out of scope for OL just to benchmark:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;mod-notes - No problems all tests pass.&lt;/li&gt;
	&lt;li&gt;acquisitions - Lots of failures. This is easily the karate project with the most tests.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The fact that the spot checks in other (non OL modules) suggests that failures are due to a poorly functioning environment rather than OL.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Interested Parties&lt;/b&gt;&lt;br/&gt;
 &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Af8c71e3c-04a0-49c1-bae9-f8f88e253821&quot; class=&quot;user-hover&quot; rel=&quot;557058:f8c71e3c-04a0-49c1-bae9-f8f88e253821&quot; data-account-id=&quot;557058:f8c71e3c-04a0-49c1-bae9-f8f88e253821&quot; accountid=&quot;557058:f8c71e3c-04a0-49c1-bae9-f8f88e253821&quot; rel=&quot;noreferrer&quot;&gt;Hanna Hulevich&lt;/a&gt;, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3Ab8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; class=&quot;user-hover&quot; rel=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; data-account-id=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; accountid=&quot;557058:b8e64633-1f7c-402d-9caf-9959a5ba5d0d&quot; rel=&quot;noreferrer&quot;&gt;Jakub Skoczen&lt;/a&gt;, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5ee89462f7aa140abd82d11d&quot; class=&quot;user-hover&quot; rel=&quot;5ee89462f7aa140abd82d11d&quot; data-account-id=&quot;5ee89462f7aa140abd82d11d&quot; accountid=&quot;5ee89462f7aa140abd82d11d&quot; rel=&quot;noreferrer&quot;&gt;Julian Ladisch&lt;/a&gt;, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5cd423bebc70090d6ce241b1&quot; class=&quot;user-hover&quot; rel=&quot;5cd423bebc70090d6ce241b1&quot; data-account-id=&quot;5cd423bebc70090d6ce241b1&quot; accountid=&quot;5cd423bebc70090d6ce241b1&quot; rel=&quot;noreferrer&quot;&gt;Ian Hardy&lt;/a&gt;, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5af5e627525ba96b58654f12&quot; class=&quot;user-hover&quot; rel=&quot;5af5e627525ba96b58654f12&quot; data-account-id=&quot;5af5e627525ba96b58654f12&quot; accountid=&quot;5af5e627525ba96b58654f12&quot; rel=&quot;noreferrer&quot;&gt;Ann-Marie Breaux&lt;/a&gt;, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=62a96ae7192edb006f9f1bf9&quot; class=&quot;user-hover&quot; rel=&quot;62a96ae7192edb006f9f1bf9&quot; data-account-id=&quot;62a96ae7192edb006f9f1bf9&quot; accountid=&quot;62a96ae7192edb006f9f1bf9&quot; rel=&quot;noreferrer&quot;&gt;Khalilah Gambrell&lt;/a&gt;&lt;/p&gt;</comment>
                                                            <comment id="196588" author="62e181430b4bf7ad924b3732" created="Thu, 16 Sep 2021 22:56:10 +0000"  >&lt;p&gt;If anyone would like to replicate what I&apos;m seeing do the following:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Clone and checkout my branch&#160;&lt;a href=&quot;https://github.com/folio-org/folio-integration-tests/tree/core-platform-team-optimistic-locking&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://github.com/folio-org/folio-integration-tests/tree/core-platform-team-optimistic-locking&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;Run&#160;&lt;tt&gt;./runtests.sh mod-audit scratch&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;Run&#160;&lt;/tt&gt;&lt;tt&gt;./runtests.sh mod-audit testing&lt;/tt&gt;&#160;- to see the same tests succeed against the testing ref env&lt;/li&gt;
&lt;/ol&gt;
</comment>
                                                            <comment id="196589" author="62e181430b4bf7ad924b3732" created="Thu, 16 Sep 2021 23:52:26 +0000"  >&lt;p&gt;After thinking about this for a bit I think it&apos;s likely the env is memory starved.&#160;The cluster&apos;s memory&#160;&lt;em&gt;is&lt;/em&gt; maxed out. I can&apos;t allocate more memory to a pod without taking it away from somewhere else.&lt;/p&gt;

&lt;p&gt;Part of the problem is the nice &lt;a href=&quot;https://rancher.dev.folio.org/c/c-479xv/monitoring&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;cluster dashboard&lt;/a&gt; hasn&apos;t been visible all day. Probably because there&apos;s not enough memory to create it...&lt;/p&gt;

&lt;p&gt;These scratch envs may not have been intended to run platform-complete. We&apos;ve been heaping on a lot of modules onto this thing. More modules need more of course.&lt;/p&gt;</comment>
                                                            <comment id="196590" author="712020:f3088ade-db74-499a-8e6f-5c8d4eb6f5bf" created="Fri, 17 Sep 2021 11:38:11 +0000"  >&lt;p&gt;We assume, that there&apos;s a problem not only with memory, but with CPU too. The charts show, that some of cluster nodes are overloaded in a partiqular time slot. Thus, it can affect the tests are not running. As we found out from metrics and from the general information, it&apos;s hardly possible to increase the memory or CPU limits in this cluster without infrastructure scaling&#160; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;64403_thumb&quot; href=&quot;/rest/api/3/attachment/content/64403&quot; title=&quot;MicrosoftTeams-image (1)-1.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;64403&quot; file-preview-title=&quot;MicrosoftTeams-image (1)-1.png&quot;&gt;&lt;jira-attachment-thumbnail url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64403?default=false&amp;width=355&amp;height=128&quot; jira-url=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64403&quot; filename=&quot;MicrosoftTeams-image (1)-1.png&quot;&gt;&lt;img src=&quot;https://folio-org.atlassian.net/rest/api/3/attachment/thumbnail/64403&quot; data-attachment-name=&quot;MicrosoftTeams-image (1)-1.png&quot; data-attachment-type=&quot;thumbnail&quot; data-media-services-id=&quot;4d5f751f-477e-4163-be00-a15a837664a6&quot; data-media-services-type=&quot;file&quot; height=&quot;128&quot; width=&quot;355&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/jira-attachment-thumbnail&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                                                            <comment id="196591" author="62e181430b4bf7ad924b3732" created="Sat, 18 Sep 2021 15:25:07 +0000"  >&lt;p&gt;Based on &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=712020%3Af3088ade-db74-499a-8e6f-5c8d4eb6f5bf&quot; class=&quot;user-hover&quot; rel=&quot;712020:f3088ade-db74-499a-8e6f-5c8d4eb6f5bf&quot; data-account-id=&quot;712020:f3088ade-db74-499a-8e6f-5c8d4eb6f5bf&quot; accountid=&quot;712020:f3088ade-db74-499a-8e6f-5c8d4eb6f5bf&quot; rel=&quot;noreferrer&quot;&gt;Hleb Surnovich&lt;/a&gt;&apos;s chart I asked around about the process for getting more resources for our cluster. &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5f9abc1eb45b2e007453f423&quot; class=&quot;user-hover&quot; rel=&quot;5f9abc1eb45b2e007453f423&quot; data-account-id=&quot;5f9abc1eb45b2e007453f423&quot; accountid=&quot;5f9abc1eb45b2e007453f423&quot; rel=&quot;noreferrer&quot;&gt;John Malconian&lt;/a&gt;&#160;was able to add three more nodes to increase the underlying compute resources available.&lt;/p&gt;

&lt;p&gt;However I&apos;m not sure if adding these nodes has had an effect on the cpu and ram for the cluster. Comparing what I could see before the change and what I can see after, the number of cores and ram seems to be the same.&lt;/p&gt;

&lt;p&gt;Screenshot before adding nodes:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;/rest/api/3/attachment/content/64404&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Screenshot after adding nodes:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;/rest/api/3/attachment/content/64405&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;My thought is that perhaps our workloads have not been allocated to these new nodes, but I don&apos;t have visibility into the nodes due to permissions.&lt;/p&gt;

&lt;p&gt;Another thing that came up in my conversation with John, was that it could be that we shouldn&apos;t be setting&#160;resources.limit.cpu on the pod level. This may be what is driving up that high percentage (88%) of reserved cpu. Hleb&apos;s chart suggests that our cpu can go in excess of 100% which seems like it could lead to unpredictable behavior.&lt;/p&gt;

&lt;p&gt;At our institution we don&apos;t set any value for&#160;resources.limit.cpu in our folio k8s deployment, which we think tells k8s to let any pod use whatever cpu it wants. Our reserved cpu is therefore much lower.&lt;/p&gt;

&lt;p&gt;I&apos;ve concentrated on trying to closely observe &lt;a href=&quot;https://github.com/folio-org/folio-integration-tests/blob/master/mod-audit/src/main/resources/domain/mod-audit/features/requestEvent.feature&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;this particular karate test&lt;/a&gt;, which includes 3 scenarios. (mod-audit is in scope for OL.) When run against the scratch env, I&apos;ve seen arbitrary combinations of the 3 tests succeed or fail &#8211; something that does not happen at all when run against a reference env. Since these tests rely on pauses, I&apos;ve increased those to 30 seconds, from 5 seconds, with the idea that giving the updates more time to propagate through a resource constrained system would improve matters. It hasn&apos;t.&lt;/p&gt;

&lt;p&gt;It could be that these tests are failing because of OL, but I don&apos;t think we can yet rule out that the scratch env is configured correctly.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                                                            <comment id="196592" author="62e181430b4bf7ad924b3732" created="Sun, 19 Sep 2021 20:38:13 +0000"  >&lt;p&gt;A few updates on this.&lt;/p&gt;

&lt;p&gt;I am able to add ram and modify cpu quotas on pods, whereas before John&apos;s change I wasn&apos;t, so perhaps those views summarizing ram and cpu are noise, and the new nodes have opened things up.&lt;/p&gt;

&lt;p&gt;But the &lt;a href=&quot;https://github.com/folio-org/folio-integration-tests/blob/core-platform-team-optimistic-locking/mod-audit/src/main/resources/domain/mod-audit/features/requestEvent.feature&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;karate test I&apos;m focusing on&lt;/a&gt; still is failing in the same unpredictable way.&lt;/p&gt;

&lt;p&gt;I&apos;m uploading some logs from mod-circulation and mod-audit, two modules that are in scope for the test in question. There are some things of note there, mostly from postgres complaining about indexes not being present. So new theory: maybe the db isn&apos;t being configured the same in the scratch env as in the reference envs, making things super slow. Seems plausible, but is just a guess.&lt;/p&gt;

&lt;p&gt;At this point I think we really have two possibilities:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Everything is fine with the cluster, and it&apos;s just these tests that we need to worry about. Other functional tests can proceed.&lt;/li&gt;
	&lt;li&gt;These problems are signs of a bigger configuration problem with the cluster, like the db config being broken, or k8s cpu quotas being broken, causing requests to take ridiculous amounts of time. In this case functional testing should probably pause while we figure out how to fix the env.&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;Again I would be grateful if I could get a few more eyes on this. Maybe something will occur to someone. Instructions for running the tests are above.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Why I still think this is a performance issue&lt;/b&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&#160;From the beginning the cluster has been super slow to register modules (but unusually slow for mod-audit and mod-circulation), the two modules that are in the problematic test. Increasing timeouts in my karate tests has helped that.&lt;/li&gt;
	&lt;li&gt;I&apos;ve seen at least once vertx exceeding its timeout in a module. Can&apos;t recall which one.&lt;/li&gt;
	&lt;li&gt;The problematic tests rely heavily on pauses, which implies that the requests they are waiting to complete, are &quot;fire and forget&quot;. But I&apos;ve increased these pauses to what I would consider crazy levels (5 secs to 1 minute) without any effect.&lt;/li&gt;
	&lt;li&gt;There&apos;s no &quot;smoking gun&quot; unhandled exception or other error in the logs for mod-inventory or mod-inventory-storage which would imply that this has something to do with OL.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                                                            <comment id="196593" author="62e181430b4bf7ad924b3732" created="Tue, 21 Sep 2021 22:10:20 +0000"  >&lt;p&gt;The performance issues have been resolved by removing resources.limits.cpu and resources.requests.cpu from each pod spec in our namespace. To protect other team&apos;s clusters, I added a namespace-wide limit of 16000m to our namespace (core-platform).&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                                                <inwardlinks description="is blocked by">
                                        <issuelink>
            <issuekey id="73984">MODPATBLK-101</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="64403" name="MicrosoftTeams-image (1)-1.png" size="100935" author="712020:f3088ade-db74-499a-8e6f-5c8d4eb6f5bf" created="Fri, 17 Sep 2021 11:30:30 +0000"/>
                            <attachment id="64404" name="image-2021-09-18-11-07-54-237.png" size="6784" author="62e181430b4bf7ad924b3732" created="Sat, 18 Sep 2021 15:07:55 +0000"/>
                            <attachment id="64405" name="image-2021-09-18-11-08-19-748.png" size="5984" author="62e181430b4bf7ad924b3732" created="Sat, 18 Sep 2021 15:08:20 +0000"/>
                            <attachment id="64406" name="mod-audit.log" size="15890" author="62e181430b4bf7ad924b3732" created="Sun, 19 Sep 2021 20:42:10 +0000"/>
                            <attachment id="64407" name="mod-circulation-storage.log" size="726" author="62e181430b4bf7ad924b3732" created="Sun, 19 Sep 2021 20:42:10 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10144"><![CDATA[Core: Platform]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10063" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>PO Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|hzx3o9:hzzr</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="481">CP: sprint 122</customfieldvalue>
    <customfieldvalue id="901">CP: sprint 123</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10044" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 16 Sep 2021 22:06:40 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>