<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:22:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[FOLIO-2638] many platform-core and ui-users builds are dying OOM in CI. why???</title>
                <link>https://folio-org.atlassian.net/browse/FOLIO-2638</link>
                <project id="10290" key="FOLIO">FOLIO</project>
                    <description>&lt;p&gt;We have been seeing build failures for numerous PRs in ui-users and platform-core. One option discussed on Slack is &lt;a href=&quot;https://folio-project.slack.com/archives/C58TABALV/p1591367823335100&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;more build nodes&lt;/a&gt;. &lt;br/&gt;
Example from &lt;a href=&quot;https://jenkins-aws.indexdata.com/job/folio-org/job/ui-users/job/UIU-1273/1/execution/node/75/log/&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;ui-users PR 1347&lt;/a&gt;. Everything below &lt;tt&gt;Last few GCs&lt;/tt&gt; repeats until the process times out after 30-60 minutes. &lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
$ stripes test karma --bundle --karma.singleRun --karma.browsers ChromeDocker --karma.reporters mocha junit --coverage
Starting Karma tests...

START:

&amp;lt;--- Last few GCs ---&amp;gt;

[1304:0x4376ec0]    67996 ms: Scavenge 2040.2 (2050.1) -&amp;gt; 2039.6 (2050.1) MB, 12.7 / 0.0 ms  (average mu = 0.139, current mu = 0.003) allocation failure 
[1304:0x4376ec0]    68075 ms: Scavenge 2040.4 (2050.1) -&amp;gt; 2039.7 (2050.1) MB, 9.9 / 0.0 ms  (average mu = 0.139, current mu = 0.003) allocation failure 
[1304:0x4376ec0]    68162 ms: Scavenge 2040.5 (2050.1) -&amp;gt; 2039.8 (2050.3) MB, 10.2 / 0.0 ms  (average mu = 0.139, current mu = 0.003) allocation failure 


&amp;lt;--- JS stacktrace ---&amp;gt;

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13c03d9]
    1: StubFrame [pc: 0x1347261]
Security context: 0x0ee6c78808d1 &amp;lt;JSObject&amp;gt;
    2: &lt;span class=&quot;code-comment&quot;&gt;/* anonymous */&lt;/span&gt;(aka &lt;span class=&quot;code-comment&quot;&gt;/* anonymous */&lt;/span&gt;) [0x1cd67e0ed9e1] [/home/jenkins/workspace/folio-org_ui-users_UIU-1273/project/node_modules/terser/dist/bundle.min.js:~1] [pc=0x9a8771d5b6d](&lt;span class=&quot;code-keyword&quot;&gt;this&lt;/span&gt;=0x05738c1004b1 &amp;lt;undefined&amp;gt;,0x153379fcd419 &amp;lt;AST_Object map = 0x32e8d1462089&amp;gt;,0x1b1316d46479 &amp;lt;On map = 0x183e21109129&amp;gt;)
    3: &lt;span class=&quot;code-comment&quot;&gt;/* anonymous */&lt;/span&gt; [0...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200605.194727.1304.0.001.json
Node.js report completed
 1: 0xa02f90 node::Abort() [/usr/bin/node]
 2: 0xa033b5 node::OnFatalError(&lt;span class=&quot;code-object&quot;&gt;char&lt;/span&gt; &lt;span class=&quot;code-keyword&quot;&gt;const&lt;/span&gt;*, &lt;span class=&quot;code-object&quot;&gt;char&lt;/span&gt; &lt;span class=&quot;code-keyword&quot;&gt;const&lt;/span&gt;*) [/usr/bin/node]
 3: 0xb76ffe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, &lt;span class=&quot;code-object&quot;&gt;char&lt;/span&gt; &lt;span class=&quot;code-keyword&quot;&gt;const&lt;/span&gt;*, bool) [/usr/bin/node]
 4: 0xb77379 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, &lt;span class=&quot;code-object&quot;&gt;char&lt;/span&gt; &lt;span class=&quot;code-keyword&quot;&gt;const&lt;/span&gt;*, bool) [/usr/bin/node]
 5: 0xd23ad5  [/usr/bin/node]
 6: 0xd24166 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/bin/node]
 7: 0xd309e5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/bin/node]
 8: 0xd31895 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
 9: 0xd3434c v8::internal::Heap::AllocateRawWithRetryOrFail(&lt;span class=&quot;code-object&quot;&gt;int&lt;/span&gt;, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
10: 0xcfaf1b v8::internal::Factory::NewFillerObject(&lt;span class=&quot;code-object&quot;&gt;int&lt;/span&gt;, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
11: 0x103d85e v8::internal::Runtime_AllocateInYoungGeneration(&lt;span class=&quot;code-object&quot;&gt;int&lt;/span&gt;, unsigned &lt;span class=&quot;code-object&quot;&gt;long&lt;/span&gt;*, v8::internal::Isolate*) [/usr/bin/node]
12: 0x13c03d9  [/usr/bin/node]
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;Similar results for &lt;a href=&quot;https://jenkins-aws.indexdata.com/job/folio-org/job/platform-core/view/change-requests/job/PR-601/3/console&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;platform-core PRs&lt;/a&gt;. &lt;/p&gt;</description>
                <environment></environment>
        <key id="81669">FOLIO-2638</key>
            <summary>many platform-core and ui-users builds are dying OOM in CI. why???</summary>
                <type id="10001" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10303?size=medium">Bug</type>
                                            <priority id="10005" iconUrl="https://dev.folio.org/assets/jira-priority/tbd.svg">TBD</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="5f9abc1eb45b2e007453f423">John Malconian</assignee>
                                                                <reporter accountid="615afd1cd9820f0070a09ef0">Zak Burke</reporter>
                                    <labels>
                    </labels>
                <created>Mon, 8 Jun 2020 11:47:20 +0000</created>
                <updated>Wed, 10 Jun 2020 13:08:49 +0000</updated>
                            <resolved>Tue, 9 Jun 2020 17:30:52 +0000</resolved>
                                                                        <due></due>
                            <votes>0</votes>
                                    <watches>4</watches>
                                                                <comments>
                                                            <comment id="199067" author="615afd1cd9820f0070a09ef0" created="Mon, 8 Jun 2020 11:49:25 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5f9abc1eb45b2e007453f423&quot; class=&quot;user-hover&quot; rel=&quot;5f9abc1eb45b2e007453f423&quot; data-account-id=&quot;5f9abc1eb45b2e007453f423&quot; accountid=&quot;5f9abc1eb45b2e007453f423&quot; rel=&quot;noreferrer&quot;&gt;John Malconian&lt;/a&gt;, I&apos;m investigating whether a code change on our end, or a change in some third-party dep could be responsible, but there are no red flags so far. Is it easy to throw RAM at this? If so, that may be best to take off the pressure while folks are trying to publish releases this week, and we can continue the investigation later. &lt;/p&gt;</comment>
                                                            <comment id="199069" author="5f9abc1eb45b2e007453f423" created="Tue, 9 Jun 2020 11:28:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=615afd1cd9820f0070a09ef0&quot; class=&quot;user-hover&quot; rel=&quot;615afd1cd9820f0070a09ef0&quot; data-account-id=&quot;615afd1cd9820f0070a09ef0&quot; accountid=&quot;615afd1cd9820f0070a09ef0&quot; rel=&quot;noreferrer&quot;&gt;Zak Burke&lt;/a&gt; This is pretty strange.   Nothing has changed on the build nodes or build images recently that I&apos;m aware of.   Have we updated Node recently &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=61cd0ca0bce5e00069e98be7&quot; class=&quot;user-hover&quot; rel=&quot;61cd0ca0bce5e00069e98be7&quot; data-account-id=&quot;61cd0ca0bce5e00069e98be7&quot; accountid=&quot;61cd0ca0bce5e00069e98be7&quot; rel=&quot;noreferrer&quot;&gt;David Crossley&lt;/a&gt;?   I don&apos;t think so.    &lt;/p&gt;

&lt;p&gt;Currently we have five build nodes,  each with 32 GB of RAM (t2.2xlarge)  and only two jobs can run on a build node at one time.    If I add build nodes,  we&apos;d still have the same issue unless I restricted jobs to one job per node.   Build jobs would queue up.  The other option would be to to upgrade the instance type.   The next tier would double RAM to 64GB per node.   I think that could get expensive.   What do you think, &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5ced27478b03050f27825a93&quot; class=&quot;user-hover&quot; rel=&quot;5ced27478b03050f27825a93&quot; data-account-id=&quot;5ced27478b03050f27825a93&quot; accountid=&quot;5ced27478b03050f27825a93&quot; rel=&quot;noreferrer&quot;&gt;Peter Murray&lt;/a&gt;? &lt;/p&gt;</comment>
                                                            <comment id="199071" author="61cd0ca0bce5e00069e98be7" created="Tue, 9 Jun 2020 12:21:25 +0000"  >&lt;p&gt;No recent build image changes, as far as i know.&lt;/p&gt;</comment>
                                                            <comment id="199074" author="615afd1cd9820f0070a09ef0" created="Tue, 9 Jun 2020 12:58:11 +0000"  >&lt;p&gt;I&apos;ve been able to duplicate this &lt;a href=&quot;https://jenkins-aws.indexdata.com/job/folio-org/job/platform-core/view/change-requests/job/PR-601/3/console&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;platform-core build failure&lt;/a&gt; locally, sometimes:&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
export NODE_OPTIONS=&lt;span class=&quot;code-quote&quot;&gt;&quot;--max-old-space-size=2048 $NODE_OPTIONS&quot;&lt;/span&gt;; stripes build stripes.config.js --okapi https:&lt;span class=&quot;code-comment&quot;&gt;//localhost:9130 --tenant diku ./output&lt;/span&gt;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;Unfortunately, I didn&apos;t notice when &lt;a href=&quot;https://jenkins-aws.indexdata.com/job/Automation/job/build-platform-core-snapshot/&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;build-platform-core-snapshot&lt;/a&gt; first started failing++  so the diff between the current and last-good &lt;tt&gt;yarn.lock&lt;/tt&gt; files has grown, but I&apos;ll see if I can get anywhere with it today. &lt;/p&gt;

&lt;p&gt;The ui-users#master build started failing on Thursday, and we &lt;em&gt;do&lt;/em&gt; still have those old &lt;tt&gt;yarn.lock&lt;/tt&gt; files. That build doesn&apos;t have the &lt;tt&gt;--max-old-space-size&lt;/tt&gt; but I will add it and see if I can replicate the failure. &lt;/p&gt;


&lt;p&gt;++ I gotta figure out a way to monitor the UI master builds and the platform-core and platform-complete snapshot builds to get notified when they fail; I made a &lt;a href=&quot;http://zburke.com/folio-tools/dashboard/&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;dashboard&lt;/a&gt; but I have to go look at it. I know we have Slack#folio-ci, but that&apos;s awfully noisy and I only care about certain failed builds. &lt;/p&gt;</comment>
                                                            <comment id="199076" author="615afd1cd9820f0070a09ef0" created="Tue, 9 Jun 2020 13:03:15 +0000"  >&lt;blockquote&gt;&lt;p&gt;The next tier would double RAM to 64GB per node.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Re-reading that, if we currently have 32GB, is it possible to dedicate more to node, e.g. &lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
export NODE_OPTIONS=&lt;span class=&quot;code-quote&quot;&gt;&quot;--max-old-space-size=4096 $NODE_OPTIONS&quot;&lt;/span&gt;;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                                                            <comment id="199077" author="615afd1cd9820f0070a09ef0" created="Tue, 9 Jun 2020 13:21:49 +0000"  >&lt;p&gt;Also, why does &lt;tt&gt;build-platform-core-snapshot&lt;/tt&gt; fail if &lt;tt&gt;build-platform-complete-snapshot&lt;/tt&gt; succeeds?!?&lt;/p&gt;</comment>
                                                            <comment id="199080" author="5ced27478b03050f27825a93" created="Tue, 9 Jun 2020 14:15:28 +0000"  >&lt;p&gt;I&apos;m up for an experiment.  How hard would it be to create a higher-capacity instance and pipe some jobs through it to see if the issue goes away?&lt;/p&gt;

&lt;p&gt;The EC2 reserved instances we have now are &quot;Convertable&quot;, so if we need to move to a higher capacity instance or a different instance family, we can certainly do that.  (As long as we stay in the same AWS region, we just pay the difference in cost at the time of conversion.)&lt;/p&gt;</comment>
                                                            <comment id="199082" author="5f9abc1eb45b2e007453f423" created="Tue, 9 Jun 2020 14:23:03 +0000"  >&lt;p&gt;platform-core is built with the following NODE_OPTIONS.   max-old-space-size is half the amount allocated to platform-complete.  &lt;/p&gt;

&lt;p&gt;export NODE_OPTIONS=&quot;--max-old-space-size=2048 $NODE_OPTIONS&quot;; stripes build stripes.config.js --okapi &lt;a href=&quot;https://localhost:9130&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://localhost:9130&lt;/a&gt; --tenant diku ./output&lt;/p&gt;

&lt;p&gt;I&apos;ll update that to 4096.  &lt;/p&gt;

&lt;p&gt;Additionally,  NODE_OPTIONS is set for &apos;stripes build&apos; but I&apos;m not sure what, if anything, NODE_OPTIONS is set to during unit test runs.   It looks like we just need to allocate more memory overall to Node.  &lt;/p&gt;</comment>
                                                            <comment id="199084" author="615afd1cd9820f0070a09ef0" created="Tue, 9 Jun 2020 15:15:48 +0000"  >&lt;p&gt;OK, so, we allocate more memory to Node but leave the AWS instance alone &lt;img class=&quot;emoticon&quot; src=&quot;/images/icons/emoticons/check.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;. LMK when this is in place so I can test some of the PRs that have been blocked. &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5f9abc1eb45b2e007453f423&quot; class=&quot;user-hover&quot; rel=&quot;5f9abc1eb45b2e007453f423&quot; data-account-id=&quot;5f9abc1eb45b2e007453f423&quot; accountid=&quot;5f9abc1eb45b2e007453f423&quot; rel=&quot;noreferrer&quot;&gt;John Malconian&lt;/a&gt;, do you think it&apos;s worth trying to figure out what may have changed in ui-users or platform-core to trigger this now, or are we content with &quot;Node is a RAM hog; let&apos;s just feed the beast and move on to more interesting problems&quot;? &lt;/p&gt;</comment>
                                                            <comment id="199086" author="5f9abc1eb45b2e007453f423" created="Tue, 9 Jun 2020 17:28:35 +0000"  >&lt;p&gt;Ok.  So I merged a change to the Node-based pipeline that explicitly sets NODE_OPTIONS=&quot;--max-old-space-size=3072&quot; prior to unit tests running.   Tested against a branch of ui-users and it seems to do the trick.   I&apos;m not sure why unit tests require more memory now, but apparently they do.    Not sure it&apos;s worth spending a ton of time figuring out why.  It seems to be the Way of the Node.   I also doubled max-old-space-size for platform-core builds and that issue seems to be resolved.  &lt;/p&gt;

&lt;p&gt;I think we can close this issue out,  but let&apos;s keep it open for another day so that we can verify that the issue has really been resolved. &lt;/p&gt;</comment>
                                                            <comment id="199088" author="615afd1cd9820f0070a09ef0" created="Tue, 9 Jun 2020 17:30:52 +0000"  >&lt;p&gt;We&apos;ve been building platform-complete with 4096MB but others with &quot;only&quot; 2048MB, Now we allocate 4096 to other builds as well. &lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10155"><![CDATA[FOLIO DevOps]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i0196f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1878">DevOps: sprint 90</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10044" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 9 Jun 2020 11:28:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>