<!-- 
RSS generated by JIRA (1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d) at Thu Feb 08 23:19:54 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>FOLIO Jira</title>
    <link>https://folio-org.atlassian.net</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>1001.0.0-SNAPSHOT</version>
        <build-number>100246</build-number>
        <build-date>07-02-2024</build-date>
    </build-info>

<item>
            <title>[FOLIO-2334] Spike: Investigate using JVM features to manage container memory</title>
                <link>https://folio-org.atlassian.net/browse/FOLIO-2334</link>
                <project id="10290" key="FOLIO">FOLIO</project>
                    <description>&lt;p&gt;To enable the JVM to use defaults in a container environment, Java 10 introduced &quot;UseContainerSupport&quot; and that was backported to Java 8 (8u191+). Use that in conjunction with &quot;MaxRAMPercentage&quot;.&lt;/p&gt;

&lt;p&gt;The following steps to explore that:&lt;/p&gt;

&lt;p&gt;1. Roll 2 new base FOLIO JVM images (folioci/openjdk8-jre-alpine and folioci/openjdk8-jre)&lt;br/&gt;
2. Update a Dockerfile in two test modules (which use each of those)&lt;br/&gt;
3. Test launching containers with legacy settings in the LaunchDescriptor&lt;br/&gt;
4. Test launching containers with new settings in the LaunchDescriptor&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Outcome&lt;/b&gt;:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;update documentation on dev.folio.org of what memory setting should be used in individual MDs and how the Dockerfile should be updated to ulitize the new base image that includes updated JVM that support those settings.&lt;/li&gt;
	&lt;li&gt;Update 
    &lt;span class=&quot;jira-issue-macro&quot; data-jira-key=&quot;FOLIO-2315&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/FOLIO-2315&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Re-assess the memory allocation in default LaunchDescriptor of each back-end ModuleDescriptor&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10307?size=medium&quot; /&gt;
            FOLIO-2315
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-complete jira-macro-single-issue-export-pdf&quot;&gt;Blocked&lt;/span&gt;
            &lt;/span&gt;
 and linked issues to link to this documentation&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="81478">FOLIO-2334</key>
            <summary>Spike: Investigate using JVM features to manage container memory</summary>
                <type id="10003" iconUrl="https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium">Task</type>
                                            <priority id="10001" iconUrl="https://dev.folio.org/assets/jira-priority/jira-p2.svg">P2</priority>
                        <status id="6" iconUrl="https://folio-org.atlassian.net/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10003">Done</resolution>
                                                        <assignee accountid="61cd0ca0bce5e00069e98be7">David Crossley</assignee>
                                                                <reporter accountid="61cd0ca0bce5e00069e98be7">David Crossley</reporter>
                                    <labels>
                            <label>devops</label>
                            <label>platform-backlog</label>
                    </labels>
                <created>Thu, 31 Oct 2019 07:22:46 +0000</created>
                <updated>Wed, 3 Jun 2020 16:40:14 +0000</updated>
                            <resolved>Mon, 18 Nov 2019 14:41:56 +0000</resolved>
                                                                        <due></due>
                            <votes>0</votes>
                                    <watches>7</watches>
                                                                <comments>
                                                            <comment id="193902" author="61cd0ca0bce5e00069e98be7" created="Fri, 15 Nov 2019 13:48:32 +0000"  >&lt;p&gt;Results of spike investigation:&lt;/p&gt;

&lt;p&gt;Built new docker image folioci/alpine-jre-openjdk8:latest&lt;br/&gt;
based on alpine:latest currently &quot;3.10.3&quot;&lt;br/&gt;
and openjdk version &quot;1.8.0_222&quot;&lt;br/&gt;
The &quot;UseContainerSupport&quot; setting is true by default.&lt;/p&gt;

&lt;p&gt;Verified with various modules (and local vagrant VM), rebuilding each module within it and launching their new local docker container.&lt;/p&gt;

&lt;p&gt;JAVA_OPTIONS: &quot;-XX:MaxRAMPercentage=66.0 -XX:+PrintFlagsFinal&quot;&lt;/p&gt;

&lt;p&gt;(Note that the value must be &quot;66.0&quot; not &quot;66&quot;.)&lt;/p&gt;

&lt;p&gt;Verified that that sets &quot;MaxHeapSize&quot; to be 66% of the &quot;Memory&quot; value specified in the module default LaunchDescriptor.&lt;br/&gt;
That leaves the remainder of the memory allocation for other non-heap needs.&lt;/p&gt;

&lt;p&gt;So some modules may need to reassess their LaunchDescriptor &quot;Memory&quot; setting.&lt;/p&gt;

&lt;p&gt;Also verified that existing LD settings will still operate as is now (probably better with this newer underlying docker image).&lt;/p&gt;

&lt;p&gt;Some notes for the roll-out process:&lt;/p&gt;

&lt;p&gt;Each module can utilise the new docker image, and adjust to the new LD settings. Then a follow-up job is to adjust their folio-ansible group_vars settings, which over-ride these. As shown above, the current group_vars settings will still operate until then.&lt;/p&gt;</comment>
                                                            <comment id="193904" author="61cd0ca0bce5e00069e98be7" created="Fri, 15 Nov 2019 13:49:00 +0000"  >&lt;p&gt;Some useful resources discovered while investigating:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://medium.com/adorsys/usecontainersupport-to-the-rescue-e77d6cfea712&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://medium.com/adorsys/usecontainersupport-to-the-rescue-e77d6cfea712&lt;/a&gt;&lt;br/&gt;
&quot;Please note that setting -Xmx and -Xms disables the automatic heap sizing.&quot;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://stackoverflow.com/a/55463537&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://stackoverflow.com/a/55463537&lt;/a&gt;&lt;br/&gt;
Andrei Pangin&lt;br/&gt;
and follow the links to his other answer&lt;br/&gt;
&lt;a href=&quot;https://stackoverflow.com/a/53624438&quot; class=&quot;external-link&quot; rel=&quot;nofollow noreferrer&quot;&gt;https://stackoverflow.com/a/53624438&lt;/a&gt;&lt;br/&gt;
and then to his beaut presentation.&lt;/p&gt;</comment>
                                                            <comment id="193907" author="61cd0ca0bce5e00069e98be7" created="Fri, 15 Nov 2019 13:49:17 +0000"  >&lt;p&gt;Still investigating some other FOLIO modules.&lt;/p&gt;</comment>
                                                            <comment id="193908" author="61cd0ca0bce5e00069e98be7" created="Mon, 18 Nov 2019 14:41:34 +0000"  >&lt;p&gt;See the results of this spike listed above, and this ticket&apos;s modified issue Description to specify the Outcome and next steps. Please wait for 
    &lt;span class=&quot;jira-issue-macro&quot; data-jira-key=&quot;FOLIO-2315&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/FOLIO-2315&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Re-assess the memory allocation in default LaunchDescriptor of each back-end ModuleDescriptor&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10307?size=medium&quot; /&gt;
            FOLIO-2315
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-complete jira-macro-single-issue-export-pdf&quot;&gt;Blocked&lt;/span&gt;
            &lt;/span&gt;
 (and all linked tickets) to be updated.&lt;/p&gt;</comment>
                                                            <comment id="193910" author="61cd0ca0bce5e00069e98be7" created="Fri, 29 Nov 2019 00:57:53 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5bfd62c75f77da6929583d6c&quot; class=&quot;user-hover&quot; rel=&quot;5bfd62c75f77da6929583d6c&quot; data-account-id=&quot;5bfd62c75f77da6929583d6c&quot; accountid=&quot;5bfd62c75f77da6929583d6c&quot; rel=&quot;noreferrer&quot;&gt;steve.osguthorpe&lt;/a&gt; asked on 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;ERM-638&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/ERM-638&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Use JVM features to manage container memory&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            ERM-638
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 to confirm that modules do benefit from the new memory settings.&lt;/p&gt;

&lt;p&gt;Further summary to accompany the results listed above:&lt;/p&gt;

&lt;p&gt;During testing we added &quot;-XX:+PrintFlagsFinal&quot; and investigated the docker logs. This shows that &quot;MaxHeapSize&quot; is correctly set to 66% of the container memory allocation.&lt;/p&gt;

&lt;p&gt;After rollout of the new base docker image and MaxRAMPercentage setting, we monitor the folio-snapshot-load reference environment each day.&lt;/p&gt;

&lt;p&gt;Every hour we assess &apos;docker stats&apos; for all modules. This shows that their total memory usage is remaining below that level. For a longer-running system there would probably be an increase, as non-heap memory is futher utilised and returned.&lt;/p&gt;

&lt;p&gt;We also grep each module&apos;s docker logs every hour to ensure no &quot;java.lang.OutOfMemoryError&quot;.&lt;/p&gt;

&lt;p&gt;So we believe that this usage of UseContainerSupport and MaxRAMPercentage does provide an appropriate way to manage the memory allocation. It reserves one-third of the container memory for non-heap use. Developers can adjust the total container memory via their LaunchDescriptor to raise it if they need more, but hopefully trim it down to provide a more lean system. The 66% reservation is an average estimate, and could also be adjusted for certain modules.&lt;/p&gt;

&lt;p&gt;(When the other devops people return from the Thanksgiving break, then they might be able to expand my answers.)&lt;/p&gt;</comment>
                                                            <comment id="193911" author="61cd0ca0bce5e00069e98be7" created="Fri, 29 Nov 2019 00:58:12 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5bfd62c75f77da6929583d6c&quot; class=&quot;user-hover&quot; rel=&quot;5bfd62c75f77da6929583d6c&quot; data-account-id=&quot;5bfd62c75f77da6929583d6c&quot; accountid=&quot;5bfd62c75f77da6929583d6c&quot; rel=&quot;noreferrer&quot;&gt;steve.osguthorpe&lt;/a&gt; also asked on 
    &lt;span class=&quot;jira-issue-macro resolved&quot; data-jira-key=&quot;ERM-638&quot; &gt;
                &lt;a href=&quot;https://folio-org.atlassian.net/browse/ERM-638&quot; class=&quot;jira-issue-macro-key issue-link&quot;  title=&quot;Use JVM features to manage container memory&quot; &gt;
            &lt;img class=&quot;icon&quot; src=&quot;https://folio-org.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10322?size=medium&quot; /&gt;
            ERM-638
        &lt;/a&gt;
                                                    &lt;span class=&quot;aui-lozenge aui-lozenge-subtle aui-lozenge-success jira-macro-single-issue-export-pdf&quot;&gt;Closed&lt;/span&gt;
            &lt;/span&gt;
 &quot;whether the CGroup is correctly available&quot;. Could you please explain how to verify that.&lt;/p&gt;</comment>
                                                            <comment id="193914" author="5f9abc1eb45b2e007453f423" created="Mon, 2 Dec 2019 16:32:52 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5bfd62c75f77da6929583d6c&quot; class=&quot;user-hover&quot; rel=&quot;5bfd62c75f77da6929583d6c&quot; data-account-id=&quot;5bfd62c75f77da6929583d6c&quot; accountid=&quot;5bfd62c75f77da6929583d6c&quot; rel=&quot;noreferrer&quot;&gt;steve.osguthorpe&lt;/a&gt; is most likely referring to this error that occurs in the container log when using the old fabric8-based container:&lt;/p&gt;

&lt;p&gt;cat: can&apos;t open &apos;/sys/fs/cgroup/memory/memory.memsw.limit_in_bytes&apos;: No such file or directory&lt;/p&gt;

&lt;p&gt;This specific error is generated when cgroup swap auditing is not enabled in the host kernel.  This is default for recent Debian/Ubuntu kernels (it can be enabled via a kernel parameter).   The old fabric8 base image had hooks for setting this control group, because it was originally written for Redhat/CentOS where cgroup swap auditing is enabled by default.   At any rate,   we are no longer using a base image based on fabric8 so this error should no longer appear.   We do not set any container limitations on swap anyway - just RAM.    Control group auditing for RAM is enforced. &lt;/p&gt;</comment>
                                                            <comment id="193917" author="5bfd62c75f77da6929583d6c" created="Tue, 3 Dec 2019 10:00:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=61cd0ca0bce5e00069e98be7&quot; class=&quot;user-hover&quot; rel=&quot;61cd0ca0bce5e00069e98be7&quot; data-account-id=&quot;61cd0ca0bce5e00069e98be7&quot; accountid=&quot;61cd0ca0bce5e00069e98be7&quot; rel=&quot;noreferrer&quot;&gt;David Crossley&lt;/a&gt; - Thanks for the confirmation.&lt;br/&gt;
&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5f9abc1eb45b2e007453f423&quot; class=&quot;user-hover&quot; rel=&quot;5f9abc1eb45b2e007453f423&quot; data-account-id=&quot;5f9abc1eb45b2e007453f423&quot; accountid=&quot;5f9abc1eb45b2e007453f423&quot; rel=&quot;noreferrer&quot;&gt;John Malconian&lt;/a&gt; - Thank you too for the expansion and yes it is.&lt;br/&gt;
I do however have another question and it&apos;s to do with the none-heap settings, Metaspace (which used to be called permgen) is always set to a high number (the maximum), which basically removes any upper limits. SO even though you&apos;ve specified max ram that&apos;s only for Heap. Are there any plans/recommendations for us to incorporate that setting? Should as a developer just, set my max ram percentage for value in the memory section of the descriptor? How do external ops teams know that at 800Mb we can only have 50% for heap, but at 3GB you can allocate up to 90%?&lt;/p&gt;</comment>
                                                            <comment id="193921" author="5c706fbb47a54a6728e59df2" created="Tue, 3 Dec 2019 14:03:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5bfd62c75f77da6929583d6c&quot; class=&quot;user-hover&quot; rel=&quot;5bfd62c75f77da6929583d6c&quot; data-account-id=&quot;5bfd62c75f77da6929583d6c&quot; accountid=&quot;5bfd62c75f77da6929583d6c&quot; rel=&quot;noreferrer&quot;&gt;steve.osguthorpe&lt;/a&gt; there are two keys in the module descriptor that can communicate that kind of information to the external operators:&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;launchDescriptor/dockerArgs/HostConfig/Memory&lt;/tt&gt;: Total memory allocation for the container&lt;br/&gt;
&lt;tt&gt;launchDescriptor/env&lt;/tt&gt;: You can set the &lt;tt&gt;JAVA_OPTIONS&lt;/tt&gt; environment variable as you see fit. For example, you could use &lt;tt&gt;MinRAMPercentage&lt;/tt&gt; instead of &lt;tt&gt;MaxRAMPercentage&lt;/tt&gt; if you felt that was more appropriate.&lt;/p&gt;

&lt;p&gt;One rule of thumb proposed by &lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5cf6c546b87c300f36eb7b9a&quot; class=&quot;user-hover&quot; rel=&quot;5cf6c546b87c300f36eb7b9a&quot; data-account-id=&quot;5cf6c546b87c300f36eb7b9a&quot; accountid=&quot;5cf6c546b87c300f36eb7b9a&quot; rel=&quot;noreferrer&quot;&gt;Craig McNally&lt;/a&gt;, which seems sensible to me, is that you set the size for a single tenant with a standard workload, whatever that means to you, with the expectation that the operator will scale containers horizontally to meet higher demand. Like all rules of thumb, it probably won&apos;t work for every circumstance, but it seems a reasonable starting point.&lt;/p&gt;

&lt;p&gt;Beyond that, you can of course communicate specific resource needs in the module README.&lt;/p&gt;

&lt;p&gt;Does that address your concerns?&lt;/p&gt;</comment>
                                                            <comment id="193926" author="5bfd62c75f77da6929583d6c" created="Tue, 3 Dec 2019 18:11:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://folio-org.atlassian.net/secure/ViewProfile.jspa?accountId=5c706fbb47a54a6728e59df2&quot; class=&quot;user-hover&quot; rel=&quot;5c706fbb47a54a6728e59df2&quot; data-account-id=&quot;5c706fbb47a54a6728e59df2&quot; accountid=&quot;5c706fbb47a54a6728e59df2&quot; rel=&quot;noreferrer&quot;&gt;Wayne Schneider&lt;/a&gt; Thanks. That seems completely reasonable to me.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10000">
                    <name>Blocks</name>
                                            <outwardlinks description="blocks">
                                        <issuelink>
            <issuekey id="63024">MODSOURCE-80</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="57810">MODSOURMAN-223</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="81465">FOLIO-2358</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="79826">FOLIO-2315</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10003">
                    <name>Relates</name>
                                            <outwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="81473">FOLIO-2367</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="relates to">
                                        <issuelink>
            <issuekey id="81114">FOLIO-1729</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="81250">FOLIO-2185</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10000" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummarycf">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10057" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Development Team</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10144"><![CDATA[Core: Platform]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10019" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i00c9j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10020" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1606">CP: sprint 76</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10044" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10024" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>[CHART] Date of First Response</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 2 Dec 2019 16:32:52 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10025" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>[CHART] Time in Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>