[FOLIO-1729] Use container memory limits to manage memory in reference environments Created: 23/Jan/19  Updated: 11/Aug/20  Resolved: 21/Oct/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P2
Reporter: Wayne Schneider Assignee: Unassigned
Resolution: Done Votes: 0
Labels: ci, platform-backlog, sysops_mgt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by FOLIO-2242 Update group_vars files in folio-ansi... Closed
Duplicate
is duplicated by FOLIO-1032 Error setting max JVM heap size based... Closed
Relates
relates to FOLIO-1772 Upgrade to OpenJDK 11 Closed
relates to FOLIO-2334 Spike: Investigate using JVM features... Closed
relates to FOLIO-2185 SPIKE: how to maintain resource deplo... Closed
relates to FOLIO-2315 Re-assess the memory allocation in de... Blocked
relates to FOLIO-1696 SPIKE: provide operational "metadata"... Closed
Sprint: CP: ready for planning
Development Team: Core: Platform

 Description   

Right now, backend modules launched as containers by Okapi have no memory limits imposed by default. In addition, for Java-based modules, in a container, the JVM by default sees all system memory as available to it, regardless of any container memory limit.

We are currently managing this by setting an environment variable JAVA_OPTIONS=-Xmx256m (or whatever is required according to the developer) for all Java-based modules. This solution works OK, but it requires that the DevOps team configure each module individually, and the memory requirements of the module are not documented except in the deployment variables file in folio-ansible.

A better solution might be:

  • Update the launch descriptors in each module's module descriptor template to set the HostConfig/Memory key in dockerArgs to the appropriate default as determined by the module development team. This will limit each container's memory usage. It could still be overridden in a deployment descriptor
  • Update the base Java image for FOLIO Docker containers to use the -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap switches as part of the Java command line to allow the JVM to see the container memory limit


 Comments   
Comment by Jakub Skoczen [ 28/Jan/19 ]

Wayne Schneider Is this still something we want to do? What's the reason/benefit?

Comment by Wayne Schneider [ 28/Jan/19 ]

Jakub Skoczen: I see the benefits this way:

  1. Current memory requirements for modules are set in a variables file setting in folio-ansible. Each module that is added to the system needs to be added to the variables file with a memory setting. This means that a system built from dependency resolution may include modules that are not already configured, and so the memory consumption of the module will not be limited by container boundaries.
  2. The current method of restricting memory use is Java-only (using the -Xmx Java command line switch). Using container memory limits would be more language-agnostic.
  3. Currently, few (or possibly no) modules document their memory requirements. The only way a sysop can find this out is indirectly, by examining the variables file in folio-ansible. This would put the burden of documenting memory requirements on the development team for the module, where it at least arguably belongs.
Comment by Jakub Skoczen [ 11/Mar/19 ]

Wayne Schneider My only concern here is that this involves changing the method of constraining memory (JVM-specific to container-specific) which could have unintended consequences and definitely needs testing. Would it make sense to address the issue of "hardcoding" JVM params first and seperately e.g by passing Launch parameters down to the container and then down to the JVM?

Comment by Oleksii Popov [ 04/Apr/19 ]

Requires final agreement.
Key points:

  • update JDK
  • memory config
Comment by Wayne Schneider [ 04/Apr/19 ]

Possible umbrella issue for module metadata: FOLIO-1696 Closed

Comment by Jakub Skoczen [ 16/Sep/19 ]

Wayne Schneider David Crossley Guys, is this still a todo or has this been closed through the work done wih LDs last week?

Comment by Wayne Schneider [ 16/Sep/19 ]

I believe this is closed through the LD work – we could close now (because a solution has been identified), or wait until FOLIO-2242 Closed (the final cleanup task of that project) is complete.

Comment by Jakub Skoczen [ 21/Oct/19 ]

Completed based on the previous comment – re-open if not the case.

Generated at Thu Feb 08 23:15:28 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.