Skip to end of banner
Go to start of banner

Log4Shell

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 50 Next »

FOLIO is affected by the Log4Shell security issue (CVE-2021-44228 and CVE-2021-45046) and a denial of service attack issue (CVE-2021-45105) in the log4j-core library used by many FOLIO backend modules and FOLIO's Okapi. Advisories of the Log4j project: https://logging.apache.org/log4j/2.x/security.html

This page gives advice for FOLIO installations and FOLIO developers to mitigate the risks of these issues.

Please contribute, this is a wiki page!

Sysops/Devops

Okapi upgrade

All existing installations should upgrade to Okapi v4.11.1.

This version works for all releases (Lotus, Kiwi, Juniper, Iris), there were no breaking changes.

No environment variables or system variables are needed to fix Okapi.

Okapi protects all modules where it proxies requests to (the mod-* modules) from the denial-of-service attack of CVE-2021-45105 (infinite recursion in MDC lookup evaluation) by filtering malicious values in the fields used for MDC (OKAPI-1058), therefore those modules don't need log4j 2.17.0 and can safely continue to use log4j 2.16.0.

Okapi doesn't protect the edge-* modules, edge-* modules that might be affected need to upgrade to log4j 2.17.0.

Hot-fix preview

Sysops that don't want to wait for the official hot-fix release can install the module versions of the platform-complete branch:

https://github.com/julianladisch/platform-complete/actions/workflows/log4shell-scan.yml scans these branches for vulnerable log4j versions. Click on the release and on "Run cat result.txt" to see the results.  The scan runs daily.

Configuration variables for back-end modules

Many back-end modules are affected by the Log4Shell issue, until new jar files and new docker containers with a fixed version are ready the existing back-end modules should be reconfigured with an environment variable that disables the flaw for most cases in log4j:

  • LOG4J_FORMAT_MSG_NO_LOOKUPS=true
  • Append -Dlog4j2.formatMsgNoLookups=true to the JAVA_OPTIONS variable

Pick one of the two options.

Example for the second option: If the existing configuration has JAVA_OPTIONS="-XX:MaxRAMPercentage=66.0" then the new configuration should be
JAVA_OPTIONS="-XX:MaxRAMPercentage=66.0 -Dlog4j2.formatMsgNoLookups=true"

SQL query to do this, posted by Lucy Menon on #sys-ops Slack channel, assuming that all modules in use already have a JAVA_OPTIONS env entry:

okapi=# with foo as (select ('{descriptor,env,'||index-1||',value}')::text[] as path, to_jsonb(env->>'value'||' -Dlog4j2.formatMsgNoLookups=true') as nval, json->'instId' as instId from deployments, jsonb_array_elements(json->'descriptor'->'env') with ordinality arr(env, index) where env->>'name' = 'JAVA_OPTIONS') update deployments set json = jsonb_set(deployments.json, foo.path, foo.nval) from foo where deployments.json->'instId' = foo.instId;
okapi=# with foo as (select ('{launchDescriptor,env,'||index-1||',value}')::text[] as path, to_jsonb(env->>'value'||' -Dlog4j2.formatMsgNoLookups=true') as nval, modulejson->'id' as modid from modules, jsonb_array_elements(modulejson->'launchDescriptor'->'env') with ordinality arr(env, index) where env->>'name' = 'JAVA_OPTIONS') update modules set modulejson = jsonb_set(modules.modulejson, foo.path, foo.nval) from  foo where modules.modulejson->'id' = foo.modid;

If you use Okapi to deploy the modules you may use this method to set the LOG4J_FORMAT_MSG_NO_LOOKUPS variable and redeploy them:

curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d '{"name":"LOG4J_FORMAT_MSG_NO_LOOKUPS","value":"true"}' http://localhost:9130/_/env
systemctl restart okapi

Is using configuration variables secure?

Not completely. It only limits exposure while leaving some attack vectors open. Using the configuration variables is a temporary measure for the time until patched FOLIO modules are available. Please upgrade to patched modules as soon as possible.

From:  https://logging.apache.org/log4j/2.x/security.html 

History

Older (discredited) mitigation measures

This page previously mentioned other mitigation measures, but we discovered that these measures only limit exposure while leaving some attack vectors open.

Other insufficient mitigation measures are: setting system property log4j2.formatMsgNoLookups or environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true for releases >= 2.10, or modifying the logging configuration to disable message lookups with %m{nolookups}, %msg{nolookups} or %message{nolookups} for releases >= 2.7 and <= 2.14.1.

The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.

The safest thing to do is to upgrade Log4j to a safe version, or remove the JndiLookup class from the log4j-core jar.

Are Kafka and Zookeeper affected?

No, because FOLIO doesn't set the JMS configuration TopicBindingName or TopicConnectionFactoryBindingName. For details see https://kafka.apache.org/cve-list and https://issues.apache.org/jira/browse/ZOOKEEPER-4423

Is ElasticSearch affected?

No, because FOLIO uses version 7.10. For details see https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Developers

Modules that use a vulnerable log4j version must be upgraded to >= 2.17.0, however, mod-* modules that already have upgraded to 2.16.0 don't need to upgrade to 2.17.0. Okapi protects all modules where it proxies requests to from the denial-of-service attack of CVE-2021-45105 (infinite recursion in MDC lookup evaluation) by filtering malicious values in the fields used for MDC (OKAPI-1058), therefore those modules don't need log4j 2.17.0 and can safely continue to use log4j 2.16.0.

Each edge-* module must upgrade to log4j >= 2.17.0 unless you know that it doesn't use MDC lookups.

Please do the work in this order: Juniper, Kiwi, Iris. Work on edge module before back-end modules.

After all mod-* and edge-* modules have been upgraded to log4j >= 2.16.0 upgrade edge-* modules with log4j 2.16.0 to log4j 2.17.0 unless they don't use MDC lookups.

https://github.com/julianladisch/platform-complete/actions/workflows/log4shell-scan.yml scans the platfrom-complete branches R1-2021, R2-2021 and R3-2021 for vulnerable log4j versions. Click on the release and on "Run cat result.txt" to see the results. The scan runs every two hours.

RMB based modules

Modules based on Raml Module Builder (RMB) should upgrade to a fixed version:

RMB based modules should remove any direct log4j dependency from their pom.xml and use the log4j version provided by RMB if they want the log4j version being automatically updated when updating RMB.

Spring Boot based modules

For modules based off of Spring Boot are vulnerable as spring-boot-starter-log4j2 includes the vulnerable version of log4j.  Spring Boot has updated their version to 2.17.0 (as of 12/23/2021), however, this is still vulnerable and will likely not be updated till their next release.

In the meantime, you can override the variable controlling Spring Boot's log4j version through the log4j2.version property (per their instructions for Maven, as FOLIO base uses their parent POM):

<properties>
  ...
  <log4j2.version>2.17.1</log4j2.version>
</properties>

Maven based modules

For modules that use maven run

mvn dependency:tree -Dincludes=org.apache.logging.log4j

or – for verbose output –

mvn dependency:tree -Dincludes=org.apache.logging.log4j -Dverbose

to see whether log4j is used and which version. Example output:

[INFO] org.folio:mod-users:jar:18.2.0-SNAPSHOT
[INFO] +- org.folio:domain-models-runtime:jar:33.2.0-SNAPSHOT:compile
[INFO] |  +- org.apache.logging.log4j:log4j-core:jar:2.14.1:compile
[INFO] |  \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compile
[INFO] \- org.folio:folio-service-tools-test:jar:1.7.1:test
[INFO]    \- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.14.1:compile

This shows log4j 2.14.1 comes from domain-models-runtime (= RMB) so upgrading RMB resolves it.

Non-RMB modules might use other ways to update log4j.

If a module directly sets the version of log4j it should use the a <dependencyManagement> entry with log4j-bom to keep all log4j components (log4j-api, log4j-core, ...) in sync:

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-bom</artifactId>
        <version>2.17.1</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Example: https://github.com/folio-org/mod-authtoken/blob/v2.9.1/pom.xml#L60-L100

If not using the log4j-bom be sure to specify 2.17.1 in all of your log4j dependencies, check with mvn dependency:tree.

Should developers set environment or system variables in modules?

No, updating to log4j >= 2.17.1 is sufficient. LOG4J_FORMAT_MSG_NO_LOOKUPS=true or -Dlog4j2.formatMsgNoLookups=true should only be used by sysops for unpatched modules as a temporary fix. Don't add them to the ModuleDescriptors or LaunchDescriptors a module ships with. For details see section "Is using configuration variables secure?" above.

Should developers remove JndiLookup.class from modules?

No, updating to log4j >= 2.17.1 is sufficient. 

For the time while waiting for a patched module the advice from Apache (see above) suggest that the safest thing to do may be to remove JndiLookup.class from the classpath (usually from a module's fat jar).  However, the effort to remove the class and release this stripped module is as big as updating log4j and releasing a patched module.

Modules without log4j

Modules that don't use log4j don't need to do anything.

  • No labels