Log4Shell
FOLIO is affected by the Log4Shell security issue (CVE-2021-44228 and CVE-2021-45046) and a denial of service attack issue (CVE-2021-45105) in the log4j-core library used by many FOLIO backend modules and FOLIO's Okapi.
FOLIO is not affected by the remote code execution (RCE) attack using JDBC Appender (CVE-2021-44832).
Advisories of the Log4j project: https://logging.apache.org/log4j/2.x/security.html
This page gives advice for FOLIO installations and FOLIO developers to mitigate the risks of these issues.
Please contribute, this is a wiki page!
Sysops/Devops
Okapi upgrade
All existing installations should upgrade to Okapi v4.11.1.
This version works for all releases (Lotus, Kiwi, Juniper, Iris), there were no breaking changes.
No environment variables or system variables are needed to fix Okapi.
Okapi protects all modules where it proxies requests to (the mod-*
modules) from the denial-of-service attack of CVE-2021-45105 (infinite recursion in MDC lookup evaluation) by filtering malicious values in the fields used for MDC (OKAPI-1058), therefore those modules don't need log4j >= 2.17.0 and can safely continue to use log4j 2.16.0.
Okapi doesn't protect the edge-*
modules, edge-*
modules that might be affected need to upgrade to log4j >= 2.17.0.
Hot-fix preview
Sysops that don't want to wait for the official hot-fix release can install the module versions of the platform-complete branch:
- https://github.com/folio-org/platform-complete/blob/R1-2021/install.json for Iris (R1-2021)
- https://github.com/folio-org/platform-complete/blob/R2-2021/install.json for Juniper (R2-2021)
- https://github.com/folio-org/platform-complete/blob/R3-2021/install.json for Kiwi (R3-2021)
https://github.com/julianladisch/platform-complete/actions/workflows/log4shell-scan.yml scans these branches for vulnerable log4j versions. Click on the release and on "Run cat result.txt" to see the results. The scan runs daily. The result history shows that
- all platform-complete modules of Juniper (R2-2021) have been fixed since December 22, 2021
- all platform-complete modules of Kiwi (R3-2021) have been fixed since December 22, 2021
There are no plans to completely fix Iris (R1-2021) because Iris hasn't officially been declare to receive long term support and FOLIO supports only the latest two flower releases.
Configuration variables for back-end modules
Many back-end modules are affected by the Log4Shell issue, until new jar files and new docker containers with a fixed version are ready the existing back-end modules should be reconfigured with an environment variable that disables the flaw for most cases in log4j:
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
- Append
-Dlog4j2.formatMsgNoLookups=true
to theJAVA_OPTIONS
variable
Pick one of the two options.
Example for the second option: If the existing configuration has JAVA_OPTIONS="-XX:MaxRAMPercentage=66.0"
then the new configuration should beJAVA_OPTIONS="-XX:MaxRAMPercentage=66.0 -Dlog4j2.formatMsgNoLookups=true"
SQL query to do this, posted by Lucy Menon on #sys-ops Slack channel, assuming that all modules in use already have a JAVA_OPTIONS env entry:
okapi=# with foo as (select ('{descriptor,env,'||index-1||',value}')::text[] as path, to_jsonb(env->>'value'||' -Dlog4j2.formatMsgNoLookups=true') as nval, json->'instId' as instId from deployments, jsonb_array_elements(json->'descriptor'->'env') with ordinality arr(env, index) where env->>'name' = 'JAVA_OPTIONS') update deployments set json = jsonb_set(deployments.json, foo.path, foo.nval) from foo where deployments.json->'instId' = foo.instId; okapi=# with foo as (select ('{launchDescriptor,env,'||index-1||',value}')::text[] as path, to_jsonb(env->>'value'||' -Dlog4j2.formatMsgNoLookups=true') as nval, modulejson->'id' as modid from modules, jsonb_array_elements(modulejson->'launchDescriptor'->'env') with ordinality arr(env, index) where env->>'name' = 'JAVA_OPTIONS') update modules set modulejson = jsonb_set(modules.modulejson, foo.path, foo.nval) from foo where modules.modulejson->'id' = foo.modid;
If you use Okapi to deploy the modules you may use this method to set the LOG4J_FORMAT_MSG_NO_LOOKUPS
variable and redeploy them:
curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d '{"name":"LOG4J_FORMAT_MSG_NO_LOOKUPS","value":"true"}' http://localhost:9130/_/env
systemctl restart okapi
Is using configuration variables secure?
Not completely. It only limits exposure while leaving some attack vectors open. Using the configuration variables is a temporary measure for the time until patched FOLIO modules are available. Please upgrade to patched modules as soon as possible.
From: https://logging.apache.org/log4j/2.x/security.html
History
Older (discredited) mitigation measures
This page previously mentioned other mitigation measures, but we discovered that these measures only limit exposure while leaving some attack vectors open.
Other insufficient mitigation measures are: setting system property log4j2.formatMsgNoLookups or environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true for releases >= 2.10, or modifying the logging configuration to disable message lookups with %m{nolookups}, %msg{nolookups} or %message{nolookups} for releases >= 2.7 and <= 2.14.1.
The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.
The safest thing to do is to upgrade Log4j to a safe version, or remove the JndiLookup class from the log4j-core jar.
Are Kafka and Zookeeper affected?
No, because they use log4j 1.x and FOLIO doesn't set the JMS configuration TopicBindingName or TopicConnectionFactoryBindingName. For details see https://kafka.apache.org/cve-list and https://issues.apache.org/jira/browse/ZOOKEEPER-4423
Is ElasticSearch affected?
No, because FOLIO uses version 7.10. For details see https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476
Developers
Modules that use a vulnerable log4j version must be upgraded to >= 2.17.0, however, mod-*
modules that already have upgraded to 2.16.0 don't need to upgrade to 2.17.0. Okapi protects all modules where it proxies requests to from the denial-of-service attack of CVE-2021-45105 (infinite recursion in MDC lookup evaluation) by filtering malicious values in the fields used for MDC (OKAPI-1058), therefore those modules don't need log4j 2.17.0 and can safely continue to use log4j 2.16.0.
Modules don't need to be upgraded to log4j 2.17.1 because they are not affected by the remote code execution (RCE) attack using JDBC Appender (CVE-2021-44832) that can only be triggered through malicious configuration files; since these are hard-coded into each module, sysop permissions would be required to change them and, if an attacker has sysop permissions, they can do far more than exploit log4j. Sysop permissions are needed to inject a configuration file and/or set it via the -Dlog4j.configurationFile
system property; a search through the source code shows that no FOLIO module (re)configures log4j using code (programmatic API, setConfigLocation) and therefore cannot be (re)configured at runtime without sysop permissions.
Each edge-* module must upgrade to log4j >= 2.17.0 unless you know that it doesn't use MDC lookups.
Please do the work in this order: Juniper, Kiwi. Work on edge module before back-end modules.
After all mod-*
and edge-*
modules have been upgraded to log4j >= 2.16.0 upgrade edge-*
modules with log4j 2.16.0 to log4j 2.17.0 unless they don't use MDC lookups.
https://github.com/julianladisch/platform-complete/actions/workflows/log4shell-scan.yml scans the platfrom-complete branches R1-2021, R2-2021 and R3-2021 for vulnerable log4j versions. Click on the release and on "Run cat result.txt" to see the results. The scan runs daily.
RMB based modules
Modules based on Raml Module Builder (RMB) should upgrade to a fixed version:
RMB based modules should remove any direct log4j dependency from their pom.xml and use the log4j version provided by RMB if they want the log4j version being automatically updated when updating RMB.
Spring Boot based modules
For modules based off of Spring Boot are vulnerable as spring-boot-starter-log4j2
includes the vulnerable version of log4j
. Spring Boot has updated their version to 2.17.0 (as of 12/23/2021), therefore, pulling in the newest spring-boot-starter-log4j2
should be sufficient.
In you need to specifically override the version provided by Spring Boot, you can override the variable through the log4j2.version
property like so (per their instructions for Maven, as FOLIO base uses their parent POM):
<properties>
...
<log4j2.version>2.17.1</log4j2.version>
</properties>
Maven based modules
For modules that use maven run
mvn dependency:tree -Dincludes=org.apache.logging.log4j
or – for verbose output –
mvn dependency:tree -Dincludes=org.apache.logging.log4j -Dverbose
to see whether log4j is used and which version. Example output:
[INFO] org.folio:mod-users:jar:18.2.0-SNAPSHOT
[INFO] +- org.folio:domain-models-runtime:jar:33.2.0-SNAPSHOT:compile
[INFO] | +- org.apache.logging.log4j:log4j-core:jar:2.14.1:compile
[INFO] | \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compile
[INFO] \- org.folio:folio-service-tools-test:jar:1.7.1:test
[INFO] \- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.14.1:compile
This shows log4j 2.14.1 comes from domain-models-runtime (= RMB) so upgrading RMB resolves it.
Non-RMB modules might use other ways to update log4j.
If a module needs to directly set the version of log4j it should use the a <dependencyManagement>
entry with log4j-bom
to keep all log4j components (log4j-api, log4j-core, ...) in sync:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-bom</artifactId>
<version>2.17.1</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Example: https://github.com/folio-org/mod-authtoken/blob/v2.9.1/pom.xml#L60-L100
If not using the log4j-bom
be sure to specify the version in all of your log4j dependencies, check with mvn dependency:tree
.
Should developers set environment or system variables in modules?
No, updating to log4j >= 2.16.0 is sufficient. LOG4J_FORMAT_MSG_NO_LOOKUPS=true
or -Dlog4j2.formatMsgNoLookups=true
should only be used by sysops for unpatched modules as a temporary fix. Don't add them to the ModuleDescriptors or LaunchDescriptors a module ships with. For details see section "Is using configuration variables secure?" above.
Should developers remove JndiLookup.class from modules?
No, updating to log4j >= 2.16.0 is sufficient.
For the time while waiting for a patched module the advice from Apache (see above) suggest that the safest thing to do may be to remove JndiLookup.class from the classpath (usually from a module's fat jar). However, the effort to remove the class and release this stripped module is as big as updating log4j and releasing a patched module.
Modules without log4j
Modules that don't use log4j don't need to do anything.