[FOLIO-3480] folioci/alpine-jre-openjdk11:1.3.1 ZipException: Corrupt GZIP trailer Created: 25/Apr/22  Updated: 09/Jun/22  Resolved: 29/Apr/22

Status: Closed
Project: FOLIO
Components: Continuous Integration
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: Serhii_Nosko Assignee: David Crossley
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks MODSOURMAN-782 Resolve Corrupt GZIP trailer exceptio... Closed
Relates
relates to FOLIO-3483 Rebuild all alpine-jre-openjdk11 base... Closed
relates to FOLIO-3500 Remove folio-java-docker workarounds ... Closed
relates to FOLIO-3484 Rebuild all released alpine-jre-openj... Closed
relates to FOLIO-3487 folioci/alpine-jre-openjdk11: apk upg... Closed
relates to MODSOURCE-498 Rebuild container fixing zlib ZipExce... Closed
relates to FOLIO-3463 Rebuild/upgrade folioci/alpine-jre-op... Closed
Sprint: DevOps Sprint 138
Development Team: FOLIO DevOps
RCA Group: Related dependency upgrade

 Description   

When using the latest Vagrant snapshot boxes I started to get java.util.zip.ZipException: Corrupt GZIP trailer exception during processing of Kafka messages. I observed it most often in module mod-source-record-manager, also in mod-inventory, mod-quick-marc.
For older snapshot boxes such as 'folio/snapshot' version '1.0.0-20220411.7616' I have not observed it.

Workaround for mod-source-record-manager is to downgrade the alpine-jre-openjdk11 version from 1.3.1 to 1.3.0 in Dockerfile:

FROM folioci/alpine-jre-openjdk11:1.3.0

Note: I'm using Windows, on Linux it seems not to be reproducible.

Full log:

Caused by: org.apache.kafka.common.KafkaException: Error checking for remaining bytes after reading batch
at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.ensureNoneRemaining(DefaultRecordBatch.java:654) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:611) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:575) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1631) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1666) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1507) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:733) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:684) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1304) ~[mod-source-record-manager.jar:?]
... 8 more
Caused by: java.util.zip.ZipException: Corrupt GZIP trailer
at java.util.zip.GZIPInputStream.readTrailer(Unknown Source) ~[?:?]
at java.util.zip.GZIPInputStream.read(Unknown Source) ~[?:?]
at java.io.BufferedInputStream.fill(Unknown Source) ~[?:?]
at java.io.BufferedInputStream.read(Unknown Source) ~[?:?]
at java.io.FilterInputStream.read(Unknown Source) ~[?:?]


 Comments   
Comment by Julian Ladisch [ 26/Apr/22 ]

All changes in the folioci/alpine-jre-openjdk11 container when comparing 1.3.0 and 1.3.1:

busybox-1.34.1-r3 x86_64 {busybox} (GPL-2.0-only) [installed]
busybox-1.34.1-r5 x86_64 {busybox} (GPL-2.0-only) [installed]
ca-certificates-bundle-20191127-r7 x86_64 {ca-certificates} (MPL-2.0 AND MIT) [installed]
ca-certificates-bundle-20211220-r0 x86_64 {ca-certificates} (MPL-2.0 AND MIT) [installed]
libcrypto1.1-1.1.1l-r7 x86_64 {openssl} (OpenSSL) [installed]
libcrypto1.1-1.1.1n-r0 x86_64 {openssl} (OpenSSL) [installed]
libretls-3.3.4-r2 x86_64 {libretls} (ISC AND (BSD-3-Clause OR MIT)) [installed]
libretls-3.3.4-r3 x86_64 {libretls} (ISC AND (BSD-3-Clause OR MIT)) [installed]
libssl1.1-1.1.1l-r7 x86_64 {openssl} (OpenSSL) [installed]
libssl1.1-1.1.1n-r0 x86_64 {openssl} (OpenSSL) [installed]
ssl_client-1.34.1-r3 x86_64 {busybox} (GPL-2.0-only) [installed]
ssl_client-1.34.1-r5 x86_64 {busybox} (GPL-2.0-only) [installed]
tzdata-2021e-r0 x86_64 {tzdata} (Public-Domain) [installed]
tzdata-2022a-r0 x86_64 {tzdata} (Public-Domain) [installed]
zlib-1.2.11-r3 x86_64 {zlib} (Zlib) [installed]
zlib-1.2.12-r0 x86_64 {zlib} (Zlib) [installed]
JAVA_VERSION jdk-11.0.14+9
JAVA_VERSION jdk-11.0.14.1+1
Comment by Serhii_Nosko [ 26/Apr/22 ]

Thanks Julian Ladisch, can we try to remove these 2 differences from new image and we will try to import again?

zlib-1.2.11-r3 x86_64 {zlib} (Zlib) [installed]
zlib-1.2.12-r0 x86_64 {zlib} (Zlib) [installed]

zlib its basically data-compression library and we have ZipException in our modules, so it could help

Comment by Jakub Skoczen [ 26/Apr/22 ]

Serhii_Nosko Julian Ladisch I don't see how zlib should be related as the exception is from the built-in Java Gzip library.

Comment by Serhii_Nosko [ 26/Apr/22 ]

Jakub Skoczen yes, its strange but I don't see any other libraries that were updated in 1.3.1 that can influence more that zlib.
Because when I revert back image to 1.3.0 - all works good without java ZipException.
Also I see in github some other issues with zlib that cause java ZipException.
Here are couple of them:
https://github.com/madler/zlib/issues/613
https://github.com/madler/zlib/issues/305

Comment by Julian Ladisch [ 26/Apr/22 ]

JRE can be configured to either use a system zlib or a bundled zlib:
https://bugs.openjdk.java.net/browse/JDK-8249963

Comment by Julian Ladisch [ 26/Apr/22 ]

I've created mod-source-record-manager:3.4.0-SNAPSHOT.606-zlib-1.2.11-r3 with this Dockerfile:

FROM folioci/mod-source-record-manager:3.4.0-SNAPSHOT.606
USER root
RUN echo 'https://dl-cdn.alpinelinux.org/alpine/v3.11/main' > /etc/apk/repositories
RUN apk --no-cache add zlib=1.2.11-r3
USER folio

It's available from https://hub.docker.com/r/julianladisch/mod-source-record-manager/tags

Comment by Julian Ladisch [ 26/Apr/22 ]

The other way round to check the other possibility:

I've created mod-source-record-manager:3.4.0-SNAPSHOT.606-JAVA_VERSION-jdk-11.0.14_9 with the previous JAVA_VERSION but latest Alpine packages with this Dockerfile and the jar from commit 6a065eb4:

FROM folioci/alpine-jre-openjdk11:1.3.0

USER root
RUN apk --no-cache upgrade
USER folio

ENV VERTICLE_FILE mod-source-record-manager-server-fat.jar

# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles

# Copy your fat jar to the container
COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}

# Expose this port locally in the container.
EXPOSE 8081
Comment by Serhii_Nosko [ 26/Apr/22 ]

Julian Ladisch Basically I see this ZipException at least for mod-source-record-manager, mod-source-record-storage, mod-inventory and mod-quick-mark, and today even saw in mod-search, so to make it working I need redeploy all these modules locally with 1.3.0 image and after this local vagrant on Windows can work for data import(if any other modules will not throw this exception), can we build some test vagrant box with all modules built on the same image and I will verify behaviour on Windows?

Comment by Julian Ladisch [ 26/Apr/22 ]

Serhii_Nosko: Can you replace mod-source-record-manager in your Vagrant box by julianladisch/mod-source-record-manager:3.4.0-SNAPSHOT.606-zlib-1.2.11-r3 and report whether this actually fixes the issue?

If yes I can downgrade zlib in folioci/alpine-jre-openjdk11 and I can re-create all affected docker containers.

Comment by Julian Ladisch [ 26/Apr/22 ]

It is very likely that it is this issue:
https://github.com/madler/zlib/issues/613 "zlib v1.2.12 causes java.util.zip.ZipException"
It has been fixed on the development branch: https://github.com/madler/zlib/commit/ec3df00224d4b396e2ac6586ab5d25f673caa4c2
But it hasn't released yet.

Comment by Serhii_Nosko [ 26/Apr/22 ]

Julian Ladisch I run old vagrant with 1.3.0 image(to be sure that other modules are working) and redeploy my local srm with 1.3.1 image - ZipException occured.
After this I redeployed the same local srm from base image

julianladisch/mod-source-record-manager:3.4.0-SNAPSHOT.606-zlib-1.2.11-r3

After this I have not seen ZipException any more, and imports were successful, so I think we can use it in building our new vagrant boxes, thank you a lot

Comment by Julian Ladisch [ 26/Apr/22 ]

Serhii_Nosko: Thanks. We are going to create a new folioci/alpine-jre-openjdk11 container with the zlib downgrade: https://github.com/folio-org/folio-tools/pull/214

Comment by Julian Ladisch [ 26/Apr/22 ]

David Crossley, John Malconian: Please merge https://github.com/folio-org/folio-tools/pull/214 and release a new folioci/alpine-jre-openjdk11 container.

Comment by Julian Ladisch [ 26/Apr/22 ]

According to upstream bug report https://github.com/madler/zlib/issues/613 the root cause is the handling of higher order bytes that affects x86-64 but not x86 32-bit, arm 32-bit and arm 64-bit.

Comment by David Crossley [ 27/Apr/22 ]

This is now merged, built, verified, and deployed as v1.3.2

Thanks for your effort Julian Ladisch

Attention: Serhii_Nosko

Comment by Julian Ladisch [ 27/Apr/22 ]

Thanks!

Alpine has published zlib 1.2.12-r1 with the fix, but only for edge (= unstable development branch), not for the stable branch 3.15:

Comment by Julian Ladisch [ 28/Apr/22 ]

The stable branch 3.15 finally got zlib 1.2.12-r1 with the fix:

https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33688

Comment by David Crossley [ 28/Apr/22 ]

Re-opened this ticket. I will re-build.

Comment by David Crossley [ 29/Apr/22 ]

Re-built, verified, and pushed as 1.3.3 and latest.

 

Generated at Thu Feb 08 23:28:27 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.