[FOLIO-3480] folioci/alpine-jre-openjdk11:1.3.1 ZipException: Corrupt GZIP trailer Created: 25/Apr/22 Updated: 09/Jun/22 Resolved: 29/Apr/22 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | Continuous Integration |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P2 |
| Reporter: | Serhii_Nosko | Assignee: | David Crossley |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||
| Sprint: | DevOps Sprint 138 | ||||||||||||||||||||||||||||||||||||
| Development Team: | FOLIO DevOps | ||||||||||||||||||||||||||||||||||||
| RCA Group: | Related dependency upgrade | ||||||||||||||||||||||||||||||||||||
| Description |
|
When using the latest Vagrant snapshot boxes I started to get java.util.zip.ZipException: Corrupt GZIP trailer exception during processing of Kafka messages. I observed it most often in module mod-source-record-manager, also in mod-inventory, mod-quick-marc. Workaround for mod-source-record-manager is to downgrade the alpine-jre-openjdk11 version from 1.3.1 to 1.3.0 in Dockerfile: FROM folioci/alpine-jre-openjdk11:1.3.0 Note: I'm using Windows, on Linux it seems not to be reproducible. Full log:
Caused by: org.apache.kafka.common.KafkaException: Error checking for remaining bytes after reading batch
at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.ensureNoneRemaining(DefaultRecordBatch.java:654) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:611) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:575) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1631) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1666) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1507) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:733) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:684) ~[mod-source-record-manager.jar:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1304) ~[mod-source-record-manager.jar:?]
... 8 more
Caused by: java.util.zip.ZipException: Corrupt GZIP trailer
at java.util.zip.GZIPInputStream.readTrailer(Unknown Source) ~[?:?]
at java.util.zip.GZIPInputStream.read(Unknown Source) ~[?:?]
at java.io.BufferedInputStream.fill(Unknown Source) ~[?:?]
at java.io.BufferedInputStream.read(Unknown Source) ~[?:?]
at java.io.FilterInputStream.read(Unknown Source) ~[?:?]
|
| Comments |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
All changes in the folioci/alpine-jre-openjdk11 container when comparing 1.3.0 and 1.3.1:
busybox-1.34.1-r3 x86_64 {busybox} (GPL-2.0-only) [installed]
busybox-1.34.1-r5 x86_64 {busybox} (GPL-2.0-only) [installed]
ca-certificates-bundle-20191127-r7 x86_64 {ca-certificates} (MPL-2.0 AND MIT) [installed]
ca-certificates-bundle-20211220-r0 x86_64 {ca-certificates} (MPL-2.0 AND MIT) [installed]
libcrypto1.1-1.1.1l-r7 x86_64 {openssl} (OpenSSL) [installed]
libcrypto1.1-1.1.1n-r0 x86_64 {openssl} (OpenSSL) [installed]
libretls-3.3.4-r2 x86_64 {libretls} (ISC AND (BSD-3-Clause OR MIT)) [installed]
libretls-3.3.4-r3 x86_64 {libretls} (ISC AND (BSD-3-Clause OR MIT)) [installed]
libssl1.1-1.1.1l-r7 x86_64 {openssl} (OpenSSL) [installed]
libssl1.1-1.1.1n-r0 x86_64 {openssl} (OpenSSL) [installed]
ssl_client-1.34.1-r3 x86_64 {busybox} (GPL-2.0-only) [installed]
ssl_client-1.34.1-r5 x86_64 {busybox} (GPL-2.0-only) [installed]
tzdata-2021e-r0 x86_64 {tzdata} (Public-Domain) [installed]
tzdata-2022a-r0 x86_64 {tzdata} (Public-Domain) [installed]
zlib-1.2.11-r3 x86_64 {zlib} (Zlib) [installed]
zlib-1.2.12-r0 x86_64 {zlib} (Zlib) [installed]
JAVA_VERSION jdk-11.0.14+9
JAVA_VERSION jdk-11.0.14.1+1
|
| Comment by Serhii_Nosko [ 26/Apr/22 ] |
|
Thanks Julian Ladisch, can we try to remove these 2 differences from new image and we will try to import again?
zlib-1.2.11-r3 x86_64 {zlib} (Zlib) [installed]
zlib-1.2.12-r0 x86_64 {zlib} (Zlib) [installed]
zlib its basically data-compression library and we have ZipException in our modules, so it could help |
| Comment by Jakub Skoczen [ 26/Apr/22 ] |
|
Serhii_Nosko Julian Ladisch I don't see how zlib should be related as the exception is from the built-in Java Gzip library. |
| Comment by Serhii_Nosko [ 26/Apr/22 ] |
|
Jakub Skoczen yes, its strange but I don't see any other libraries that were updated in 1.3.1 that can influence more that zlib. |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
JRE can be configured to either use a system zlib or a bundled zlib: |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
I've created mod-source-record-manager:3.4.0-SNAPSHOT.606-zlib-1.2.11-r3 with this Dockerfile:
FROM folioci/mod-source-record-manager:3.4.0-SNAPSHOT.606
USER root
RUN echo 'https://dl-cdn.alpinelinux.org/alpine/v3.11/main' > /etc/apk/repositories
RUN apk --no-cache add zlib=1.2.11-r3
USER folio
It's available from https://hub.docker.com/r/julianladisch/mod-source-record-manager/tags |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
The other way round to check the other possibility: I've created mod-source-record-manager:3.4.0-SNAPSHOT.606-JAVA_VERSION-jdk-11.0.14_9 with the previous JAVA_VERSION but latest Alpine packages with this Dockerfile and the jar from commit 6a065eb4:
FROM folioci/alpine-jre-openjdk11:1.3.0
USER root
RUN apk --no-cache upgrade
USER folio
ENV VERTICLE_FILE mod-source-record-manager-server-fat.jar
# Set the location of the verticles
ENV VERTICLE_HOME /usr/verticles
# Copy your fat jar to the container
COPY mod-source-record-manager-server/target/${VERTICLE_FILE} ${VERTICLE_HOME}/${VERTICLE_FILE}
# Expose this port locally in the container.
EXPOSE 8081
|
| Comment by Serhii_Nosko [ 26/Apr/22 ] |
|
Julian Ladisch Basically I see this ZipException at least for mod-source-record-manager, mod-source-record-storage, mod-inventory and mod-quick-mark, and today even saw in mod-search, so to make it working I need redeploy all these modules locally with 1.3.0 image and after this local vagrant on Windows can work for data import(if any other modules will not throw this exception), can we build some test vagrant box with all modules built on the same image and I will verify behaviour on Windows? |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
Serhii_Nosko: Can you replace mod-source-record-manager in your Vagrant box by julianladisch/mod-source-record-manager:3.4.0-SNAPSHOT.606-zlib-1.2.11-r3 and report whether this actually fixes the issue? If yes I can downgrade zlib in folioci/alpine-jre-openjdk11 and I can re-create all affected docker containers. |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
It is very likely that it is this issue: |
| Comment by Serhii_Nosko [ 26/Apr/22 ] |
|
Julian Ladisch I run old vagrant with 1.3.0 image(to be sure that other modules are working) and redeploy my local srm with 1.3.1 image - ZipException occured. julianladisch/mod-source-record-manager:3.4.0-SNAPSHOT.606-zlib-1.2.11-r3 After this I have not seen ZipException any more, and imports were successful, so I think we can use it in building our new vagrant boxes, thank you a lot |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
Serhii_Nosko: Thanks. We are going to create a new folioci/alpine-jre-openjdk11 container with the zlib downgrade: https://github.com/folio-org/folio-tools/pull/214 |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
David Crossley, John Malconian: Please merge https://github.com/folio-org/folio-tools/pull/214 and release a new folioci/alpine-jre-openjdk11 container. |
| Comment by Julian Ladisch [ 26/Apr/22 ] |
|
According to upstream bug report https://github.com/madler/zlib/issues/613 the root cause is the handling of higher order bytes that affects x86-64 but not x86 32-bit, arm 32-bit and arm 64-bit. |
| Comment by David Crossley [ 27/Apr/22 ] |
|
This is now merged, built, verified, and deployed as v1.3.2 Thanks for your effort Julian Ladisch Attention: Serhii_Nosko |
| Comment by Julian Ladisch [ 27/Apr/22 ] |
|
Thanks! Alpine has published zlib 1.2.12-r1 with the fix, but only for edge (= unstable development branch), not for the stable branch 3.15: |
| Comment by Julian Ladisch [ 28/Apr/22 ] |
|
The stable branch 3.15 finally got zlib 1.2.12-r1 with the fix: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33688 |
| Comment by David Crossley [ 28/Apr/22 ] |
|
Re-opened this ticket. I will re-build. |
| Comment by David Crossley [ 29/Apr/22 ] |
|
Re-built, verified, and pushed as 1.3.3 and latest.
|