MinioS3Client#write fails after 52428800000 bytes
Description
CSP Request Details
None
CSP Rejection Details
None
Potential Workaround
None
Checklist
hideTestRail: Results
Activity
Show:
Julian Ladisch March 13, 2023 at 8:52 AM
Pull request for code review: https://github.com/folio-org/folio-s3-client/pull/6
Done
Details
Details
Assignee
Unassigned
UnassignedReporter
Julian LadischPriority
P3Development Team
Firebird
Fix versions
Release
Orchid (R1 2023) Bug Fix
RCA Group
Implementation coding issue
Affected releases
Orchid (R1 2023)
TestRail: Cases
Open TestRail: Cases
TestRail: Runs
Open TestRail: Runs
Created March 13, 2023 at 8:08 AM
Updated October 12, 2023 at 3:50 AM
Resolved March 14, 2023 at 12:01 PM
TestRail: Cases
TestRail: Runs
MinioS3Client#write fails when the stream exceeds 52428800000 bytes.
This is caused by setting the partSite to MIN_MULTIPART_SIZE = 5242880.
https://github.com/folio-org/folio-s3-client/blob/v1.0.0/src/main/java/org/folio/s3/client/MinioS3Client.java#L169
S3 allows a maximum of only 10000 parts:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
When starting the 10001st part it fails.
The write method gets the data to upload from an InputStream that needs to be processed in sequence. Splitting it into parts for parallel upload cannot give significant performance advantages. Therefore the maximum part size, not the minimum part size, should be used.