/
DR-000005 - Platform agnostic object storage for Exports

DR-000005 - Platform agnostic object storage for Exports

May 06, 2024

Submitted Date

Apr 1, 2020 

Approved Date

Jun 24, 2020 

Status

ACCEPTED

Impact

MEDIUM

 

Overrides/Supersedes 

This decision was migrated from the Tech Leads Decision Log as part of a consolidation process.  The original decision record can be found here.

RFC 

NA

Stakeholders

  • #sys-ops

  • #development

Contributors

@Kruthi Vuppala 

Approvers

This decision was made by the Tech Leads group prior to the adoption of current decision making processes within the FOLIO project.

Background/Context

The initial implementation of data export utilized AWS S3 object storage.  Since FOLIO shouldn't rely on specific cloud vendors, we need to decide on a platform agnostic storage solution for storing the files generated by the data export process.  The following technologies were considered:

  • MinIO

  • OpenIO

  • Databases

See also:  https://folio-org.atlassian.net/browse/MDEXP-19

Assumptions

NA

Constraints

The following considerations/requirements were taken into account:

MinIO

POC: https://github.com/KVupp/folio-export-aws/blob/master/src/main/java/org/folio/folio_export_aws/MinIo.java

Pros:

Cons:

Reference: https://docs.min.io/

OpenIO

Pros: 

Cons:

Reference: https://www.openio.io/

Databases

Relation or Non relational databases have an overhead of storing static files either  as large files or when stored in chunks.

Costs are higher compared to object storage, and backing them up becomes more difficult as the file sizes grow.

Though database can be logical because all the data is present in a single area, but the disadvantages that come with it overshadow the advantages

Specifics related to postgresDB identified in a different spike are documented here: (https://folio-org.atlassian.net/wiki/x/OQ4V)

Other options Considered:

Rationale

MINIO with it's advantages seems like the go to option with it's easy setup and minimal code changes required for data-export

Decision

Data export will use MinIO for cloud-agnostic object storage

Implications

  • Pros

    • Greater flexibility - not tied to a specific cloud vendor

    • Backwards compatibility - data export still works with AWS S3

    • Few code changes are required

  • Cons

    • If not using AWS S3 (or compatible service), hosting providers will need to setup/maintain additional infrastructure (MinIO)

Other Related Resources

-