2021-08-20 Meeting notes

How should FOLIO store files (like PDFs attached to orders, agreements, etc.)?

TC discusses whether FOLIO should accept min.io as an official part of FOLIO platform:

• TC 2021-08-04 Meeting notes
• TC 2021-08-11 Meeting notes

Background:

mod-invoice-storage stores files (PDFs) into a JSONB property using base64 encoding:
https://github.com/folio-org/acq-models/blob/master/mod-invoice-storage/schemas/document.json

mod-agreements and mod-licenses store files (PDFs) into pg_largeobject without any tenant or module separation. The ERM development team rejected the request to convert it into a PostgreSQL solution like bytea that provides tenant and module separation (ERM-1779). The ERM development team wants to move to an external solution (UXPROD-3172) like min.io (or some other s3 compatible file storage).

mod-data-export-worker already uses min.io and the the FOLIO Ansible scripts install min.io for this module.

PostgreSQL supports storing binary files: https://wiki.postgresql.org/wiki/BinaryFilesInDB

• "When should files be stored in the database? The common suggestion here is when the files have to be ACID."

• "When is it bad idea to store binary files in the database? Very large files (100MB+), where performance is critical to the application."

• Do smaller binary files result in bad performce? No, because "bytea and text data types both use TOAST (details here)."
• For a detailed discussion see above BinaryFilesInDB link.

min.io server for multi-tenancy is licensed under GNU Affero Public License Version 3 (AGPLv3), this was changed in April 2021, it had been Apache 2 before. Min.io server for bare-metal or single-tenant continues to be released under Apache v2.0.

Proposal for a Security Team decision:

• Binary files must be stored with strict tenant and module separation.
  • This is to support multi-tenant installations.
  • This is to support modules the sysOp doesn't fully trust.