Story:
Jira Legacy | ||||||
---|---|---|---|---|---|---|
| ||||||
Options that were considered:
Table of Contents |
---|
Local File Storage:
Generated files could be stored in the local file System. This method is already planned to be used in File uploading when Instance UUIDs are uploaded.
...
- Ephemeral - The generated files will be lost if the hosting environment is restarted
- Space constraints - We can only generate a fixed sized files
- stateless- If more then one instance of the module is running, it will be difficult to keep track of where the files were generated
Postgres DB
PostgreSQL provides two distinct ways to store binary data. Binary data can be stored in a table using the data type bytea or by using the Large Object
Bytea | Large Object |
---|---|
data type is not well suited for storing very large amounts of binary data | storing binary data is better suited to storing very large values |
can hold up to 1 GB of binary data | can store upto 4TB |
it would require a huge amount of memory to process such a large value | have some security issues since anyone connected to the database can view and/or modify any Large Object, even if they don't have permissions to view/update the row containing the Large Object reference |
Processing: CREATE TABLE files (imgname text, img bytea); File file = new File("myTxt.txt"); FileInputStream fis = new FileInputStream(file); PreparedStatement ps = conn.prepareStatement("INSERT INTO files VALUES (?, ?)"); ps.setString(1, file.getName()); ps.setBinaryStream(2, fis, file.length()); ps.executeUpdate(); ps.close(); fis.close(); | Processing: CREATE TABLE fileslo (fileName text, filesoid oid); // Get the Large Object Manager to perform operations with LargeObjectManager lobj = ((org.postgresql.PGConnection)conn).getLargeObjectAPI(); int oid = lobj.create(LargeObjectManager.READ | LargeObjectManager.WRITE); LargeObject obj = lobj.open(oid, LargeObjectManager.WRITE); // Now open the file File file = new File("myTxt.txt"); FileInputStream fis = new FileInputStream(file); // Copy the data from the file to the large object byte buf[] = new byte[2048]; int s, tl = 0; while ((s = fis.read(buf, 0, 2048)) > 0) { obj.write(buf, 0, s); tl += s; } // Close the large object obj.close(); // Now insert the row into imageslo PreparedStatement ps = conn.prepareStatement("INSERT INTO fileslo VALUES (?, ?)"); ps.setString(1, file.getName()); ps.setInt(2, oid); ps.executeUpdate(); ps.close(); fis.close(); |
...
- RMB doesn't provide an easy way to stream the files to DB.
- We can use https://vertx.io/docs/vertx-pg-client/js/, but we will have to come up with our own implementation
AWS S3:
S3 can be used to store the files, and the link to the files can be stored in the logs. We also have a precedent where the edge modules are already using AWS as one of the implementations, among several other options(https://github.com/folio-org/edge-common#secure-stores) and there were no major complaints
...
- Hosting implementation specific
Generic Implementation
This can be figured out in another spike, we could explore other Document store DBs or object Stores
...