Terms
Data export - a general solution that should be applied for all new features intended to export data from Folio modules to the destination (file, database, etc.).
A potential application of this solution isn't limited only by SRS/Inventory Data export feature (ui-data-export and mod-data-export modules). It should be a system-wide solution leveraged for all data export business cases.
Business goals
The new Data export approach was designed for the following features:
- Export orders in bulk via delimited file format: https://issues.folio.org/browse/UXPROD-2318
- Build a report with rollover errors and store in CSV file: https://issues.folio.org/browse/MODFISTO-173
- Circulation log export to CSV: https://issues.folio.org/browse/UXPROD-2691
- Cornell Library's go-live requirements to transfer fees/fines to the Cornell bursar system: https://issues.folio.org/browse/UXPROD-2862
- The ability to import/export fund updates via CSV file in order to bulk edit funds: https://issues.folio.org/browse/UXPROD-199
There is no intent to replace any existing data export solution for now. If later there will be a requirement to significantly extend any existing Data export solution, the new approach should be applied.
Architecture design
To eliminate limitations of the existing mod-data-export module (see Data export by using mod-data-export) and speed up development 2 new modules should be implemented based on the Spring way approach (see Pic. 1):
- mod-data-export-spring
- mod-data-export-worker
Pic. 1.
mod-data-export-spring
The module that is responsible for:
- Receiving data export Job requests via REST API
- Storing Job requests in a database and providing Job search capabilities
- Sending Job commands to mod-data-export-worker module(s) via Kafka
- Receiving Job status updates from mod-data-export-worker via Kafka and updating them in the database
- Providing file download link, once data export is finished
The module should be based on the Spring way approach. Sample Spring way module: mod-spring-template.
Technologies that should be leveraged to implement the module:
- Spring framework
- Spring Boot
- Spring Kafka (to easy up work with Kafka)
- folio-spring-base (for integration with Okapi)
- HikariCP (connection pool)
- Liquibase (for database schema migration)
- PostgreSQL as database
- Lombok
Data model
Pic. 2.
mod-data-export-worker
The module retrieves data from other Folio modules via their REST API and adds it to CSV file parts. Once all required data is retrieved, the worker uploads the file parts to the Folio Object storage (AWS S3, MinIO, etc.) via MinIO and assembles the target CSV file from the parts there. Then the target files can moved to some file storage like AWS S3, MinIO, FTP/SFTP, etc.
Once the file is uploaded, the module generates a download URL and sends it back to mod-data-export-spring via Kafka.
The worker should be able to export data from the following modules:
- mod-audit
- mod-orders
- mod-finance
The worker should be extensible to easy up:
- The implementation of data export form other modules
- Introduction of the new export file types
Technologies that should be leveraged to implement the module:
- Spring framework
- Spring Batch
- Spring Boot
- Spring Kafka (to easy up work with Kafka)
- folio-spring-base (for integration with Okapi)
- HSQLDB as in-memory database for Spring Batch
- Lombok
mod-data-export-worker is based on the Spring Batch framework, which has the following advantages:
- Good separation of concerns. It has concepts of:
- Jobs
- Steps
- Tasks
- Task partitions
- Readers
- Writers
- etc.
It makes it easy to implement Batch export workers and extend them later. - Chunk based processing
Spring batch is designed a way to process batch jobs chunk by chunk.
Each chunk is extracted from a data source, transformed (if required), and loaded to a destination file/storage. - Parallel Job execution by using multiple threads.
A Task could be split into smaller parts by Partitioner, then each part could be processed by a separate thread. - Ability to save job state in a database and continue execution from the place at which the job was interrupted.
- It also provides a lot of different hooks and custom steps, which also makes it easy to create a new job or extend an existing one.
NOTE: Mod-data-export-worker can be used with PostgreSQL database for spring batch. According investigation in scope of MODEXPW-215 using PostgreSQL with Spring Batch requires:
- Providing READ_COMMITTED isolation level for spring batch config for job repository .
- Using unique set of job parameters for job launching as spring batch does not allow to run several jobs with same job's parameters.
Kafka
The following topics should be created in Kafka:
- <tenant>.data-export.job.command
The topic to send start job commands from mod-data-export-spring to mod-data-export-workers. Each worker gets messages from a dedicated topic partition(s).
It helps to make sure that the same Job isn't executed by multiple workers. The worker sends message processing acknowledgment only after Job is finished.
If the worker fails in the middle of the job execution, it can retrieve uncommitted messages from Kafka again and re-process the Job. - <tenant>.data-export.job.update
The topic to send Job status updates from mod-data-export-workers to mod-data-export-spring. The later updates Job status in PostgreSQL database according to received messages.
Scalability
The solution could be easily scaled horizontally by increasing the number of mod-data-export-spring and mod-data-export-worker instances.
Both modules are stateless. mod-data-export-spring persists its state in the PostgreSQL database. All Start job requests for mod-data-export-worker are stored in Kafka.
Security
All modules should leverage Folio module security components and best practices.
Kafka security is out of the scope of this document.