| Page Properties | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
A potential application of this solution isn't limited only by SRS/Inventory Data export feature (ui-data-export and mod-data-export modules). It should be a system-wide solution leveraged for all data export business cases.
Business goals
The new Data export approach was designed for the following features:
- Export orders in bulk via delimited file format: https://issues.folio.org/browse/UXPROD-2318
- Build a report with rollover errors and store in CSV file: https://issues.folio.org/browse/MODFISTO-173
- Circulation log export to CSV: https://issues.folio.org/browse/UXPROD-2691
- Cornell Library's go-live requirements to transfer fees/fines to the Cornell bursar system: https://issues.folio.org/browse/UXPROD-2862
- The ability to import/export fund updates via CSV file in order to bulk edit funds: https://issues.folio.org/browse/UXPROD-199
There is no intent to replace any existing data export solution for now. If later there will be a requirement to significantly extend any existing Data export solution, the new approach should be applied.
Architecture design
To eliminate limitations of the existing mod-data-export module (see Data export by using mod-data-export) and speed up development 2 new modules should be implemented based on the Spring way approach (see Pic. 1):
...
The following topics should be created in Kafka:
- dataExportJobCommandsTopic<tenant>.data-export.job.command
The topic to send start job commands from mod-data-export-spring to mod-data-export-workers. Each worker gets messages from a dedicated topic partition(s).
It helps to make sure that the same Job isn't executed by multiple workers. The worker sends message processing acknowledgment only after Job is finished.
If the worker fails in the middle of the job execution, it can retrieve uncommitted messages from Kafka again and re-process the Job. - dataExportJobExecutionUpdatesTopic<tenant>.data-export.job.update
The topic to send Job status updates from mod-data-export-workers to mod-data-export-spring. The later updates Job status in PostgreSQL database according to received messages.
...