...
When a user clicks on 'Export package CSV' the modal window is getting shown. The user can select all or multiple Package fields, Title fields, and additional fields to export. Then a user presses the 'Export' button, the modal panel disappears, and an export process is getting started. The green toast message is displayed, it shows the name of generating file, and the approximate duration of export (30 mins). Is the user automatically directed to Export Manager ?
The name of the generated file depends on what is going to be exported from the packages:
...
Works similar to Package export. Then a user presses the 'Export' button, the modal panel disappears, and an export process is getting started. The green toast message is displayed, it shows the name of generating file, and the approximate duration of export (30 mins). Is the user automatically directed to Export Manager ?
The name of a generated file is always <<YYYY_MM_DD>>_<<Title name>>_titledetails.csv, for example 2022_04_11_WileyOnlineLibrary_titledetails.csv
...
The Export Manager application can satisfy the given requirements. It provides functionality to process batches of data in a flexible and configured manner. There are sources we can reuse to retrieve various objects, generate CSV files, upload files into vendor-specific storage, and share access to the stored files. This application can manage 'immediate' export jobs and 'scheduled' export jobs, that have to be configured before running. Export Manager consists of backend modules mod-data-export-spring, mod-data-export-worker, and UI module ui-export-manager.
...
This module should be able to display and filter new eHoldings jobs, that we will use to export packages & titles.
What should be done in this module:
- add a new job type - 'eHoldings';
mod-data-export-spring is designed to manage, configure, and run jobs. This module is the entry point to start data export, it calls mod-data-export-worker to execute jobs sending events to the mod-data-export-worker's Kafka topic.
...
What should be done in this module:
- add new export type - 'eHoldings';
- add new request parameters (to ExportTypeSpecificParameters.json), needed to pass export fields, search params for titles search, and other params;
- add new JobCommandBuilder, needed to take request parameters to pass in Kafka event;
...
mod-data-export-worker is intended to receive events from mod-data-export-spring,andexecute its jobs. The module is built based on Spring Batch Framework, and jobs are configured
...
by a set of steps. The execution of a job happens in 3-
...
stages: retrieve data, process data, and write data to the file. Uploading files to some vendor-specific storage is preconfigured already (using AWS S3 bucket) by the listener and happens when all the
...
3 stages are completed.
...
What should be done in this module:
- create a
...
Reader extending base functionality (CsvItemReader.java, see CirculationLogCsvItemReader.java as an example). The reader should retrieve packages/titles using REST clients, taking search parameters from the incoming Kafka event (from job parameters);
- create a Processor (implementing ItemProcessor). The processor has to take only selected fields for export from the incoming packages/titles. The list of fields for export comes from job parameters;
- create a Writer extending base functionality (we can just use CsvWriter.java if nothing special is needed);
- create
...
a Configurationto build a job and set Reader, Writer, and Processor (see CirculationLogJobConfig.java);
- configure a cleaner to purge deprecated files (that we generated more than 30 days back);
ui-eholdings will be able to create and send the export jobs from UI
What should be done in this module:
- store the fields available to export (see attached user story)
- send jobs to mod-data-export-spring;
mod-kb-ebsco-java
...
stores packages & titles, and provides REST API to retrieve these objects. REST methods are already provided and needed for mod-data-export-worker:
Retrieve package details: GET /eholdings/packages/{packageId}
Retrieve package titles: GET /eholdings/titles/{titleId}
...
Questions to the story:
- Should a user be automatically directed to the Export Manager after pressing the 'Export' button?
- Should the list of package&title fields be configured in Settings? Or it is hardcoded?
- ToDo
...
Questions/Answers