...
To ensure more stable application's work under load and with a large number of users, it is necessary to revise the mechanism for calculating the progress of work. The best solution for this would be to add a new API to support the landing page UI and send lightweight DTOs, rather than the full job execution objects that are involved in the data import process. New objects will be stored separately from job execution and updated in the background. Adding indexes will help to reduce size of logs, plus help with log sorting and retrieval. This will also help when multiple users open the landing page.
| Story | Estimation |
|---|---|
| MODSOURMAN-468 Create a new API and database table that should store and represent information for the Data-Import landing page. | 5 |
| MODSOURMAN-469 Change data-import progress mechanism with a new plain DB table counter and background job | 8 |
| UIDATIMP-918 Use new API for DataImport landing page | 3 |
...
To ensure a more stable operation of the data import function when using KAFKA, it is necessary to work out in more detail the handling of errors and the application's response to them. To do this, we need to make changes to the general library Folio-Kafka-wrapper and the modules that use it. All errors should be logged and included in the data-import journal. It is also necessary to increase the test coverage of this library and cover edge cases. This should help to keep jobs from getting stuck; they should either complete, complete with errors, or fail.
| Story | Draft estimation |
|---|---|
| MODPUBSUB-167 Reconsider error handling in KafkaConsumerWrapper | 8 |
| MODPUBSUB-168 Cover with tests folio-kafka-wrapper | 5 |
| MODSOURMAN-474 Implement ProcessRecordErrorHandler for Kafka Consumers | 5 |
| MODSOURCE-290 Implement ProcessRecordErrorHandler for Kafka Consumers | 3 |
| MODINVOICE-252 Implement ProcessRecordErrorHandler for Kafka Consumers | 2 |
| MODINV-408 Implement ProcessRecordErrorHandler for Kafka Consumers | 5 |
...