accompanies MODDATAIMP-412
P2 priority are red; P3 are green
Motivation
At the moment, the data import functionality has been transferred to a new solution that uses KAFKA as a direct transport. This made it possible to solve the main problems associated with the limitations of the HTTP protocol for data transfer and to implement a message delivery queue. A large amount of data in the payload has significantly increased the usage of KAFKA's disk storage. Also, the current implementation of calculating progress uses a single mechanism displaying progress on the UI, which leads to a slowdown in data import.
...
To implement this solution, we need to implement an API and storage in SRM that will allow you to receive this data. We should load, and take needed values from the cache in other modules involved in data import. Also, we need to move payload zipping functionality to the KAFKA side, to reduce CPU usage by modules
Draft estimation| Estimation |
|---|
| MODSOURMAN-463 Create storage and API for MappingRules and MappingParams | 3 |
| MODSOURMAN-464 Store snapshots of MappingRules and MappingParams to the database |
3| 2 |
| MODSOURMAN-465 Remove MappingRules, MappingParams, and JobProfileSnapshot from the event payload |
2| 1 |
| MODSOURMAN-466 Remove zipping mechanism for data import event payloads |
1| 2 |
| MODSOURCE-286 Remove zipping mechanism for data import event payloads and use cache for params |
2| 5 |
| MODINV-405 Remove zipping mechanism for data import event payloads and use cache for params |
2| 5 |
| MODINVOICE-251 Remove zipping mechanism for data import event payloads and use cache for params | 2 |
Separate the mechanism for calculating progress and displaying it on the UI
To ensure more stable application's work under load and with a large number of users, it is necessary to revise the mechanism for calculating the progress of work. The best solution for this would be to add a new API to support the landing page UI and send lightweight DTOs, rather than the full job execution objects that are involved in the data import process. New objects will be stored separately from job execution and updated in the background.
| Story | Draft estimationEstimation |
|---|
| MODSOURMAN-468 Create a new API and database table that should store and represent information for the Data-Import landing page. | 5 |
| MODSOURMAN-469 Change data-import progress mechanism with a new plain DB table counter and background job | 8 |
| UIDATIMP-918 Use new API for DataImport landing page | 3 |
...