| Category | Problem definition | Business impact | Proposed solution | Priority DEV | Priority PO | Complexity | Existing Jira item(s) | Final feature(s) |
|---|
| 1 | Performance | Kafka producer closed after sending | Low performance of import | Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes. | High |
| L |
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 2 |
| WARN message when no handler found | none | Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages | Low |
| S | | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 3 | Stability/Reliability | Race condition on start (Kafka consumers start working before DB is configured) | Imports might get stuck on module restart | Need investigation / check | Low |
| M |
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 4 | Performance Stability/Reliability | High CPU/Memory consumption on modules | Low performance of import. Higher costs for hosting | Significantly decrease size of payload: - Remove immutable parts. Instead fetch them on demand and cache locally for reuse.
- Change message handling mechanism (currently relies on pt1 - profile) (optional)
- Move archiving to Kafka instead of module level
| High |
| XXL | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-439 |
|---|
|
| Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODSOURMAN-519 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 5 | Performance | Kafka cache resource consumption | Low performance of import. Higher costs of hosting. | Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6) | Mid |
| M | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODINV-444 |
|---|
|
| Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODINV-401 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 6 | Stability/Reliability | Duplicates created upon import | Data inconsistency on import. | Make consumers behave idempotent. Add pass-through identifier to de-duplicate messages. | High |
| XL | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-474 |
|---|
|
| Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-440 |
|---|
|
| Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-491 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 7 | Stability/Reliability | Kafka consumers stop reading messages eventually, breaking job progress until module restart. | Imports eventually get stuck until module restart | Need investigation | High |
| ? | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODINV-417 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 8 | Stability/Reliability | Test coverage is not high enough (Unit) | Higher amount of bugs | Write more tests | Mid |
| S | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODPUBSUB-168 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 9 | Stability/Reliability | Test coverage is not high enough (Karate) | Higher amount of bugs | Write more tests (define test cases) | Mid |
| L | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-2697 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-2697 |
|---|
|
|
| 10 | Stability/Reliability | mod-data-import stores input file in memory, limiting size of uploaded file and possibly having oom | Data import file size is limited | Split to chunks, put to database, work with database/temp storage. Partially done (to be investigated) | Mid |
| L | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-390 |
|---|
|
| Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-392 |
|---|
|
| Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODDATAIMP-465 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 11 | Performance | Data import impacts other processes | Slower response of system during data import | Need investigation (possible solution - configure rate limiter) |
|
|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 12 | Performance | High resource consumption to get job(s) status/progress | Slow performance of import and landing page. | Add some kind of caching for progress tracking (database or in-memory) | Low |
| S | | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODSOURMAN-469 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 13 | Stability/Reliability | SRS can fail when processing messageduring import
| Import can end up creating some instances but not creating holdings/items for some MARC records | Generate "INSTANCE CREATED" from mod-inventory. Consume in SRS to update HRID in BIB and in INVENTORY to continue processing.
Remove unnecessary topics (* ready for post processing and hrid set) | Mid |
| L |
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|
| 14 | Stability/Reliability | Periodical DB shutdown after SRS restart. Jobs get stuck if not able to update status in DB (messages ACKed even if we could not process them) | Sometimes DI import jobs get stuck if there was a restart of SRS during job run. | Investigate the issue with DB (possible OOM on PG server) Do not ACK messages in Kafka if there's not a logic, but infrastructure error/exception. Split failed processing results into 2 categories: - IO errors - do not ack. retry until fixed
- Business logic - DI_ERROR and Ack current message
| Mid |
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | MODSOURCE-339 |
|---|
|
| | Jira Legacy |
|---|
| server | System Jira |
|---|
| serverId | 01505d01-b853-3c2e-90f1-ee9b165564fc |
|---|
| key | UXPROD-3135 |
|---|
|
|