...
Complexity
S, M, L, XL, XXL
Table
| Category | Problem definition | Business impact | Proposed solution | Priority DEV | Priority PO | Complexity | Existing Jira item(s) | Current feature(s) | Final feature (s) |
|---|
| 1 | Performance | Kafka producer closed after sending | Low performance of import | Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes. | High |
| L | |
FOLIO Issue Tracker| System Jira | | columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution |
|---|
| columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
|---|
| serverId |
|---|
|
|
6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-499 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 2 |
| WARN message when no handler found | none | Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages | Low |
| S | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURCE-340 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 3 | Stability/Reliability | Race condition on start (Kafka consumers start working before DB is configured) OR Periodical DB shutdown after SRS restart. Jobs get stuck if not able to update status in DB (messages ACKed even if we could not process them) | Imports might get stuck on module restart | Need investigation / check Investigate the issue with DB (possible OOM on PG server)
| Mid |
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURCE-339 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 4 | Performance Stability/Reliability | High CPU/Memory consumption on modules | Low performance of import. Higher costs for hosting | Significantly decrease size of payload: - Remove immutable parts. Instead fetch them on demand and cache locally for reuse.
- Change message handling mechanism (currently relies on pt1 - profile) (optional)
- Move archiving to Kafka instead of module level
| High |
| XXL | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-439 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-519 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-405 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-408 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-460 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINVOICE-251 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINVOICE-252 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODPUBSUB-167 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURCE-286 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURCE-290 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-463 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-464 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-465 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-466 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-468 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-469 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-474 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-519 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 5 | Performance | Kafka cache resource consumption | Low performance of import. Higher costs of hosting. | Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6) | Mid |
| M | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-444 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-401 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 6 | Stability/Reliability | Duplicates created upon import | Data inconsistency on import. | Make consumers behave idempotent. Add pass-through identifier to de-duplicate messages. | High |
| XL | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-474 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-440 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-491 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-495 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 7 | Stability/Reliability | Kafka consumers stop reading messages eventually, breaking job progress until module restart. | Imports eventually get stuck until module restart | Need investigation | High |
| ? | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-417 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 8 | Code quality | Test coverage is not high enough (Unit) | Higher amount of bugs | Write more tests | Mid |
| S | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODPUBSUB-168 |
|---|
|
| |
FOLIO Issue Tracker | System Jira |
|---|
| columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution |
|---|
| columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
|---|
| serverId |
|---|
|
|
6ccf3fe43301368a983e20c466b11a493135 |
|
| 9 | Code quality | Test coverage is not high enough (Karate) | Higher amount of bugs | Write more tests (define test cases) | Mid |
| L | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-2697 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-2697 |
|---|
|
|
|
| 10 | Stability/Reliability | mod-data-import stores input file in memory, limiting size of uploaded file and possibly having oom | Data import file size is limited | Split to chunks, put to database, work with database/temp storage. Partially done (to be investigated) | Mid |
| L | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-390 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-392 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-465 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 11 | Performance | Data import impacts other processes | Slower response of system during data import | Need investigation (possible solution - configure rate limiter) Relates to number 4 |
|
|
| no ticket | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 12 | Performance | High resource consumption to get job(s) status/progress | Slow performance of import and landing page. | Add some kind of caching for progress tracking (database or in-memory) | Low |
| S | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-469 |
|---|
|
|
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UIDATIMP-918 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 13 | Stability/Reliability | SRS can fail when processing messageduring import
| Import can end up creating some instances but not creating holdings/items for some MARC records | Generate "INSTANCE CREATED" from mod-inventory. Consume in SRS to update HRID in BIB and in INVENTORY to continue processing.
Remove unnecessary topics (* ready for post processing and hrid set) | Mid |
| L | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-500 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 14 | Stability/Reliability | If we have infrastructure issue (like DB not available, module being restarted or network failure), we are sending DI_ERROR instead of retrying | Records that can potentially be processed during import are not processed if we have temporary infrastructure issues (DB down, network connectivity loss, etc) | Do not ACK messages in Kafka if there's not a logic, but infrastructure error/exception. Split failed processing results into 2 categories: - IO errors - do not ack. retry until fixed
- Business logic - DI_ERROR and Ack current message
| Mid |
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODDATAIMP-501 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 15 |
| Consumer gets disconnected from Kafka cluster | Jobs get stuck until module restart | Need investigation | Mid |
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODINV-417 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
| 16 |
| De-duplication of status messages for progress bar | Progress bar might display incorrect progress | De-duplicate status messages per-record while tracking progress | Mid |
| L (depends on 12) | |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | MODSOURMAN-522 |
|---|
|
| |
FOLIO Issue Tracker6ccf3fe43301368a983e20c466b11a49| ee9b165564fc | | key | UXPROD-3135 |
|---|
|
|
|
Filters
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| columnIds | issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution |
|---|
| columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
|---|
| maximumIssues | 20 |
|---|
| jqlQuery | labels in (di-configuration, di-data-integrity, di-performance, di-functional-bug, di-change-request, di-architecture, di-other) and resolution in (Unresolved) ORDER BY status DESC |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
|
Issues to potentially remove from scope
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
| key | MODDATAIMP-410 |
|---|
|
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
| key | MODDATAIMP-430 |
|---|
|
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
| key | MODDATAIMP-444 |
|---|
|
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
| key | MODSOURCE-300 |
|---|
|
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
| key | MODSOURMAN-481 |
|---|
|
| Jira Legacy |
|---|
| server | FOLIO Issue TrackerSystem Jira |
|---|
| serverId | 6ccf3fe401505d01-3301b853-368a3c2e-983e90f1-20c466b11a49ee9b165564fc |
|---|
| key | MODSOURMAN-521 |
|---|
|
Links
Data Import Observations for Improvements