[FOLIO-3703] Poor performance on reference environments Created: 13/Feb/23 Updated: 27/Feb/23 Resolved: 27/Feb/23 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Wayne Schneider | Assignee: | Wayne Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue links: |
|
||||||||
| Sprint: | DevOps Sprint 159, DevOps Sprint 160 | ||||||||
| Development Team: | FOLIO DevOps | ||||||||
| RCA Group: | TBD | ||||||||
| Description |
|
Multiple users have reported poor performance in the FOLIO reference environments (folio-snapshot/folio-snapshot-2) starting with the builds overnight Feb 12/13. folio-snapshot build |
| Comments |
| Comment by Wayne Schneider [ 13/Feb/23 ] |
|
The issue seems to be caused on both environments by mod-data-export-worker thrashing attempting to process a Kafka message it cannot parse. See attached stack trace log For the moment, I have stopped the mod-data-export-worker container on both environments, and they are again responsive. |
| Comment by Ann-Marie Breaux (Inactive) [ 14/Feb/23 ] |
|
Hi Wayne Schneider In your comment above, you mention mod-data-import-worker in the first sentence, but there's no mod-data-import-worker module. I think? (hope?) it was just a typo! Also, could you assign the dev team or else mark the dev team as Other dev so that it is not orphaned? Thank you! |
| Comment by Jakub Skoczen [ 14/Feb/23 ] |
|
Ann-Marie Breaux This is a ticket for the DevOps teams. Mikita Siadykh and Ivan Kryzhanovskyi mentioned on Slack that the problem in the module have been found and it being addressed – please link that ticket here as a blocker. Thank you. |
| Comment by Wayne Schneider [ 16/Feb/23 ] |
|
As of 16 Feb 15:45 UTC, folio-snapshot using mod-data-export-worker-3.0.0-SNAPSHOT.370 still exhibits the issue with deserialization. folio-snapshot-2 with mod-data-export-worker-3.0.0-SNAPSHOT-373 seems to be fine. Holding this ticket open until both environments are cleaned up, hopefully tomorrow. |
| Comment by Wayne Schneider [ 27/Feb/23 ] |
|
This is now resolved. |