[FOLIO-3703] Poor performance on reference environments Created: 13/Feb/23  Updated: 27/Feb/23  Resolved: 27/Feb/23

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Wayne Schneider Assignee: Wayne Schneider
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: File mod-data-export-worker.log    
Issue links:
Blocks
is blocked by MODEXPW-333 Update the module to Spring boot v3.0... Closed
Sprint: DevOps Sprint 159, DevOps Sprint 160
Development Team: FOLIO DevOps
RCA Group: TBD

 Description   

Multiple users have reported poor performance in the FOLIO reference environments (folio-snapshot/folio-snapshot-2) starting with the builds overnight Feb 12/13.

folio-snapshot build
folio-snapshot-2 build



 Comments   
Comment by Wayne Schneider [ 13/Feb/23 ]

The issue seems to be caused on both environments by mod-data-export-worker thrashing attempting to process a Kafka message it cannot parse. See attached stack trace log.

For the moment, I have stopped the mod-data-export-worker container on both environments, and they are again responsive.

Comment by Ann-Marie Breaux (Inactive) [ 14/Feb/23 ]

Hi Wayne Schneider In your comment above, you mention mod-data-import-worker in the first sentence, but there's no mod-data-import-worker module. I think? (hope?) it was just a typo! Also, could you assign the dev team or else mark the dev team as Other dev so that it is not orphaned? Thank you!

Comment by Jakub Skoczen [ 14/Feb/23 ]

Ann-Marie Breaux This is a ticket for the DevOps teams. Mikita Siadykh and Ivan Kryzhanovskyi mentioned on Slack that the problem in the module have been found and it being addressed – please link that ticket here as a blocker. Thank you.

Comment by Wayne Schneider [ 16/Feb/23 ]

As of 16 Feb 15:45 UTC, folio-snapshot using mod-data-export-worker-3.0.0-SNAPSHOT.370 still exhibits the issue with deserialization. folio-snapshot-2 with mod-data-export-worker-3.0.0-SNAPSHOT-373 seems to be fine.

Holding this ticket open until both environments are cleaned up, hopefully tomorrow.

Comment by Wayne Schneider [ 27/Feb/23 ]

This is now resolved.

Generated at Thu Feb 08 23:30:04 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.