[FOLIO-3248] folio-tools/clear-loans runs slow like snail Created: 23/Jul/21  Updated: 11/Aug/21  Resolved: 11/Aug/21

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: Zak Burke Assignee: Zak Burke
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Sprint: Prokopovych - Sprint 120
Story Points: 3
Development Team: Prokopovych

 Description   

Overview: The script runs slowly and opaquely when the data set is large. Resolve promises in parallel and add timestamps to debugging output.

Details:

  • replace console.log with one that logs a timestamp first
  • refactor eachPromise so the list can be partitioned and the partitions handled in parallel. Yeah, that's a little goofy given the original idea was to handle the list in series, but if handling every request on the list simultaneously is a bad idea, so is handling them one-by-one.


 Comments   
Comment by Zak Burke [ 26/Jul/21 ]

Two fixes:

  1. Implement multi-threading: Default to a single thread, but use --threads CLI param to set the number of parallel threads. This will help with both fetching and iterating through results lists. 
  2. Implement run-time page-size: Default to the current 100 entities-per-page when result-sets are retrieved in batches, but use --page-size to allow larger pages and, thus, fewer overall requests. This will help with fetching only.
Generated at Thu Feb 08 23:26:42 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.