Purpose/Overview:

After a recent update, performance of mod-user-import degraded dramatically. Historically, we've been able to load about 200 records per second in batches of 250, now we load 7 records per second in batches of 10. 

This leads to excessive API calls and load times – especially when loading more than 200K users at once. 

Hosting reports 7 calls/second on /perms/users  during loads which imposes heavy load on the database. Given the frequency, it appears that a call is being made on /perms/users for every single user.

Given that tens of thousands of users is common for FOLIO libraries, and some have over 100K, this creates significant problems for user maintenance.  This problem probably affects everyone – have witnessed this behavior on multiple clusters, including on an unshared one (Massey in ap-southeast-2) 

Requirements:

• Users shall be able to use mod-user-import to bulk load users in batches of at least 200 that can be processed at least 100 users per second

• Database load will not be excessive nor interfere with other operations

Desired functionality:

• Batch size of 1000 or more works reliably

• Loads at 500 users/second can be expected