Some performance testing showed bad performance of mod-patron-blocks under heavy load. Tracing showed that a HTTP request was accepted but actual execution of database queries took place towards the end of the request. A sample HTTP took 8 seconds to complete, no action for the first 7.8 seconds but then database queries execute afterwards.

Review of the code found that mod-patron-blocks' use of CQL2PgJSON is the culprit. CQL2PgJSON is initialized almost every HTTP request to the mod-patron-blocks. CQL2PgJSON initialization is expensive because it uses blocking file APIs to read the schema.json and other Jackson related activities. Even more critical - the event loop is blocked while initializing CQL2PgJSON because of the blocking file retrievals and general “long” duration of the task on the eventloop. I believe this is a major cause of the 8 second slowdown while CPU on the module instance was at less than 10%. It is very inefficient to initialize the class for the same “table” multiple times. For some HTTP requests, CQL2PgJSON is initialize at least three times for different tables.

Here is a CPU profile of /automated-patron-blocks/{userId} with about 300 invocations lasting less than 30 seconds.

Open

93% of CPU for this api call was spent in initializing PatronBlocksService which invokes many CQL2PgJSON classes under the covers.

Here is a memory profile from the same run as above.

Open

Most of the memory allocations is caused by instantiation of CQL2PgJSON. 99% of the memory allocations is caused by instantiation of PatronBlocksService. 72% of the memory allocations for the total duration of the module.

Remediation

After some rough code changes to initialize CQL2PgJSON only once for a “table”, here are the updated stats

Open

Open

Deploying the changes show massive decrease in execution times for mod-patron-blocks. The misuse of CQL2PgJSON is pervasive across older FOLIO modules in Inventory and Circulation. There should be some work to update CQL2PgJSON if it is to be used this way, but more efficient use of CQL2PgJSON should be employed in this module.