Page Comparison

Tickets:

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key CIRC-573

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key UICIRC-736

...

This endpoint would follow a similar design structure as to those described here:

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key FOLIO-2050

• https://issues.folio.org/secure/attachment/20079/Possible%20batch%20api%20workflow.png

References:

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODUIMP-57

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODINVSTOR-478

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key MODINVSTOR-353

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key RMB-505

• Jira Legacy
  server System JiraJIRA serverId 01505d01-b853-3c2e-90f1-ee9b165564fc key FOLIO-1156

• https://s3.amazonaws.com/foliodocs/api/mod-inventory-storage/instance-storage-batch.html
• https://s3.amazonaws.com/foliodocs/api/mod-inventory-storage/instance-sync.html
• https://www.citusdata.com/blog/2017/11/08/faster-bulk-loading-in-postgresql-with-copy/
• https://stackoverflow.com/questions/46715354/how-does-copy-work-and-why-is-it-so-much-faster-than-insert
• How to design batch API (General recommendations)
• https://www.postgresql.org/docs/12/sql-values.html

...

The existing renewal process is too slow. However, having a batch perform one operation at a time has also proven problematic. The suggested design is to have a batch queue up a defined process renewals and when it has accrued a designated number of renewals and then perform the operationsuccessful renewals, submit them to the storage module as a sub-batch.
The business logic can perform a pre-process where it checks to see if any of the renewals would fail for business logic reasons. Queue Hold all renewals that pass the business logic test and then when the queue sub-batch size is reached perform the SQL operation on that queuemake a request to the storage module using the sub-batch. This is potentially problematic in that if something fails in the queue sub-batch the entire queue sub-batch might be considered a failure. If this is undesired then a queue sub-batch size of 1 must be used. The size of the sub-batch will need to be determined, and could be configurable or a property specified in the request if desired. This approach would require a bulk update endpoint to be available from the storage module.

This refers to queue the sub-batch as a subset of renewals that have passed the business logic processing. In this way, a queue is used to refer not to a batch but instead to a generic collection.

To restrict the amount of memory consumption there should be a max batch size of 10,000. When more than the maximum batch size is sent to the API endpoints then an HTTP 413 "Payload Too Large" should be immediately returned without processing anything.

...

The "error" represents technical failures, such as an SQL failure.

Image RemovedImage Added

SVGJPG: circ-573.svgjpg

Example Multi-update SQL

Postgresql can update multiple rows in a single statement, such as:

...

Ticket