From MODDATATIMP-475: 

Review logs for Iris Bugfest job that finished prematurely and did not create all the expected records

• https://bugfest-iris.folio.ebsco.com

• Job 6436 run by @Ann-Marie Breaux

• 5,000 record file, TAMU_sample_bibs_5k_2.mrc, attached

• Using job profile: PTF Create Instance Holdings Item

• Job started at 8:47 am Iris Bugfest time, and it finished at 8:50 am Bugfest, so only 3 minutes!

• UI log summary shows Completed with errors

• UI log detail shows 1,950 SRS MARC, Instances, Holdings, and Items created

• No indication of why it stopped after processing 1,950 incoming records, instead of all 5,000

After investigation in the scope of MODDATAIMP-475, it seems like the root cause of this issue is that there is deadlock for the "job_monitoring"-table.

There are some logs from srm-module from Iris-Bugfest(See full log as attached-file):

2021-06-24T06:48:59.474Z io.vertx.pgclient.PgException: { "message": "deadlock detected", "severity": "ERROR", "code": "40P01", "detail": "Process 12999 waits for ShareLock on transaction 88425453; blocked by process 13000.\nProcess 13000 waits for ShareLock on transaction 88425452; blocked by process 12999.", "hint": "See server log for query details.", "where": "while rechecking updated tuple (2,38) in relation \"job_monitoring\"", "file": "deadlock.c", "line": "1146", "routine": "DeadLockReport" }

It seems like fix for this issue should fix this unexpected behaviour.

Note: there are a lot of errors "Couldn't update JobExecution status, JobExecution already marked as ERROR". But it seems like they were caused by deadlock (error handling mechanism worked in this way, so ignore them).