FOLIO Production Library Import Statistics
When running large files in production, especially after migration to a new flower version of FOLIO, please add statistics for large data import jobs in your environment. The more statistics we can gather from production environments, the better data that we will have to compare to performance statistics gathered by the developers and the Performance Task Force.
| Library | Date | Reporter | FOLIO version | Number of records | Elapsed time | Summary of the job | Besides import, how busy was FOLIO at the time? | |
|---|---|---|---|---|---|---|---|---|
| 1 | Cornell | 3/9/23 | Nolana | 1000 | < 5 mins | updated based on 001, simple marc update | run during open hours, but on the early/late edges | |
| 2 | Cornell | 3/9/23 | Nolana | 5000 | ca. 40 mins | updated based on 035, profile contains both marc and instance updates, plus holdings | run during open hours, but on the early/late edges | |
| 3 | 5 Colleges | 3/14/23 | Morning Glory | 349 | 3 min | updated marc srs bib, match on 999 $i, | Not busy - snow day | |
| 4 | 5 Colleges | 3/10/23 | Sara Colglazier | Morning Glory HF1 | 3000 | 7 min | create marc srs, instance, holdings, item | run during open hours, but after main work hours for most (just after 5:30 pm on a Friday) |
| 5 | 5 Colleges | 3/15/23 | Morning Glory | 193 | 1 min | update marc srs bib match on 999 $i | 9:24 am | |
| 6 | 5 Colleges | 3/16/23 | Nolana - dry run | 4 | 4 min | match on instance sys ctrl no, if no match create FOLIO inventory & srs, modify marc to remove 877, 876, 852 | This same file and same DI job profile in our MG production was done in 1min | |
| 7 | 5 colleges | 3/16/23 | Nolana - dry run | 7 | 8 min | match on instance sys ctrl no, if no match create FOLIO inventory & srs, modify marc to remove 877, 876, 852 | This same file and same DI job in our MG production was done in 1 min. | |
| 8 | Wellesley | 3/15/23 | Lynne Fors | Nolana - dry run | 100,000 | 5 hours, 3 minutes (3:03pm-8:11pm) | create marc srs, instance, holdings, modify marc srs | Not very busy. We have a limited number of users who have access to our dry run environment. Unsure if our dry run tenant is hosted with others and if there were impacts there. No idea as to how the dry run is provisioned. |
| 9 | Michigan State | 3/17/23 | Joshua Barton | Morning Glory HF1 | 270 | ca. 17 mins | update instance/srs based on match of incoming 947 to instance system control number; if no match creates new instance/srs and holdings | run mid-afternoon on a work day with other jobs (mostly consisting of single records via DI) being run every 2-5 minutes |
| 10 | 5 Colleges | 3/17/2023 | Sara Colglazier | Morning Glory HF1 | 4000 | 9 min | on no match on sys control no create marc srs, instance, holdings, item | run during closed hours (5:28 pm on a Friday, Spring Break) |
| 11 | 5 colleges | 3/17/2023 | Nolana dry run | 8574 | 80 min | update marc srs match on 999 $i | run overnight - no activity | |
| 12 | 5 Colleges | 3/17/2023 | Morning Glory HF1 | 2000 | never finished | on no match on sys control no create marc srs, instance, holdings, item with marc modify at end | started 5:58. File contained duplicates. Job never completed - ticket with hosting vendor created | |
| 13 | University of Chicago | 3/17/2023 | Christie Thomas | Nolana | 1150 | <2 minutes | no match points, create instance and create holdings for each record. | run mid-day during normal operating hours |
| 14 | University of Chicago | 3/15/2023 | Christie Thomas | Nolana | 2401 | <2 minutes | no match points, create instance and create holdings for each record. | run over the weekend in the evening when there would be minimal activity in the system |
| 15 | 5 Colleges | 3/22/2023 | Nolana dry run | 9000 | 17 min | on no match on sys control no create marc srs, instance, holdings, item with marc modify at end | ||
| 16 | 5 Colleges | 3/23/2023 | Morning Glory HF1 | 4000 | 7 min | update srs marc bib match on 999 $i | completed with errors but file completed. Problem in FOLIO, 2 instances linked to same marc srs bib | |
| 17 | Villanova | 3/23/2023 | Jesse Flavin | Nolana | 25 | 11 min | update instance, holdings, and item matching on UUID | run at 4 pm |
| 18 | 5 Colleges | 3/23 | Nolana dry run | 9000 | 9 min | match on system control no, create holdings and items | not busy in dry run | |
| 19 | 5 Colleges | 3/23 | Nolana dry run | 12000 | didn't finish | on no match on sys control no create marc srs, instance, holdings, item with marc modify at end | No error message. not busy | |
| 20 | University of Chicago | 3/25/2023 | Christie Thomas | Nolana | 1247 | < 2 minutes in production environment | match on 035 to instance OCLC, instance status, and holdings type = electronic - update instance and holdings. If not match, create new | 7 minutes in test / staging environment. Loaded into producti9on mid-day on weekend with little other activity in the system. |
| 21 | Wellesley | 3/27/2023 | Morning Glory HF1 | 128 | 8 minutes | match 999ff$i to Instance UUID, update statistical codes; next, match 999ff$s to MARC SRS 999ff$s and modify MARC to delete 903 and 949 fields | Spring Break, not busy; Started at 10:31 am; Completed at 10:39 am | |
| 22 | 5 Colleges | 3/27/2023 | Morning Glory HF1 | 1 invoice | never completed | Edifact file for 1 invoice with 62 POLs | Early morning 8-8:15am. Never completed. Ticket with EBSCO. Found the issue. edifact file was invalid. Here it was difficult to understand since there was no log as the job didn't complete. | |
| 23 | 5 Colleges | 3/27/3023 | Sara Colglazier, Jennifer Eustis | Morning Glory HF1 | 27 | all records discarded | match on instance sys contr no and create holdings and items | Error: Date when processing started is not set, expected snapshot status is PARSING_IN_PROGRESS, actual - FILE_UPLOADED. When the file was uploaded a 2nd time right after, it completed with no errors. |
| 24 | 5 colleges | 3/28/2023 | MG HF1 | 1000 | early morning 1min. mid afternoon 18 min. | match on srs 999 $i and update srs bib | I did 8 jobs between 8:45am and 10:26 that each took about 1-2 min. There were single record imports and the like. The afternoon one, 1:31 - 1:49pm was when there were NO single record imports occurring. Not sure how busy FOLIO was really - perhaps with circulation | |
| 25 | University of Chicago | 3/31/2023 | Christie Thomas | Nolana | 1 | never completed | match on 035$z to Instance OCLC number, instance status static match and holdings static match / for non matches match on 035$a to Instance OCLC number, instance status static match and holdings static match | Never finished in production. Completed as expected in uchicago staging environment in less than one minute. Job completed after 9.5 hours at 1:15 in the morning. |
| 26 | 5 Colleges | 4/3/2023 | Sara Colglazier | Morning Glory HF1 | 25 | 1st upload: did not move from 0%, so deleted after X minutes, 2nd upload no prob (under a minute, as expected)–this happened 3 times in a row, after the initial create ALL file was not a problem as well as the 1st of the 4 create additional HOL & Item | match on instance sys control number and create holdings and items | not especially, just normal late Monday afternoon activity |
| 27 | Villanova University | 4/3/2023 | Jesse Flavin | Nolana | 1 | Tried twice to import a single record—never made any progress. The job was manually stopped each time after around 10 minutes. | Update instance and holdings after matching on the POL (this job has successfully been run previously in Nolana) | Late morning: 11:04 & 11:27. Not aware of any heavy use at the time. |
| 28 | Villanova University | 4/4/2023 | David Burke | Nolana | 2714 | 23 minutes | simple load, creating instance, holdings, and item | run 8:30 in the morning, EST. Trying to avoid heavy use period. |
| 29 | Villanova University | 4/4/2023 | David Burke | Nolana | 91 | 3 minutes | Same as above, but much smaller file | About 9:00 AM. Not aware of any traffic |
| 30 | Missouri State University | 4/4/2023 | Nolana | 482 | 19 minutes | Look for match based on 001/HRID. If none, create instance only | 1:15 PM. No other import activity | |
| 31 | Villanova University | 4/4/2023 | David Burke | Nolana | 512 | 31 minutes | simple record load | 10:13 AM |
| 32 | 5 Colleges | 4/5/2023 | Nolana | 1 | Never finished | Inventory single record import | 2:03 | |
| 33 | Missouri State University | 4/6/2023 | Nolana | 666 | 18 minutes | Look for match based on 001/HRID. If none, create instance only | 7:30am, before the library opened so no other traffic | |
| 34 | Missouri State University | 4/6/2023 | Nolana | 214 | 1 minute | Match based on 001/HRID. If found, update the instance only | 7:51am. Compare elapsed time to previous entry. | |
| 35 | 5 Colleges | 4/6/2023 | Nolana | 1 | never finished | Inventory single record import | started around 9. This wasn't the 1st ISRI. | |
| 36 | University of Missouri System | 4/6/2023 | Seth Huber | Nolana | 123 | 9 minutes; 1 error returned "org.folio.processing.exceptions.MatchingException: Found multiple records matching specified conditions," even though the 035 matchpoint only occurs once in catalog | Match based on 035 prefix, no action if no match, update MARC bibliographic if match | early afternoon, no other jobs processing (aside from normal single record imports) |
| 37 | University of Missouri System | 4/6/2023 | Seth Huber | Nolana | 2061 | 19 minutes | Match based on 035, update if match, create MARC/holdings/item if no match | Same as row 36 |
| 38 | 5 Colleges | 4/6/2023 | Nolana | 1 | 22 min | inventory single record import | This hasn't happened before. Since Nolana all imports are taking longer and ISRI's are either hanging or taking a ridiculous amount of time. | |
| 39 | University of Missouri System | 4/6/2023 | Seth Huber | Nolana | 493 | 17 minutes | Match on instance HRID, suppress matching records | late afternoon, around 4PM CST |
| 40 | 5 Colleges | 4/10/2023 | Nolana | 35 | Never completed. io.vertx.core.impl.NoStackTraceThrowable: No such Tenant fs00001006 | That same file was run with the same job profile (create inventory) a couples later. This error didn't appear. There was just No record. This same file using the same job profile completed successfully in our Sandbox. Reported to hosting vendor | Late/early evening. No activity. | |
| 41 | University of Missouri System | 4/11/2023 | Seth Huber | Nolana | 162 | 7 minutes; 22 errors returned "org.folio.processing.exceptions.MatchingException: Found multiple records matching specified conditions," even though the 035 matchpoint only occurs once in catalog | Match on 035 (OCoLC)*, create holdings/items for no match | Late morning, 10:22AM CST |
| 42 | Missouri State University | 4/11/2023 | Nolana | 7 5 | 6 minutes 24 minutes | Match on 035/OCLC (there were no matches); create I/H/I. | Late morning. No other DI activity. Usually my batches take less than a minute. This is the second time since upgrading to Nolana that I have had a batch take such a long time. No other processes happening (at least to my knowledge). | |
| 43 | 5 Colleges | 4/15/2023 | Nolana | 3000 | Create srs and inventory using default profile | 14 min | This is double the amount of time and doing less that what we did in Morning Glory. | |
| 44 | 5 Colleges | 4/15/2023 | Nolana | 3000 | add holdings and items match on instance system control number | 20 min | This is again so much more time than in MG. | |
| 45 | Missouri State University | 4/20/2023 | Monica Arnold | Nolana | 1000 | Update instances based on 001/HRID | 10 min | Early afternoon, about 1:15pm. The same import in our test system took 3 minutes. |
| 46 | Michigan State University | 5/15/23 | Nolana | 1 | Update SRS / Instance; create holdings & item; match on Instance HRID | 2 min | About 9:23 AM; no concurrent jobs running. | |
| 47 | University of Missouri System | 5/15/23 | Seth Huber | Nolana | 834 | 1 hr. 47 minutes | Match on 035 (OCoLC)*, create holdings/items for no match | All attempts started early morning; first 3 sat at 0% and were manually stopped after 10-15 minutes |