FOLIO Production Library Import Statistics
When running large files in production, especially after migration to a new flower version of FOLIO, please add statistics for large data import jobs in your environment. The more statistics we can gather from production environments, the better data that we will have to compare to performance statistics gathered by the developers and the Performance Task Force.
Library | Date | Reporter | FOLIO version | Number of records | Elapsed time | Summary of the job | Besides import, how busy was FOLIO at the time? | |
|---|---|---|---|---|---|---|---|---|
| 1 | Cornell | 3/9/23 | @Jenn Colt | Nolana | 1000 | < 5 mins | updated based on 001, simple marc update | run during open hours, but on the early/late edges |
| 2 | Cornell | 3/9/23 | @Jenn Colt | Nolana | 5000 | ca. 40 mins | updated based on 035, profile contains both marc and instance updates, plus holdings | run during open hours, but on the early/late edges |
| 3 | 5 Colleges | 3/14/23 | @Jennifer Eustis | Morning Glory | 349 | 3 min | updated marc srs bib, match on 999 $i, | Not busy - snow day |
| 4 | 5 Colleges | 3/10/23 | @Sara Colglazier | Morning Glory HF1 | 3000 | 7 min | create marc srs, instance, holdings, item | run during open hours, but after main work hours for most (just after 5:30 pm on a Friday) |
| 5 | 5 Colleges | 3/15/23 | @Jennifer Eustis | Morning Glory | 193 | 1 min | update marc srs bib match on 999 $i | 9:24 am |
| 6 | 5 Colleges | 3/16/23 | @Jennifer Eustis | Nolana - dry run | 4 | 4 min | match on instance sys ctrl no, if no match create FOLIO inventory & srs, modify marc to remove 877, 876, 852 | This same file and same DI job profile in our MG production was done in 1min |
| 7 | 5 colleges | 3/16/23 | @Jennifer Eustis | Nolana - dry run | 7 | 8 min | match on instance sys ctrl no, if no match create FOLIO inventory & srs, modify marc to remove 877, 876, 852 | This same file and same DI job in our MG production was done in 1 min. |
| 8 | Wellesley | 3/15/23 | @Lynne Fors | Nolana - dry run | 100,000 | 5 hours, 3 minutes (3:03pm-8:11pm) | create marc srs, instance, holdings, modify marc srs | Not very busy. We have a limited number of users who have access to our dry run environment. Unsure if our dry run tenant is hosted with others and if there were impacts there. No idea as to how the dry run is provisioned. |
| 9 | Michigan State | 3/17/23 | @Joshua Barton | Morning Glory HF1 | 270 | ca. 17 mins | update instance/srs based on match of incoming 947 to instance system control number; if no match creates new instance/srs and holdings | run mid-afternoon on a work day with other jobs (mostly consisting of single records via DI) being run every 2-5 minutes |
| 10 | 5 Colleges | 3/17/2023 | @Sara Colglazier | Morning Glory HF1 | 4000 | 9 min | on no match on sys control no create marc srs, instance, holdings, item | run during closed hours (5:28 pm on a Friday, Spring Break) |
| 11 | 5 colleges | 3/17/2023 | @Jennifer Eustis | Nolana dry run | 8574 | 80 min | update marc srs match on 999 $i | run overnight - no activity |
| 12 | 5 Colleges | 3/17/2023 | @Jennifer Eustis | Morning Glory HF1 | 2000 | never finished | on no match on sys control no create marc srs, instance, holdings, item with marc modify at end | started 5:58. File contained duplicates. Job never completed - ticket with hosting vendor created |
| 13 | University of Chicago | 3/17/2023 | @Christie Thomas | Nolana | 1150 | <2 minutes | no match points, create instance and create holdings for each record. | run mid-day during normal operating hours |
| 14 | University of Chicago | 3/15/2023 | @Christie Thomas | Nolana | 2401 | <2 minutes | no match points, create instance and create holdings for each record. | run over the weekend in the evening when there would be minimal activity in the system |
| 15 | 5 Colleges | 3/22/2023 | @Jennifer Eustis | Nolana dry run | 9000 | 17 min | on no match on sys control no create marc srs, instance, holdings, item with marc modify at end |
|
| 16 | 5 Colleges | 3/23/2023 | @Jennifer Eustis | Morning Glory HF1 | 4000 | 7 min | update srs marc bib match on 999 $i | completed with errors but file completed. Problem in FOLIO, 2 instances linked to same marc srs bib |
| 17 | Villanova | 3/23/2023 | @Jesse Flavin | Nolana | 25 | 11 min | update instance, holdings, and item matching on UUID | run at 4 pm |
| 18 | 5 Colleges | 3/23 | @Jennifer Eustis | Nolana dry run | 9000 | 9 min | match on system control no, create holdings and items | not busy in dry run |
| 19 | 5 Colleges | 3/23 | @Jennifer Eustis | Nolana dry run | 12000 | didn't finish | on no match on sys control no create marc srs, instance, holdings, item with marc modify at end | No error message. not busy |
| 20 | University of Chicago | 3/25/2023 | @Christie Thomas | Nolana | 1247 | < 2 minutes in production environment | match on 035 to instance OCLC, instance status, and holdings type = electronic - update instance and holdings. If not match, create new | 7 minutes in test / staging environment. Loaded into producti9on mid-day on weekend with little other activity in the system. |
| 21 | Wellesley | 3/27/2023 | @Lynne Fors | Morning Glory HF1 | 128 | 8 minutes | match 999ff$i to Instance UUID, update statistical codes; next, match 999ff$s to MARC SRS 999ff$s and modify MARC to delete 903 and 949 fields | Spring Break, not busy; Started at 10:31 am; Completed at 10:39 am |
| 22 | 5 Colleges | 3/27/2023 | @Jennifer Eustis | Morning Glory HF1 | 1 invoice | never completed | Edifact file for 1 invoice with 62 POLs | Early morning 8-8:15am. Never completed. Ticket with EBSCO. Found the issue. edifact file was invalid. Here it was difficult to understand since there was no log as the job didn't complete. |
| 23 | 5 Colleges | 3/27/3023 | Sara Colglazier, @Jennifer Eustis | Morning Glory HF1 | 27 | all records discarded | match on instance sys contr no and create holdings and items | Error: Date when processing started is not set, expected snapshot status is PARSING_IN_PROGRESS, actual - FILE_UPLOADED. When the file was uploaded a 2nd time right after, it completed with no errors. |
| 24 | 5 colleges | 3/28/2023 | @Jennifer Eustis | MG HF1 | 1000 | early morning 1min. mid afternoon 18 min. | match on srs 999 $i and update srs bib | I did 8 jobs between 8:45am and 10:26 that each took about 1-2 min. There were single record imports and the like. The afternoon one, 1:31 - 1:49pm was when there were NO single record imports occurring. Not sure how busy FOLIO was really - perhaps with circulation |
| 25 | University of Chicago | 3/31/2023 | @Christie Thomas | Nolana | 1 | never completed | match on 035$z to Instance OCLC number, instance status static match and holdings static match / for non matches match on 035$a to Instance OCLC number, instance status static match and holdings static match | Never finished in production. Completed as expected in uchicago staging environment in less than one minute. Job completed after 9.5 hours at 1:15 in the morning. |
| 26 | 5 Colleges | 4/3/2023 | @Sara Colglazier | Morning Glory HF1 | 25 | 1st upload: did not move from 0%, so deleted after X minutes, 2nd upload no prob (under a minute, as expected)–this happened 3 times in a row, after the initial create ALL file was not a problem as well as the 1st of the 4 create additional HOL & Item | match on instance sys control number and create holdings and items | not especially, just normal late Monday afternoon activity |
| 27 | Villanova University | 4/3/2023 | @Jesse Flavin | Nolana | 1 | Tried twice to import a single record—never made any progress. The job was manually stopped each time after around 10 minutes. | Update instance and holdings after matching on the POL (this job has successfully been run previously in Nolana) | Late morning: 11:04 & 11:27. Not aware of any heavy use at the time. |
| 28 | Villanova University | 4/4/2023 | David Burke | Nolana | 2714 | 23 minutes | simple load, creating instance, holdings, and item | run 8:30 in the morning, EST. Trying to avoid heavy use period. |
| 29 | Villanova University | 4/4/2023 | David Burke | Nolana | 91 | 3 minutes | Same as above, but much smaller file | About 9:00 AM. Not aware of any traffic |
| 30 | Missouri State University | 4/4/2023 | @Monica Arnold | Nolana | 482 | 19 minutes | Look for match based on 001/HRID. If none, create instance only | 1:15 PM. No other import activity |
| 31 | Villanova University | 4/4/2023 | David Burke | Nolana | 512 | 31 minutes | simple record load | 10:13 AM |
| 32 | 5 Colleges | 4/5/2023 | @Jennifer Eustis | Nolana | 1 | Never finished | Inventory single record import | 2:03 |
| 33 | Missouri State University | 4/6/2023 | @Monica Arnold | Nolana | 666 | 18 minutes | Look for match based on 001/HRID. If none, create instance only | 7:30am, before the library opened so no other traffic |
| 34 | Missouri State University | 4/6/2023 | @Monica Arnold | Nolana | 214 | 1 minute | Match based on 001/HRID. If found, update the instance only | 7:51am. Compare elapsed time to previous entry. |
| 35 | 5 Colleges | 4/6/2023 | @Jennifer Eustis | Nolana | 1 | never finished | Inventory single record import | started around 9. This wasn't the 1st ISRI. |
| 36 | University of Missouri System | 4/6/2023 | @Seth Huber | Nolana | 123 | 9 minutes; 1 error returned "org.folio.processing.exceptions.MatchingException: Found multiple records matching specified conditions," even though the 035 matchpoint only occurs once in catalog | Match based on 035 prefix, no action if no match, update MARC bibliographic if match | early afternoon, no other jobs processing (aside from normal single record imports) |
| 37 | University of Missouri System | 4/6/2023 | @Seth Huber | Nolana | 2061 | 19 minutes | Match based on 035, update if match, create MARC/holdings/item if no match | Same as row 36 |
| 38 | 5 Colleges | 4/6/2023 |