2022-10-31 - Data Migration Subgroup Agenda and Notes
- Darsi Rueda
Date
Attendees
Darsi Rueda jpnelson Carol Sterenberg Jeff Fleming Ian Walls Ingolf Kuss
Meeting Link
- https://zoom.us/j/204980147
- password: folio-lsp
Link to Recordings
Discussion items
Time | Item | Who | Notes |
|---|---|---|---|
| Migration status report | Jeremy | Stanford doing load at scale over last weekend, working on increasing throughput with Airflow. Increasing simultaneous dag runs from 3 to 5. Increased number of posts to 3 within each dag. And number of records bumped from 500 to 1500. Increased throughput borked okapi. So dialed it back to orig: 3 dag runs, still 5 postings. Some errors toward end of Saturday (loading 2 million ckey range total). Getting some connection timeouts, so held off to analyze during the week. Bottleneck on okapi side? a 50,000 record range (might not be 50K records in that range) takes average 22 mins. Longer if more holdings. Also need to provision enough disk space for airflow so can see all dag runs. Running a verify/remediation step making sure all instances exist, if not, re-post. We did see some cases in loads done during the past week of instances posted, but holdings/items didn’t post. Can’t run multiple dag runs when loading to SRS though, borks okapi Show and tell of dag, mapping, etc. | |
| Ingolf | Have 3 Vufind instances for FOLIO, will be hooking it up, need to write a connector. Upgrading a demo instance. Can’t directly upgrade from Kiwi to Morninglory (hazlecast trying to set itself up “fresh” but finds your old instances, has to modify the module, too complicated) | ||
| Jeffrey | Working on closed orders, redefining criteria, writing them to files and having IndexData load them. Open orders will go through apis (Jeff runs those), closed done by ID. Working on label printing. Zebra label printers. has to write the handshake, translate to zpl. (and this is all without FOLIO). Duke now planning to migrate to FOLIO in 2024. | ||