2022-10-20 Reporting SIG Meeting notes









xArthur AguileraUniversity of Colorado, Boulder
Eric LuhrsLehigh University
xSharon BeltaineCornell UniversityxLinda MillerCornell University

Erin BlockUniversity of Colorado, Boulder
Nassib NassarIndex Data

Nancy BolducCornell UniversityxElena O'MalleyEmerson

Shannon BurkeTexas A&M
Tod OlsonUniversity of Chicago

Suzette CanedaStanford UniversityxJean PajerekCornell University

Lloyd ChittendenMarmot
Michael PatrickThe University of Alabama

Tim DannayMount Holyoke College
Eric PenningtonTexas A&M

Axel DoerrerUniversity Mainz
Scott PerryUniversity of Chicago

Shelley DoljackStanford University
Natalya PikulikCornell University

Stefan DombekLeipzig University
Vandana ShahCornell University

Jennifer EustisU. Massachusetts Amherst / Five CollegexLinnea ShiehStanford University
       Lisa FurubottenTexas A&M
Clare SpitzerStanford University
xAlissa HafeleStanford University
Amelia SuttonU. Massachusetts
xJeanette KalchikStanford University
Simona TabacaruTexas A&M
xKevin KishimotoStanford University
Huey-Ning Tan

Stanford University

Ingolf Kusshbz
Vitus TangStanford University

Joanne LearyCornell UniversityxIrina TrapidoStanford University
xEliana LimaFenway Library Organization
Kevin WalkerThe University of Alabama
xAlexander LaoStanford UniversityxAngela ZossDuke University

xMike Hucka VisitorCalTech

xDarsi Rueda VisitorStanford University

Discussion Items




Attendance & NotesAngela

Attendance & Notes

  • Today's attendance-taker: Linda (or substitute)
  • Today's note-takers:  Team Leads for project updates
Presentation on FoliageMike Hucka


  • Have been in FOLIO for over a year, and there are tasks staff need to do that aren't possible in FOLIO now
  • Staff asked for some help with automation
  • Written in Python, but it's a Desktop application
  • Does FOLIO operations over the network, but the users don't have to see that or use the API directly
  • Functionality:
    • Look up related records given identifiers
    • Change record field values in a list of records (e.g., change locations of a batch all at once)
    • Delete entire instance/holdings/item hierarchies
    • Delete "phantom loans" - happen when you delete an item that was previously loaned
    • List all UUID values of a given type
  • Demo
  • Uses a web browser window as the user interface, but it's really just running on your machine; looks the same on any platform
  • Will be asked to give your credentials the first time you run it, but not later times
  • Can see the output either in summary format or the raw JSON
  • The tool populates drop-down menus with the real options for, e.g., temporary location so you don't have to look it up
  • Has some safety checks so it won't make a change if it doesn't make sense for what was requested
  • For any operation that changes or deletes a record, the program saves a backup copy of the last version of the records on your machine if you need to revert
  • Links:
  • The code currently has some things hard-wired in for CalTech, so not fully configurable, but other sites could either edit that code or it could be rewritten to be configurable
  • Questions:
    • How many users?
      • About 10 on a regular basis, a handful who use it daily (library only has between 50 and 100 staff)
    • Have you shared this with the data migration group?
      • Not directly, but they may have seen it elsewhere
      • Reporting SIG is first group to reach out
      • Staff don't know how to use the API, and these operations would be really laborious without this tool
      • Would think any institution would want to know about this for batch processing and data quality
      • This is an interim solution, and probably didn't need this to be as slick as it is, but the goal is for FOLIO to eventually do this work
      • So, this tool isn't something that will be supported indefinitely
    • Thought that the change records could be more powerful, but it turns out that changing records requires very specific information (which requires some things to be hard coded into the tool and thus makes the tool more fragile/less generalizable), and also there are many things that can't be changed (or don't need to be changed in bulk); locations are the main thing that changes a lot

Announcements /


Hoping to put together a FOLIO Forum 

  • FOLIO Forums are a great outreach opportunity - a way to share work with a large community and support the 100% open-source LDP project
  • Nassib is interested in putting together a panel for something soon
  • Could be a general reporting topic, or something more specific (e.g., LDLite, using LDP for inventory clean up)
  • Anyone interested? Message Nassib or comment on LDP Slack

Upcoming Events

How to find our latest recordings

(Always) Recruiting New Query Developers

  • The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

Review of In-Progress Projects
Announcement about Item State Architecture

From Erin Nettifee:

Hi all - I am spinning up a small group to work on requirements for two features - https://folio-org.atlassian.net/browse/UXPROD-1530 and https://folio-org.atlassian.net/browse/UXPROD-1590 - meant to establish the other two parts of the item state architecture.

I want to be clear that I am not taking over as the item state PO - I don't have the bandwidth to do that permanently, and I really only have capacity for the next 3-5 months to do more PO work - if a more permanent PO steps forward, I will gladly hand this over. I also want to be clear that there is no development team for this work, so all we are doing is working on requirements, and we have no guarantee as to when development might happen. But there are very few requirements for either jira right now, so no matter what hopefully we can make it so that when resources do appear they don't have to start the work from scratch.

The commitment is likely to be something like a bi-weekly meeting plus additional ad-hoc work depending on the topic, for a limited period of time (the next 3-5 months.)

Demo of LDLite in Google ColabAngelaFinishing up our LDLite training
Updates and Query Demonstrations from Various Reporting Related Groups and EffortsCommunity & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

RA/UM Working Group

MM Working Group

  • Meetings are 1st Tuesday of the month, 12-1pm ET via zoom using the usual FOLIO password. Our lab sessions are open to everyone. Please bring your questions, examples, and comments about reporting and metadata.
  • We are still looking for reviewers and testers for ldpmarc.
  • We have moved our Slack channel to the LDP Slack Workspace.

ERM Working Group

  • FOLIO Data Model Training (in progress)
  • ERM Query Development Status
  • Coming soon: Open Access
    • Report requests are being collected in the OA SIG Link
    • The test environment from the SIG Reporting will get the app OA
    • Test data will imported
  • Meetings are bi-weekly on tuesdays 11am ET alternating with RM Working Group
    • Next meeting will be at 25th, Oct
    • Contact Stefan Dombek if you would like to get a calendar invitation

RM Working Group

Reporting SIG Documentation Subgroup

  • Lotus documentation is live on https://docs.folio.org/docs/
  • Morning Glory documentation is complete and submitted
  • Nolana documentation will be in progress soon
  • Additional Context
    • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

External Statistics Working Group

  • no updates currently
  • new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
  • New organizational structure for External Statistics reports
    • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
    • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
    • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
    • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

Product Council

For all recent work on FOLIO Reporting SQL development:

Topics for Future MeetingsAll
  • How to deal with External Stats reports?
    • maybe subteam leads check in about that
    • probably wait until after Metadb conversion is more complete
  • ask for presenters: hosting experiences from implementers
  • Annual Reporting Goals
    • (in progress) Support the transition from LDP to Metadb (e.g., update derived table and report queries, update documentation, outreach, new training)
    • (ready to start) Developing training/onboarding for new SIG members/report users (esp. FOLIO-specific data model and transformation stuff)
    • Improve communication between SIG and developers of apps so we hear about data model changes in advance
    • continued advocacy on part of the SIG to governance groups
    • (ready to start) Review JIRA issues, clean up, revisit strategy for JIRA
  • Regular review of Milestones
  • Exploring new recruitment/onboarding strategies (e.g., buddy system)
  • Demo latest version of LDP app, any new features?
  • Training: Using APIs
  • More work on asynchronous collaboration, how to engage in discussions and question answering more broadly
    • consider connecting discuss.folio.org with a Slack channel, to make sure any forum topics get highlighted on Slack as well?
  • Open question: should we update FOLIO LDP1-based Reporting First Implementers Grid

Review and update Topics for Future Reporting SIG Meetings 

  • A test Action Item (Ingolf)