2022-04-07 Reporting SIG Meeting notes









xArthur AguileraUniversity of Colorado, BoulderxLinda MillerCornell University
xSharon BeltaineCornell UniversityxNassib NassarIndex Data

Erin BlockUniversity of Colorado, BoulderxElena O'MalleyEmerson

Nancy BolducCornell University
Tod OlsonUniversity of Chicago
xSuzette CanedaStanford UniversityxJean PajerekCornell University

Lloyd ChittendenMarmot
Michael PatrickThe University of Alabama

Axel DoerrerUniversity MainzxEric PenningtonTexas A&M

Shelley DoljackStanford UniversityxScott PerryUniversity of Chicago

Stefan DombekLeipzig UniversityxNatalya PikulikCornell University

Jennifer EustisU. Massachusetts Amherst / Five College
Vandana ShahCornell University
xAlissa HafeleStanford UniversityxLinnea ShiehStanford University
xJeanette KalchikStanford UniversityxClare SpitzerStanford University
xKevin KishimotoStanford University
Amelia SuttonU. Massachusetts

Ingolf KusshbzxSimona TabacaruTexas A&M

Kim LaineCornell University
Huey-Ning Tan

Stanford University

xJoanne LearyCornell UniversityxVitus TangStanford University
xEliana LimaFenway Library OrganizationxIrina TrapidoStanford University

Eric LuhrsLehigh University
Kevin WalkerThe University of Alabama

xAngela ZossDuke University

Discussion Items




Attendance & NotesAngela

Attendance & Notes

  • Today's attendance-taker: ?
  • Today's note-takers:  Team Leads for project updates
LDP query performanceNassib

 Basic tips for faster queries in LDP databases

  • Indexes
  • Vacuum analyze
  • Temporary tables
  • Breaking apart large queries
  • Use NOT EXISTS instead of NOT IN
  • Simple column names
  • System configuration

Announcements /


Trying to find our latest recordings

Recruiting New Query Developers

  • The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

Open Discussion
  • Kiwi demo site has an LDP app; should we use that to learn about creating reports?
    • LDP app is easy to use and is intuitive to use, but it's simple and can only do a few things
    • LDP app draws on same information as the LDP, one table at a time
    • right now, if you don't use the LDP app, you have to connect to the database through another software application, and you probably need to write SQL to query the data
    • you probably don't want to have all of the library staff connecting to the database directly
    • the LDP app is a new app in Kiwi expands access to the LDP database to a larger group of people
    • it avoids the requirement of having to write SQL
    • this current version is pretty limited, but there are mock-ups to support table joins and really be a visual query builder
    • the next feature coming is to be able to run reporting queries created by FOLIO reporting community; the app will allow you to fill in parameters and just generally give more staff access to running reports
    • don't have dedicated staff for development; different developers are available at different times
    • also able to access derived tables, which is a custom table that can join multiple FOLIO tables, or a local table (something created by someone who has direct access to the LDP database)
    • you can thus program the LDP app by creating useful tables in the LDP database that the app can then see
    • you say it will be able to run queries from GitHub, can we point to our local query repository?
      • yes
    • can we edit SQL in FOLIO?
      • yes, but you probably want to be careful
      • if you have 100 people running custom SQL queries, if they don't have SQL training, there could be performance issues
      • for example, someone could do a CROSS JOIN without any filters; if those tables are large, the query could take days to run and fill up a terabyte of data
      • but in principle, having a text box where you could type SQL is not a problem, but might want to be careful about permissions
    • requests for upcoming meeting topics?

Updates and Query Demonstrations from Various Reporting Related Groups and EffortsCommunity & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

RA/UM Working Group

MM Working Group

  • Meetings are 1st Tuesday of the month, 12-1pm ET via zoom using the usual FOLIO password. Our lab sessions are open to everyone. Please bring your questions, examples, and comments about reporting and metadata.
  • We are still looking for help with derived tables and reviewers. See our sign up sheet.
  • In next our lab session, we will use the LDP instance identifiers and go through the process of porting this over to metadb, how to contribute a pull request with the changes and update the runlist.txt, and go through a review.

ERM Working Group

RM Working Group

Reporting SIG Documentation Subgroup

  • Honeysuckle documentation is live on https://docs.folio.org/docs/
  • Iris documentation is in progress, due December 15
  • Additional Context
    • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

External Statistics Working Group

  • no updates currently
  • new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
  • New organizational structure for External Statistics reports
    • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
    • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
    • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
    • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

Product Council

For all recent work on FOLIO Reporting SQL development:

Topics for Future MeetingsAll
  • Next: Reviewing a pull request 
  • How to deal with External Stats reports?
    • maybe subteam leads check in about that
    • probably wait until after Metadb conversion is more complete
  • Definitely do GitHub Desktop demonstration at a SIG meeting in future, probably about 15 minutes
  • Converting queries to dashboards: Sharon and Vandana (April 25)

Review and update Topics for Future Reporting SIG Meetings 

  • A test Action Item (Ingolf)