2023-05-08 Reporting SIG Meeting notes









xArthur AguileraUniversity of Colorado, BoulderxEliana LimaFenway Library Organization
xSharon BeltaineCornell University
Eric LuhrsLehigh University

Erin BlockUniversity of Colorado, BoulderxLinda MillerCornell University

Nancy BolducCornell University
Nassib NassarIndex Data
xShannon BurkeTexas A&MxElena O'MalleyEmerson

Suzette CanedaStanford UniversityxTod OlsonUniversity of Chicago

Lloyd ChittendenMarmotxJean PajerekCornell University

Tim DannayMount Holyoke CollegexKimberly PamplinTexas A&M University 
xAxel DoerrerUniversity MainzxScott PerryUniversity of Chicago

Shelley DoljackStanford University
Natalya PikulikCornell University

Stefan DombekLeipzig University
Bob ScheierHoly Cross
xJennifer EustisU. Massachusetts Amherst / Five CollegexVandana ShahCornell University

Lynne ForsWellesley CollegexKimberly SmithMiddle Tennessee State University

Lisa FurubottenTexas A&M
Linnea ShiehStanford University

Alissa HafeleStanford University
Clare SpitzerStanford University
xKara HartWellesley College
Amelia SuttonU. Massachusetts

Corrie HutchinsonIndex Data
Simona TabacaruTexas A&M

Jamie JesanisWellesley College
Huey-Ning Tan

Stanford University

Jeanette KalchikStanford University
Vitus TangStanford University

Kevin KishimotoStanford University
Irina TrapidoStanford University

Ingolf Kusshbz
Catherine TuohyEmmanuel College

Alexander LaoStanford University
Kevin WalkerThe University of Alabama

Joanne LearyCornell UniversityxAngela ZossDuke University

Discussion Items




Attendance & NotesSharon

Attendance & Notes

  • Today's attendance-taker: Linda (or substitute)
  • Today's note-takers:  Team Leads for project updates

Announcements /


Upcoming meeting topics

  • May 18: SIG Onboarding Part 2 (Development Work) - tentative

Combining MM and RA/UM subgroups

Reporting Implementation Table

Any new members?

  • Welcome/introductions

How to find our latest recordings

(Always) Recruiting New Query Developers

  • The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

Demonstrations of ldpmarcJennifer, Angela
  • https://librarydataplatform.org/ (LDP1, Metadb, LDLite)
  • https://librarydataplatform.org/resources/
  • https://github.com/5-C-Folio/LDlite-Queries
  • At the Five Colleges (5C), they have LDLite on a server and a systems librarian runs it
  • Have customized to get ERM
  • Everything except for inventory is downloaded once per night; inventory and SRS are once a week over the weekend
  • LDLite is not real-time, just a snapshot
  • LDLite transformations are similar to Metadb, so the data structure looks like Metadb - two tables, one with the original JSON data, one with the data transformed into a flat tables
  • In LDLite, the data transformation drills into the arrays and objects, so many of the derived tables we use in Metadb are not useful
  • marctab table is the result of the ldpmarc transformation, converting MARC to a tabular data format
  • (marc_records_lb is parsed JSON of SRS MARC Bib)
  • marctab doesn't have state (old or active), so need to go to the original MARC JSON for that (records table)
  • What kind of queries does 5C run?
    • looking for orphan instances (instances without holdings or items attached); data import creates a lot of orphans, something is wrong; these orphans still show up in discovery, which is confusing for users
  • queries look like LDP1 and Metadb queries - a lot of left joins
  • how are you connecting using DBeaver?
    • the python script is run by an administrator from a server, and then it pushes data into a shared PostgreSQL database, and most people connect to that directly through DBeaver
  • Is LDLite similar to LDP that you run the whole database refresh once a night?
    • Our systems librarian set it up to run an update overnight for everything except inventory and marc SRS; those are big and take a long time, and if they hang it will grind FOLIO to a halt
  • What scale does this seem to be suitable for?
    • Not great at large scale
    • People get confused that it's not current data, and people who don't know SQL still can't use it
    • with billing, especially hard because can't even really get real time data
    • 5C actually doesn't pull everything
    • Even with LDLite separate database, will have some queries that hang up the reporting database, will need to kill it
    • Are the database tables indexed like LDP1 and Metadb?
  • Inventory and SRS often have the same thing; might be better to use data from inventory when possible

Recurring Items (Updated weekly, but not always discussed in meeting)

Review of In-Progress Projects (Recurring)
Review the release notes for FOLIO Analytics, LDP1, LDLite, LDP Reporting App, ldpmarc, Metadb Projects (Recurring)
Updates and Query Demonstrations from Various Reporting Related Groups and Efforts Projects (Recurring)Community & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

RA/UM Working Group

MM Working Group

ERM Working Group

  • Current topics
    • Reporting for FOLIO Apps
      • Open Access
      • eUsage
      • eHoldings
  • Meetings are bi-weekly on tuesdays 11am ET alternating with ACQ Working Group
    • Next meeting will be at 9th, May
    • Contact Stefan Dombek if you would like to get a calendar invitation

ACQ Working Group

Reporting SIG Documentation Subgroup

  • Morning Glory documentation is live on https://docs.folio.org/docs/
  • Nolana documentation is in review
  • Orchid documentation will be in progress soon, and plans are underway to include beta-level documentation for Metadb
  • Additional Context
    • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

External Statistics Working Group

  • no updates currently
  • new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
  • New organizational structure for External Statistics reports
    • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
    • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
    • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
    • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

D-A-CH Working Group (D-Reporting)

  • Discussion on draft statement: "Hosting and Support"
    • The members of D-Reporting need more time to review the document and provide feedback
  • Ongoing topics
    • Onboarding training
    • DBS statistics
      • JIRA-Issues, Rpt-Clusters
      • Gap analysis to statistic codes, user counts etc.
      • Identifying functions that we need for statistics in Germany but are not yet implemented in FOLIO
  • Meetings are currently held by appointment
    • Last meeting: 8th, May
    • Next meeting will be at 15th, May (1 pm, CEST)
    • Contact Stefan Dombek if you would like to get a calendar invitation

Product Council

For all recent work on FOLIO Reporting SQL development: