2024-03-11 Reporting SIG Meeting notes









Arthur AguileraUniversity of Colorado, Boulder
Eliana LimaFenway Library Organization
xSharon BeltaineCornell University
Eric LuhrsLehigh University

Erin BlockUniversity of Colorado, Boulder
Lisa McCollLehigh University

Kathleen Moore
xLinda MillerCornell University

Shannon BurkeTexas A&MxNassib NassarIndex Data

Suzette CanedaStanford UniversityxElena O'MalleyEmerson
xDung-Lan ChenSkidmore College
Tod OlsonUniversity of Chicago

Lloyd ChittendenMarmot
Jean PajerekCornell University

Tim DannayMount Holyoke College
Kimberly PamplinTexas A&M University 

Axel DoerrerUniversity MainzxScott PerryUniversity of Chicago

Shelley DoljackStanford UniversityxNatalya PikulikCornell University
xStefan DombekLeipzig UniversityxEmily SanfordMichigan State University
xJennifer EustisU. Massachusetts Amherst / Five College
Bob ScheierHoly Cross

Lynne ForsWellesley College
Vandana ShahCornell University

Lisa FurubottenTexas A&M
Linnea ShiehStanford University

Alissa HafeleStanford University
Susie SkowronekOakland University

Kara HartWellesley CollegexKimberly SmithMiddle Tennessee State University
xCorrie HutchinsonIndex Data
Clare SpitzerStanford University

Jamie JesanisMCPHS
Amelia SuttonU. Massachusetts

Jeanette KalchikStanford University
Simona TabacaruTexas A&M

Harry KaplanianEbsco
Huey-Ning Tan

Stanford University

Tim KiserMichigan State University
Vitus TangStanford University
xKevin KishimotoStanford University
Christie ThomasUniversity of Chicago

Ingolf KusshbzxIrina TrapidoStanford University

Alexander LaoStanford University
Catherine TuohyEmmanuel College

Joanne LearyCornell UniversityxPatrick WaiteU. Mass Amherst

Joseph MolloySpokane Public Library

Discussion Items




Attendance & NotesSharon

Attendance & Notes

  • Today's attendance-taker: Linda (or substitute)
  • Today's note-takers:  Team Leads for project updates



Changes in Reporting SIG Meetings

  • Reporting SIG meetings will take place at 11am ET on the 2nd and 4th Mondays of the month only starting Monday February 12, 2024
  • "Workshopping your queries" - part of each Reporting SIG meeting will be devoted to time to work through any query questions you may have

Cookbook Documentation Updates (Stefan)

Impacts of New Fields and Features (Sharon)

Upcoming meeting topics (tentative)

  • Open Refine
  • LDP to Metadb Transition Overview

Any new members?

  • Welcome/introductions

How to find our latest recordings

SIG Recruitment

We will need to be recruiting for a variety of roles in the coming months. Please consider whether you would be interested. Reach out to Scott Perry  or Sharon Beltaine with any questions.

  • Representative for the Documentation Working Group 
  • Query developers

Metadb UpdateNassib Nassar

Metadb Update:


Metadb 1.2

  • Metadb version 1.2 is the latest stable release
  • See documentation at https://metadb.dev/doc/
  • LDP uses nightly data transfer, while Metadb does continuous, streaming, real-time updates to the data 
  • metadb.base_table stores a list of all the tables metadb (from FOLIO) is managing
  • metadb is designed to support more than one source of data
  • transformed boolean true means table is transformed from another table, e.g., parent_schema_name and parent_table_name
  • metadb.log is like a system log for metadb, reporting errors and warnings; you can call the mdblog function (see functions section) to see log messages from the last 24 hours
  • metadb.table_update shows information about the updating of MARC and derived tables (for now) and plans are for it to cover all tables in the future; good for troubleshooting
  • ALTER TABLE can be used by a system administrator to change the data type of a column to uuid if that is the preference (ERM, for instance, does not use the uuid type so they are inferred by metadb)
  • As of Metadb version 1.2, the default data type is text. You can still use VARCHAR in your SQL.
  • Metadb attaches to the FOLIO db replication system to provide the functionality of streaming data from the database to get changes; if something happens to break the stream of data, your devops team needs to resynchronize the data as a way of doing recovery on the data; the sync command reads the entire source database as a snapshot; then the devops team runs the endsync command to clean up the data; you can still use Metadb when the sync process is running, but you can't be sure the data is up-to-date until this process is complete; historical data is lost when the sync is broken; generally speaking, it is not necessary to stop the streaming when the FOLIO source database is getting updated to a new software release; Index data stops the sync and then does a resync

Metadb 1.3

  • new feature to important historical LDP data into the main tables in Metadb
  • performance improvements to the resynchronization process

WOLFCon 2024Sharon, Scott

Here is a summary of information about WOLFCon 2024 from Jesse Koennecke:

  • Location and Dates: Senate House, University of London, September 24-26, 2024
  • Call for Proposals Now Open: Got ideas about open-source to share? Talk about it at WOLFcon. Submit a presentation, panel, short talk, or pre-conference workshop. The deadline for submissions is March 31, 2024. Submit a session here.
  • Early Bird Registration Now Open: Join us at Senate House, University of London. September 24-26, 2024. Register now through July 31, 2024 for an early bird discounted rate.
  • Learn more about WOLFcon 2024: Want to learn more about the Open Library Foundation and WOLFcon? Be sure to visit our website where you can learn more about the foundation, members projects, communities, and the annual conference.

Reporting SIG Planning for WOLFCon 2024:

  • Please help us with topics and ideas for reporting at the WOLFCon 2024 conference! 
  • See ideas we have so far at  WolfCon 2024 Planning

Workshopping Queries

Stefan, Christie

Bring your questions on queries written for the LDP or Metadb so we can "workshop" them! This week we will focus on some new updates for derived tables related to finance from Stefan.

Project "Build Metadb query cookbook-style documentation"Stefan

The cookbooks are currently being created step by step and are located in the folio-analytics wiki.


FOLIO Product Council UpdateJennifer Eustis

2024-02-29 Product Council Agenda and Meeting Notes. At the last PC meeting, the Prioritization and Roadmap working group continued with their discussion on the test in Acquisitions SIG to implement a clearer method to tracking development. This is a prototype that PC will consider its viability for SIGs in general.

There's some movement on how to track changes in property names. More to come.

WOLFCon 2024! Think of registering and/or submitting.

3/14/2024 PC and SIG conveners meeting.

3/21/2024 PC Asia friendly time meeting 8pm EST

Recurring Items (Updated weekly, but not always discussed in meeting)

Review of In-Progress Projects (Recurring)
Review the release notes for FOLIO Analytics, LDP, LDLite, LDP Reporting App, ldpmarc, Metadb Projects (Recurring)
Updates and Query Demonstrations from Various Reporting Related Groups and Efforts Projects (Recurring)Community & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

D-A-CH Working Group (D-Reporting)

  • Ongoing topics
    • DBS statistics
      • JIRA-Issues, Rpt-Clusters
      • Gap analysis to statistic codes, user counts etc.
      • Identifying functions that we need for statistics in Germany but are not yet implemented in FOLIO
  • Meetings
    • Contact Stefan Dombek if you would like to get a calendar invitation

Metadb FOLIO User Group

  • Meetings
    • Contact Nassib Nassar if you would like to get a calendar invitation

Product Council

Reporting SIG Documentation Subgroup

  • Orchid documentation is live on https://docs.folio.org/docs/
  • Poppy documentation is in development
  • Additional Context
    • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

For all recent work on FOLIO Reporting SQL development: