2021-11-29 Reporting SIG Meeting notes









xArthur AguileraUniversity of Colorado, BoulderxLinda MillerCornell University
xSharon BeltaineCornell University
Nassib NassarIndex Data

Nancy BolducCornell UniversityxElena O'MalleyEmerson
xLloyd ChittendenMarmotxTod OlsonUniversity of Chicago
regretsAxel DoerrerUniversity MainzxJean PajerekCornell University

Shelley DoljackStanford UniversityxMichael PatrickThe University of Alabama


Stefan DombekLeipzig University
Eric PenningtonTexas A&M
xJennifer EustisU. Massachusetts Amherst / Five College
Scott PerryUniversity of Chicago
xAlissa HafeleStanford UniversityxNatalya PikulikCornell University

Ingolf KusshbzxVandana ShahCornell University

Kim LaineCornell UniversityxAmelia SuttonU. Massachusetts

Joanne LearyCornell University
Simona TabacaruTexas A&M
xEliana LimaFenway Library Organization
Kevin WalkerThe University of Alabama
xEric LuhrsLehigh UniversityxAngela ZossDuke University

Discussion Items




Attendance & NotesAngela

Attendance & Notes

  • Today's attendance-taker: Linda Miller
  • Today's note-takers:  Team Leads for project updates

Announcements /


Reporting documentation is live!

Table conversion from LDP to Metadb

  • Working on a spreadsheet that matches current LDP tables up with new Metadb tables, which should make it easy to convert existing LDP queries to work with Metadb

Reporting SIG Holiday Schedule

  • Plan to cancel December 27 and January 3

Recruiting New Query Developers

  • The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.
Planning for WOLFcon 2022Angela

Request to PC:

  • WOLFcon 2022 planning underway - and we need SIG/Council input.  August 31 through September 2 at Bucerius Law School in Hamburg, Germany.  See slides for details.  Need ideas for sessions; for instance, from community council: governance retrospective/discussion, quarterly community meeting, and budget and planning session.  The early ideas can be rough; they are useful for people planning budgets.  Thinking there should be sessions for developers, SMEs and SIG members, and library leadership.  Add items to FOLIO at WOLFcon by December 1.  Refinements will happen starting in December.

Some possible proposed sessions based on previous discussions:

Meeting NameDescriptionPotential meeting lengthPrimary Group/SIGSuggest by (your name/group)
Up and running with LDP (combine with Intro to FOLIO Analytics)

introductory demonstration of using SQL and the LDP app to query an LDP 

  • demystifying LDP software, LDP database, LDP app, FOLIO Analytics, LDLite, Metadb
  • mention hosting options (not a demo/tutorials)
  • audience might be very broad, including administrators
90FOLIO communityReporting SIG
LDLite for open source FOLIO reporting  Intro to APIs; FOLIO data delivery via APIs; API-based FOLIO reporting; Reporting the API with LDLite

introductory demonstration of using LDLite to extract FOLIO data via API. session will showcase how to set up LDLite, how to use LDLite to extract different parts of the FOLIO data model, and how LDLite transforms FOLIO data so that it is easy to query using shared FOLIO Analytics queries.

  • maybe broaden to a bigger session on using APIs; idiosyncrancies of some of the APIs
  • talked about this in ERM, Axel is working on some interesting scripts for LDLite, but decided not to host in FOLIO Analytics, will have a separate repository
  • API channel did talk about this, Intro to APIs; also, pros/cons of different software; what's good about using API vs. LDP vs. LDLite
60FOLIO communityReporting SIG
Intro to FOLIO Analyticswalkthrough of the FOLIO Analytics query repository

Implementing a reporting solution for FOLIO

panel discussion with people who have implemented any FOLIO reporting solutions (opportunity to give advice, answer questions)

  • Sharon willing to participate
  • (Combine with below)

Reporting lessons from FOLIO implementers

variation on above theme; target FOLIO implementing institutions, and asked them about their lessons learned as regards reporting systems

  • Sharon willing to participate
  • here's what we did (including hosting), here's what we learned
  • maybe bring in someone who can talk about self hosting?
  • hosting considerations

Hosting a reporting environment
  • demo of hosting (Nassib?)

Maintaining a commitment to openness

A FOLIO community discussion to address issues of openness in our community. Where has FOLIO failed to maintain its commitment to openness? How can the community renew this commitment and improve openness?

  • commitment to open source
  • commitment to transparency
  • what are the implications of institutions reducing development contributions? how are vendors making the choice between development to support contractual obligations and the priorities set by the community?
  • could divide into two: openness in communications, open source software development
  • participation from councils, but also representatives from Reporting SIG
  • one group to include might be documentation; how can we have documentation and transparency around changes to data model (e.g., in marc_srs, name of hrid and uuid were suddenly changed to "external_*", SMEs were not notified; marc_srs is being developed without SMEs); as a community, how can we do better? what are the changes, and how do those get communicated?
  • to what extent is FOLIO work being informed by change management/change control principles?
  • do all vendors have the same approach to change management?
  • how do data users learn about changes to data model/APIs; right now we just have to be watching the schemas
  • does/should this happen in the sprint reviews? right now they seem focus on demos of functionality, so maybe encourage discussions of schema changes
  • or maybe change should actually be proposed to the SIG and reviewed/approved
  • marc_srs development should have SME input? SIG?
  • should this be a WOLFcon session, or should it just be an issue raises to the councils soon? we could have a reporting subgroup to champion this work
  • on December 2, MM will be talking about marc_srs, how to get more info about how it's being developed and how changes are communicated, how it can be documented
  • have this as something we discuss in a future SIG meeting to think about possible advocacy work

Updates and Query Demonstrations from Various Reporting Related Groups and EffortsCommunity & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

RA/UM Working Group

MM Working Group

  • The zoom link for the meeting is https://umass-amherst.zoom.us/j/99254861410. Usual FOLIO password. Our lab sessions are open to everyone. Please bring your questions, examples, and comments about reporting and metadata.
  • Meetings are 1st Tuesday of the month 12-1pm ET

ERM Working Group

  • On going work on ERM data with LDlite
  • Rpt. "agreements by cancellation dates"
    • Creation of new additional derived tables in MetaDB for orders and invoices / invoice lines necessary and useful with regard to reporting software solutions. This assumes that MetDB maintains the corresponding tables relationally. This is currently not the case. Therefore the creation is paused. 
  • Group will pilot to document this areas documentation on mermaid.
    • Was suspended due to absence. First meeting on Nov. 30
  • ERM Prototype and Query Development Status
  • Bi-weekly schedule (next meeting 7th, Decmber 9:00 AM ET)
    • Zoomlink (password is the common zoom password in the FOLIO community)

RM Working Group

  • working on LDP to Metadb table mapping spreadsheet to prepare for transition to Metadb
  • subscription costs and subscription counts queries PR posted to folio-analytics
  • casting numbers: need to cast numbers related to finances as decimals, not money; money rounds up, which may throw totals off
  • looking at using https://mermaid-js.github.io/mermaid/#/ to document data models in finance areas of FOLIO 
  • exploring ways to make it easier to find options for reporting parameters in your LDP (e.g., a list of locations, item statuses, etc.)
  • several queries for RM completed, but still need documentation, testing, and review
  • for latest updates, see RM Prototype and Query Development Status
  • weekly meetings on Thursdays at 9am ET on zoom – contact Sharon Beltaine for zoom link and invitation

Reporting SIG Documentation Subgroup

  • No changes
  • Additional Context

External Statistics Working Group

  • no updates currently
  • new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
  • New organizational structure for External Statistics reports
    • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
    • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
    • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
    • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

Product Council

For all recent work on FOLIO Reporting SQL development:

Topics for Future MeetingsAll
  • Follow-up on MARC status, Quickmarc/Data Import conflicts
  • How to strengthen connections to SIGs and their developers to be kept in the loop about changes to the data model
  • Show and tell
    • how are institutions using the LDP
    • examples of using the local schema
    • Cornell's report ticketing system
    • Rollout plans from institutions
    • Ask someone on the sysadmin side to talk about LDP administration (Jason Root?)
    • What is done in JIRA? (JIRA clean up)
  • Training topics
    • adding test data in FOLIO snapshot
    • How to do ad hoc querying with the derived tables
    • How to use the LDP app
    • using KNIME to build reports (LDP edition)
    • use of local schema for custom tables
    • more on MARC (ask Jennifer)
    • using different applications (other than DBeaver)
    • Insomnia for API queries?
    • SQL
    • LDP to metadb
    • Metadb implications
      • if schema changes, will that be relatively seamless? old fields still in history, but new fields in current?
      • if fields are deprecated and removed, what happens if they come back again?
      • how will deleted records show up?
    • GitHub
    • Panorama
    • LDLite (again?)
  • Discussion:
    • how to rank clusters with institutional rankings going away?
    • consortia SIG is talking about a central office app of some kind; that app might need to deal with consortia-wide reporting
    • how to train:
      • wiki pages with examples? form a small project team? (e.g., how to pull from marc, how to use local schema)
      • each dev team create training about the data structures in that area?
    • Query style (ask Nassib)
    • Gathering institutional query repositories; someone can propose a new page or a new addition to an existing page on FOLIO Analytics?
  • Upcoming:
    • SQL advice/query optimization (Axel)
    • query demo - MARC (Tod)
    • Intro to SQL training (December 6 and 13 at least)
    • revisit discussion on openness/transparency in communication and open source software; decide as a group what we would like to advocate for

Review and update Topics for Future Reporting SIG Meetings 

  • A test Action Item (Ingolf)