2022-01-24 Reporting SIG Meeting notes









xArthur AguileraUniversity of Colorado, BoulderxLinda MillerCornell University
xSharon BeltaineCornell University
Nassib NassarIndex Data
xErin BlockUniversity of Colorado, BoulderxElena O'MalleyEmerson

Nancy BolducCornell University
Tod OlsonUniversity of Chicago

Lloyd ChittendenMarmotxJean PajerekCornell University
xAxel DoerrerUniversity MainzxMichael PatrickThe University of Alabama

Shelley DoljackStanford UniversityxEric PenningtonTexas A&M


Stefan DombekLeipzig UniversityxScott PerryUniversity of Chicago
xJennifer EustisU. Massachusetts Amherst / Five CollegexNatalya PikulikCornell University
xAlissa HafeleStanford UniversityxVandana ShahCornell University
xIngolf KusshbzxAmelia SuttonU. Massachusetts

Kim LaineCornell University
Simona TabacaruTexas A&M

Joanne LearyCornell UniversityxKevin WalkerThe University of Alabama
xEliana LimaFenway Library OrganizationxAngela ZossDuke University
xEric LuhrsLehigh University

Discussion Items




Attendance & NotesAngela

Attendance & Notes

  • Today's attendance-taker: Linda Miller
  • Today's note-takers:  Team Leads for project updates

Announcements /


Possible raining topic: Git/Github?

  • should we cover that in a SIG meeting? how many interested?
    • about half

Recruiting New Query Developers

  • The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

2022 Visioning

Reviewing Goals for 2021:

  • Prove the design of using external data sources (outside FOLIO data) in the Library Reporting Database
    • ingesting non-FOLIO data into LDP - several institutions have done this, so yay!
  • User acceptance testing: are we meeting the requirements that we hoped we would?
    • have relied on implementers to test queries in some cases
    • LDP app itself had traditional user acceptance testing
  • Develop support for using MARC data in FOLIO reporting
    • not available in LDP, and non-LDP implementers really don't have a solution
    • switch to Metadb, we will be able to tackle this
    • LDlite also offers this access
    • Did advocate for these features in 2021 and helped build use cases and test
    • we're having to wait both on LDP/Metadb development and also on SRS development, which originally only had bibs
  • Develop a FOLIO Reporting Vision and Strategy
    • definitely have talked around this for a while, but I don't think we really sat down to articulate something
  • Develop support structure for those using queries
    • did articulate a support workflow
  • Understand our needs for normalizing, denormalizing, summarizing, aggregating, anonymizing data
    • much of this happens as part of software planning
    • we also use derived tables to accomplish these tasks
  • Expand outreach by sharing experiences of report development groups more broadly and getting feedback from other groups in the FOLIO community
    • maybe a slower year for outreach last year
  • Maintain, support, and distribute robust repository of shared queries
    • this is still happening, feel good about the shared queries
  • Write shared documentation on the queries, connecting reporting tools to Library Reporting Database
    • report queries are well documented (but not derived tables)
  • Write shared documentation on the Library Reporting Database implementation, maintenance and schema (e.g., data dictionary)
    • review of documentation needs revealed that existing documentation works well, so SIG doesn't need to do this part
  • Reach out to FOLIO implementers and respond to their reporting problems and needs
    • different working groups have been reaching out to implementers/working with them
  • Extend collaborative model of report development groups to a larger scale, focusing on those who will actually be using the queries
    • this is happening fairly organically as institutions go live; they are influencing the development groups

What else did we accomplish?

  • FOLIO LDP app
  • early development on Metadb queries
  • documentation added to docs.folio.org
  • successful transition between conveners
  • made connections to other reporting efforts
  • Institutions went live (Cornell, TAMU, Duke with course reserves), with support from dev groups
  • Implementers shared back about their experiences
  • New members joined, members taking on new roles
  • SIG-wide training
  • continued advocacy on part of the SIG to governance groups

Priorities for 2022:

  1. (H) Support the transition from LDP to Metadb (e.g., update derived table and report queries, update documentation, outreach, new training)
  2. (H?) continued advocacy on part of the SIG to governance groups
  3. (M) ramp up on outreach (esp. FOLIO Forums)
  4. (L?) explore connections to ReShare (e.g., ask for presentation)
  5. (M) deepening ties to other SIGs
    1. maybe each group can go to SIG and give presentation on Metadb
  6. (H?) Develop a FOLIO Reporting Vision and Strategy
    • secondary goal: come up with language that can be used to explain to our institutions the importance of the work and of devoting resources
  7. (M? H?) Review JIRA issues, clean up, revisit strategy for JIRA
  8. (H) Developing training/onboarding for new SIG members/report users (esp. FOLIO-specific data model and transformation stuff)
  9. (L) Share experiences on hosting options for reporting databases
    • do we have a doc somewhere? we have the implementers grid, which is a start
    • maybe a discussion topic for a SIG meeting where people who have different hosting providers share their experiences
  10. (H?) Improve communication between SIG and developers of apps so we hear about data model changes in advance
  11. Syncing our repository updates up with the FOLIO release schedule
  12. (M) Build ER Diagram-based documentation (mermaid) for derived tables

Updates and Query Demonstrations from Various Reporting Related Groups and EffortsCommunity & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

RA/UM Working Group

MM Working Group

  • Meetings are 1st Tuesday of the month, 12-1pm ET via zoom using the usual FOLIO password. Our lab sessions are open to everyone. Please bring your questions, examples, and comments about reporting and metadata.
  • Our goals this year:
    • Work on converting our LDP derived tables and queries to metadb
    • Work on learning GitHub, git, and more sql

ERM Working Group

  • Discussed ERM Goals 2022
    • Complete documentation on existing derived tables
    • Migrate ERM LDP queries for Metadb use
    • How to get more ERM SIG more involved for...
      • (real life) reporting requirements
      • checking results of report on real data
    • How can development for the integration of eHoldings data be funded
  • Housekeeing: Github issues left overs, how long to keep meeting recordings etc.
  • ERM Prototype and Query Development Status
  • Meetings are biweekly and will be moved to Tuesdays 11am-noon ET alternating with RM Working Group
    • Still need to change the time slot with an OLF Zoom Meeting Room
    • Contact Axel Dörrer if you would like to join these meetings

RM Working Group

  • Next meeting is Tuesday, 1/25/22
  • Meetings are biweekly  on Tuesdays11am-noon ET; contact Sharon Beltaine if you would like to join us
  • working on LDP to Metadb table mapping spreadsheet to prepare for transition to Metadb
  • looking at using https://mermaid-js.github.io/mermaid/#/ to document data models in finance areas of FOLIO 
  • exploring ways to make it easier to find options for reporting parameters in your LDP (e.g., a list of locations, item statuses, etc.)
  • several queries for RM completed, but still need documentation, testing, and review
  • for latest updates, see RM Prototype and Query Development Status

Reporting SIG Documentation Subgroup

  • Honeysuckle documentation is live on https://docs.folio.org/docs/
  • Iris documentation is in progress, due December 15
  • Additional Context
    • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

External Statistics Working Group

  • no updates currently
  • new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
  • New organizational structure for External Statistics reports
    • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
    • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
    • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
    • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

Product Council

For all recent work on FOLIO Reporting SQL development:

Topics for Future MeetingsAll
  • Follow-up on MARC status, Quickmarc/Data Import conflicts
  • How to strengthen connections to SIGs and their developers to be kept in the loop about changes to the data model
  • Show and tell
    • how are institutions using the LDP
    • examples of using the local schema
    • Cornell's report ticketing system
    • Rollout plans from institutions
    • Ask someone on the sysadmin side to talk about LDP administration (Jason Root?)
    • What is done in JIRA? (JIRA clean up)
  • Training topics
    • adding test data in FOLIO snapshot
    • How to do ad hoc querying with the derived tables
    • How to use the LDP app
    • using KNIME to build reports (LDP edition)
    • use of local schema for custom tables
    • more on MARC (ask Jennifer)
    • using different applications (other than DBeaver)
    • Insomnia for API queries?
    • SQL
    • LDP to metadb
    • Metadb implications
      • if schema changes, will that be relatively seamless? old fields still in history, but new fields in current?
      • if fields are deprecated and removed, what happens if they come back again?
      • how will deleted records show up?
    • GitHub
    • Panorama
    • LDLite (again?)
  • Discussion:
    • how to rank clusters with institutional rankings going away?
    • consortia SIG is talking about a central office app of some kind; that app might need to deal with consortia-wide reporting
    • how to train:
      • wiki pages with examples? form a small project team? (e.g., how to pull from marc, how to use local schema)
      • each dev team create training about the data structures in that area?
    • Query style (ask Nassib)
    • Gathering institutional query repositories; someone can propose a new page or a new addition to an existing page on FOLIO Analytics?
    • Follow up on how devs can work with SMEs better to decide on and advertise data model changes
  • Upcoming:
    • SQL advice/query optimization (Axel)
    • query demo - MARC (Tod)
    • Intro to SQL training (as time allows)
    • revisit discussion on openness/transparency in communication and open source software; decide as a group what we would like to advocate for

Review and update Topics for Future Reporting SIG Meetings 

  • A test Action Item (Ingolf)