/
2023-05-08 Reporting SIG Meeting notes

2023-05-08 Reporting SIG Meeting notes

May 08, 2023

Date

May 8, 2023

Attendees

 Present?

Name

Organization

Present?

Name

Organization

 Present?

Name

Organization

Present?

Name

Organization

x

Arthur Aguilera

University of Colorado, Boulder

x

Eliana Lima

Fenway Library Organization

x

Sharon Beltaine

Cornell University

 

Eric Luhrs

Lehigh University

 

Erin Block

University of Colorado, Boulder

x

Linda Miller

Cornell University

 

Nancy Bolduc

Cornell University

 

Nassib Nassar

Index Data

x

Shannon Burke

Texas A&M

x

Elena O'Malley

Emerson

 

Suzette Caneda

Stanford University

x

Tod Olson

University of Chicago

 

Lloyd Chittenden

Marmot

x

Jean Pajerek

Cornell University

 

Tim Dannay

Mount Holyoke College

x

Kimberly Pamplin

Texas A&M University 

x

Axel Doerrer

University Mainz

x

Scott Perry

University of Chicago

 

Shelley Doljack

Stanford University

 

Natalya Pikulik

Cornell University

 

Stefan Dombek

Leipzig University

 

Bob Scheier

Holy Cross

x

Jennifer Eustis

U. Massachusetts Amherst / Five College

x

Vandana Shah

Cornell University

 

Lynne Fors

Wellesley College

x

Kimberly Smith

Middle Tennessee State University

 

Lisa Furubotten

Texas A&M

 

Linnea Shieh

Stanford University

 

Alissa Hafele

Stanford University

 

Clare Spitzer

Stanford University

x

Kara Hart

Wellesley College

 

Amelia Sutton

U. Massachusetts

 

Corrie Hutchinson

Index Data

 

Simona Tabacaru

Texas A&M

 

Jamie Jesanis

Wellesley College

 

Huey-Ning Tan

Stanford University

 

Jeanette Kalchik

Stanford University

 

Vitus Tang

Stanford University

 

Kevin Kishimoto

Stanford University

 

Irina Trapido

Stanford University

 

Ingolf Kuss

hbz

 

Catherine Tuohy

Emmanuel College

 

Alexander Lao

Stanford University

 

Kevin Walker

The University of Alabama

 

Joanne Leary

Cornell University

x

Angela Zoss

Duke University

Discussion Items

Item

Who

Notes

Item

Who

Notes

Attendance & Notes

Sharon

Attendance & Notes

  • Today's attendance-taker: Linda (or substitute)

  • Today's note-takers:  Team Leads for project updates

Announcements /
Reminders

Sharon

Upcoming meeting topics

  • May 18: SIG Onboarding Part 2 (Development Work) - tentative

 

Combining MM and RA/UM subgroups

 

Reporting Implementation Table

 

Any new members?

  • Welcome/introductions

 

How to find our latest recordings

 

(Always) Recruiting New Query Developers

  • The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

 

Demonstrations of ldpmarc

Jennifer, Angela

  • https://librarydataplatform.org/ (LDP1, Metadb, LDLite)

  • https://librarydataplatform.org/resources/

  • https://github.com/5-C-Folio/LDlite-Queries

  • At the Five Colleges (5C), they have LDLite on a server and a systems librarian runs it

  • Have customized to get ERM

  • Everything except for inventory is downloaded once per night; inventory and SRS are once a week over the weekend

  • LDLite is not real-time, just a snapshot

  • LDLite transformations are similar to Metadb, so the data structure looks like Metadb - two tables, one with the original JSON data, one with the data transformed into a flat tables

  • In LDLite, the data transformation drills into the arrays and objects, so many of the derived tables we use in Metadb are not useful

  • marctab table is the result of the ldpmarc transformation, converting MARC to a tabular data format

  • (marc_records_lb is parsed JSON of SRS MARC Bib)

  • marctab doesn't have state (old or active), so need to go to the original MARC JSON for that (records table)

  • What kind of queries does 5C run?

    • looking for orphan instances (instances without holdings or items attached); data import creates a lot of orphans, something is wrong; these orphans still show up in discovery, which is confusing for users

  • queries look like LDP1 and Metadb queries - a lot of left joins

  • how are you connecting using DBeaver?

    • the python script is run by an administrator from a server, and then it pushes data into a shared PostgreSQL database, and most people connect to that directly through DBeaver

  • Is LDLite similar to LDP that you run the whole database refresh once a night?

    • Our systems librarian set it up to run an update overnight for everything except inventory and marc SRS; those are big and take a long time, and if they hang it will grind FOLIO to a halt

  • What scale does this seem to be suitable for?

    • Not great at large scale

    • People get confused that it's not current data, and people who don't know SQL still can't use it

    • with billing, especially hard because can't even really get real time data

    • 5C actually doesn't pull everything

    • Even with LDLite separate database, will have some queries that hang up the reporting database, will need to kill it

    • Are the database tables indexed like LDP1 and Metadb?

  • Inventory and SRS often have the same thing; might be better to use data from inventory when possible

 

 

Recurring Items (Updated weekly, but not always discussed in meeting)

Item

Who

Notes

Item

Who

Notes

Review of In-Progress Projects (Recurring)

 

Review the release notes for FOLIO Analytics, LDP1, LDLite, LDP Reporting App, ldpmarc, Metadb Projects (Recurring)

 

Updates and Query Demonstrations from Various Reporting Related Groups and Efforts Projects (Recurring)

Community & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

 

RA/UM Working Group

 

MM Working Group

 

ERM Working Group

  • Current topics

    • Reporting for FOLIO Apps

      • Open Access

      • eUsage

      • eHoldings

  • Meetings are bi-weekly on tuesdays 11am ET alternating with ACQ Working Group

    • Next meeting will be at 9th, May

    • Contact @Stefan Dombek if you would like to get a calendar invitation

 

ACQ Working Group

 

Reporting SIG Documentation Subgroup

  • Morning Glory documentation is live on https://docs.folio.org/docs/

  • Nolana documentation is in review

  • Orchid documentation will be in progress soon, and plans are underway to include beta-level documentation for Metadb

  • Additional Context

    • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

 

External Statistics Working Group

  • no updates currently

  • new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository

  • New organizational structure for External Statistics reports

    • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas

    • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository

    • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues

    • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

 

D-A-CH Working Group (D-Reporting)

  • Discussion on draft statement: "Hosting and Support"

    • The members of D-Reporting need more time to review the document and provide feedback

  • Ongoing topics

    • Onboarding training

    • DBS statistics

      • JIRA-Issues, Rpt-Clusters

      • Gap analysis to statistic codes, user counts etc.

      • Identifying functions that we need for statistics in Germany but are not yet implemented in FOLIO

  • Meetings are currently held by appointment

    • Last meeting: 8th, May

    • Next meeting will be at 15th, May (1 pm, CEST)

    • Contact @Stefan Dombek if you would like to get a calendar invitation

 

Product Council

 

For all recent work on FOLIO Reporting SQL development: