2021-05-17 Reporting SIG Meeting notes

Sharon Beltaine

Cornell University

Sara Colglazier

Mount Holyoke College/Five Colleges

Elizabeth Berney

Duke University

Erin Nettifee

Duke University

Joyce Chapman

Duke University

Karen Newbery

Duke University

Jean Pajerek

Cornell University

Tod Olson

University of Chicago

Claudius Herkt-Januschek

SUB Hamburg

Scott Perry

University of Chicago

Doreen Herold

Lehigh University

Stefan Stadtherr

MPIL Heidelberg

Sarah Park

Duke University

Simona Tabacaru

Texas A&M

Clint Bellanger

Auburn University

Kevin Walker

The University of Alabama

Ingolf Kuss

hbz

Christie Thomas

University of Chicago

Joshua Lambert

Missouri State

Andi Bihler

Munich Technical University Library

Joanne Leary

Cornell University

Cheryl Malmborg

University of Chicago

Michael Patrick

The University of Alabama

Vandana Shah

Cornell University

Nassib Nassar

Index Data

Angela Zoss

Duke University

Veit Köppen

University Magdeburg

Lisa DeCarolis

Smith College/Five Colleges

Linda Miller

Cornell University

Elena O'Malley

Emerson

Matt Harrington

Duke University

Holly Mistlebauer

Cornell University

Cathy Tuohy

Emmanuel College (FLO)

Nancy Bolduc

Cornell University

Shirley Moentnish

Missouri State

Eric Pennington

Texas A&M

Stefan Dombek

Leipzig University

Axel Doerrer

University Mainz

Natalya Pikulik

Cornell University

Owen Stephens

K-Int

Eliana Lima

Fenway Library Organization

Marcia Borensztajn

EBSCO

Harry Kaplanian

EBSCO

Molly Driscoll

EBSCO

Amelia Sutton

U. Massachusetts

Heather Loehr

Hanover College

Mary Morgan

Grand Valley State University

Jennifer Eustis

U. Massachusetts Amherst / Five College

Aaron Neslin

Five Colleges

Laura Daniels

Cornell University

Jesse Lambertson

University of Chicago

Attendance & Notes

Angela

Attendance & Notes

• Today's attendance-taker: Linda Miller

• Today's note-takers:  Team Leads for project updates

Announcements /
Reminders

Angela

WOLFcon

Registration for WOLFcon (June 1 to 3) is open - and free!

• More details on WOLFcon

• WOLFcon schedule

Test Data

FOLIO Reporting developers and Reporting SIG members are encouraged to use this new page to share test data cases they have entered into FOLIO reference environments to keep us all aware of what data to expect when we test our queries.

• Test Data Updates to FOLIO Snapshot Environments

Cluster Ranking

New Report Clusters are added on a regular basis, so it is important to make sure your institution is reviewing these clusters and ranking them to establish report development priorities. If you rank reports for your institution, please follow the instructions below. If someone else ranks, please pass this information along to that person so your institution's vote can be included.

• Action =>> Please review Reporting SIG-All Report Clusters (64 issues) in JIRA and RANK each report cluster for your institution (R1-R5)

• For reporting, institutions only need to rank the UXPROD Report Cluster JIRA issues. All reporting requirements, which are captured in REP-XXX issues, roll up to the UXPROD Report Clusters. Report clusters cover one or more report (REP-XXX issue) requirements.

Updates and Query Demonstrations from Various Reporting Related Groups and Efforts

Community & Coordination, Reporting Subgroup Leads

Project updates

Reporting development is using small subgroups to address priorities and complete work on report queries.  Each week, these groups will share reports/queries with the Reporting SIG.  Reporting development team leads are encouraged to enter a summary of their work group activities below.

Reporting SIG Documentation Subgroup

RA/UM Working Group

• Have a new overdue items query to post, then taking a week off because of other work and need to catch up a bit

• Context

  • Meeting notes: https://docs.google.com/document/d/1UnzG64tl917LOH2FtWhCEPlSOsnJWKL-0eu88Ouo1DU/edit

  • Current status of RA/UM issues: https://github.com/folio-org/folio-analytics/issues?q=is%3Aissue+is%3Aopen+RA%2FUM

MM Working Group

• We didn't meet last week.

• For general report development, the working group uses this spreadsheet to track JIRA issues, priority, and reports needed. However, because many institutions are going live, we would like to focus on completing the following:

  • OCLC synchronization report

  • Duplicate orders reports

  • HathiTrust reports

  • More LDP Marc testing

ERM Working Group

• Some ERM tables missing due to missing a primary key

  • could be added although but without history as a future feature in Metadb

  • primary keys will be added if there is a defined use case for them

  • Reporting SIG members may have to weigh in

• Agreement package title list in review

• ERM Prototype and Query Development Status

RM Working Group

• working on queries for the 1.2 FOLIO Analytics release

  • voucher_lines_with_po_line and voucher_summary

  • reorganizing REP issues from the RM-Costs Cluster into more specific report clusters

  • subscriptions cost and subscriptions count

  • more with invoices and orders

• for latest updates, see RM Prototype and Query Development Status

External Statistics Working Group

• no updates currently

• ACRL query set included in FOLIO-Analytics 1.1 release

• new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository

• New organizational structure for External Statistics reports

  • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas

  • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository

  • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues

  • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

For all recent work on FOLIO Reporting SQL development:

• https://github.com/folio-org/folio-analytics/commits/main

Reporting Rollout Plans from different implementing Institutions

Sharon / Angela / others

• Cornell (Sharon)

  • presented slides

  • will be using a virtualized environment with fixed IP to give LDP access (via DBeaver) to report builders

  • 120 reporting users, 40 of whom will build queries and will need to develop SQL skill set

  • excited about LDP query builder app

  • in early June, Voyager Report-a-thon (asking people to run reports in the old system as a form of backup)

  • reporting training starting in June

  • mid-June, taking a snapshot of Voyager (Postgres copy of old system); will continue to build queries against that data; will also be migrating much of the data to FOLIO, but not migrating everything; initially not going to be able to compare apples to apples the Voyager historical data and FOLIO data set, so some reports will need to be run separately on historical database and on FOLIO

  • CUL has had same system for last 20 years, so it's a big change

  • hard to explain in-app vs. LDP reports (previously didn't have any reporting in the ILS)

  • thankful to be able to use LDP, incorporating data sets from outside the ILS

  • would be interested in more details about the virtual desktop being used for DBeaver

• Duke (Angela)

Report from the Metadata Reports Working Group for the Reporting SIG

Jenn

• Jenn will present an abbreviated version of the presentation she made to MM SIG last week (slides)

  • public → two tables (srs_marc and srs_records) with SRS (here, MARC) in JSON format, with and without administrative data

  • folio_source_record → __marc is the "interpreted" or "deconstructed" MARC

    • breaks apart JSON blob into its different components 

    • there are tags that repeat, so there is an ordinality column to keep track of order

  • Data Import FAQ including info on administrative metadata

  • MM reports subgroup noticed inconsistencies between Data Import and QuickMARC

    • 2021-05-12 Data Import Subgroup meeting 

    • problems with 999 matching

    • don't know what's happening with generation (the versioning of the MARC SRS record); would expect that it starts at 0 and increments, but doesn't always work that way; QuickMARC and Data Import treat generation differently, including the id and matchedId; in QuickMARC, matchedId is masking the updated UUID, but in Data Import, everything is wiped out and given new IDs; still trying to figure all of this out

    • has impact on both reporting and tracking changes

    • hoping to work this out before people go live

    • Q: should we lean toward one app over the other for edits for now?

      • not really

      • problem with QuickMARC - if you edit any field in QuickMARC, the 006 field will update to be the leader position 6, so there's data loss there; hoping to get that fixed

  • updating FOLIO LDP takes all night; these are really large tables; also takes a long time to run the queries

    • Nassib is not aware of performance issues for queries, so send along anything that takes a long time

  • see slides for example queries, other helpful links

Topics for Future Meetings

All

• SRS MARC database structure: might it be worth separating control fields from data fields so fixed fields can be indexed differently from free text fields, maybe would reduce self joins on the table

  • could include in proposal, coordinate with Jennifer

• Rollout plans from institutions

• Follow-up on Quickmarc/Data Import conflicts

Review and update Topics for Future Reporting SIG Meetings