2022-01-10 Reporting SIG Meeting notes

x

Announcements /
Reminders

Welcome back, everyone! Happy New Year!

Recap of recent SIG activity

• Intro to SQL content in December
• Discussion about how reporting community can get advanced warning about changes to data model in FOLIO apps
  • Can subteams try to add examples sometime this week?
• Prototype for a new documentation model for derived tables from Stefan (revisiting today!)
• Change of membership on documentation subgroup: Eric Pennington will take over as SIG representative, attending monthly Documentation Working Group meetings. Thanks, Eric!

Upcoming SIG activity

• beginning of the year visioning conversation (next week)
• continue to explore LDP data model and query documentation 
• more Intro to SQL and LDP semantics training
• advocation for better communication with FOLIO devs
• talk about training/onboarding workflow as a SIG
• regular full-group query developer meetings to learn from each other (bi-weekly)
• others?

Jan 17 is US Holiday for some?

• should we meet?
• No, we will cancel

Recruiting New Query Developers

• The Reporting SIG is always on the look-out for new query developers. Please let us know if you are interested in doing query development or if there are others at your institution who might be a good fit.

ERM prototype for derived table documentation

First prototype created for documentation Link to prototype

Discussion:

• general: Do we need an extra documentation or is the previous route via Confluence, Readme, comments in the query sufficient? (Effort / benefit)
  
• The prototype contains several sections. Do we need all of it?
• Would we like a graph with mermaid?
• Which data should be recorded in the table? The previous columns are suggestions.
• How would we like to organize the creation?
• discussion:
  • always helpful to have diagrams for data models; helps you understand the data flows, how to link things together. hard to carve out the time to work through the documentation. would be great to have teams work on this and think we should do it. maybe we could streamline - what is absolutely essential?
  • yes, think this is great, it's visual and I think what Stefan did is good in a formal way, don't need to add prose, just follow the format. good to have attributes and where they come from combined with the visual. don't know if it's necessary to have the source code, could just link to that. really like it otherwise.
  • you can create a pull request on the HTML file in GitHub, so you can have discussion about the documentation and mark the lines, see changes in the source code
  • like that it has the version of FOLIO it works on; should it also have the version of LDP/Metadb it works on?
  • like the table showing which things have multiple values/aggregations, nice to see that clearly instead of having to think about it
  • some visuals might be kind of overwhelming
  • maybe collapse some sections in the HTML so it's not overwhelming; but would adding things like that take extra time
  • HTML vs. Markdown - would maybe be nice to use Markdown so it renders in GitHub
  • can you PDF the diagram and link from the Markdown
  • for FOLIO version on docs: just latest version it works with? or retain all versions it works with?
  • for LDP/Metadb version: do we track? how?
  • just derived tables? or also report queries?

RM Working Group

• first meeting of year will be next week; see Sharon Beltaine if questions, would like to join, etc.
• working on LDP to Metadb table mapping spreadsheet to prepare for transition to Metadb
• subscription costs and subscription counts queries PR posted to folio-analytics
• casting numbers: need to cast numbers related to finances as decimals, not money; money rounds up, which may throw totals off
• looking at using https://mermaid-js.github.io/mermaid/#/ to document data models in finance areas of FOLIO 
• exploring ways to make it easier to find options for reporting parameters in your LDP (e.g., a list of locations, item statuses, etc.)
• several queries for RM completed, but still need documentation, testing, and review
• for latest updates, see RM Prototype and Query Development Status

Reporting SIG Documentation Subgroup

• Honeysuckle documentation is live on https://docs.folio.org/docs/
• Iris documentation is in progress, due December 15
• Additional Context
  • The Reporting SIG has representation on the Documentation Working Group, which is building end-user documentation for https://docs.folio.org/docs/ (mostly linking to existing documentation over on GitHub)

External Statistics Working Group

• no updates currently
• new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
• New organizational structure for External Statistics reports
  • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
  • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
  • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
  • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

Product Council

For all recent work on FOLIO Reporting SQL development:

• https://github.com/folio-org/folio-analytics/commits/main

• Follow-up on MARC status, Quickmarc/Data Import conflicts
• How to strengthen connections to SIGs and their developers to be kept in the loop about changes to the data model
• Show and tell
  • how are institutions using the LDP
  • examples of using the local schema
  • Cornell's report ticketing system
  • Rollout plans from institutions
  • Ask someone on the sysadmin side to talk about LDP administration (Jason Root?)
  • What is done in JIRA? (JIRA clean up)
• Training topics
  • adding test data in FOLIO snapshot
  • How to do ad hoc querying with the derived tables
  • How to use the LDP app
  • using KNIME to build reports (LDP edition)
  • use of local schema for custom tables
  • more on MARC (ask Jennifer)
  • using different applications (other than DBeaver)
  • Insomnia for API queries?
  • SQL
  • LDP to metadb
  • Metadb implications
    • if schema changes, will that be relatively seamless? old fields still in history, but new fields in current?
    • if fields are deprecated and removed, what happens if they come back again?
    • how will deleted records show up?
  • GitHub
  • Panorama
  • LDLite (again?)
• Discussion:
  • how to rank clusters with institutional rankings going away?
  • consortia SIG is talking about a central office app of some kind; that app might need to deal with consortia-wide reporting
  • how to train:
    • wiki pages with examples? form a small project team? (e.g., how to pull from marc, how to use local schema)
    • each dev team create training about the data structures in that area?
  • Query style (ask Nassib)
  • Gathering institutional query repositories; someone can propose a new page or a new addition to an existing page on FOLIO Analytics?
  • Follow up on how devs can work with SMEs better to decide on and advertise data model changes
• Upcoming:
  • SQL advice/query optimization (Axel)
  • query demo - MARC (Tod)
  • Intro to SQL training (as time allows)
  • revisit discussion on openness/transparency in communication and open source software; decide as a group what we would like to advocate for
  • ERM prototype for derived table documentation (January 10)

Review and update Topics for Future Reporting SIG Meetings