2021-05-17 Reporting SIG Meeting notes

Announcements /
Reminders

WOLFcon

Registration for WOLFcon (June 1 to 3) is open - and free!

• More details on WOLFcon
• WOLFcon schedule

Test Data

FOLIO Reporting developers and Reporting SIG members are encouraged to use this new page to share test data cases they have entered into FOLIO reference environments to keep us all aware of what data to expect when we test our queries.

• Test Data Updates to FOLIO Snapshot Environments

Cluster Ranking

New Report Clusters are added on a regular basis, so it is important to make sure your institution is reviewing these clusters and ranking them to establish report development priorities. If you rank reports for your institution, please follow the instructions below. If someone else ranks, please pass this information along to that person so your institution's vote can be included.

• Action =>> Please review Reporting SIG-All Report Clusters (64 issues) in JIRA and RANK each report cluster for your institution (R1-R5)
• For reporting, institutions only need to rank the UXPROD Report Cluster JIRA issues. All reporting requirements, which are captured in REP-XXX issues, roll up to the UXPROD Report Clusters. Report clusters cover one or more report (REP-XXX issue) requirements.

Reporting SIG Documentation Subgroup

• • Had deadline to finish first draft by May 14, and then sometime during this week we will be sharing the documentation with the SIG for review
  • Context
    • Over the next few months, Reporting SIG Documentation Subgroup will be helping to build end-user documentation for https://docs.folio.org/docs/ (much will be linking to existing documentation over on GitHub)
    • see Reporting SIG Documentation Subgroup and Guide to Reporting Documentation pages in this wiki

RM Working Group

• working on queries for the 1.2 FOLIO Analytics release
  • voucher_lines_with_po_line and voucher_summary
  • reorganizing REP issues from the RM-Costs Cluster into more specific report clusters
  • subscriptions cost and subscriptions count
  • more with invoices and orders
• for latest updates, see RM Prototype and Query Development Status

External Statistics Working Group

• no updates currently
• ACRL query set included in FOLIO-Analytics 1.1 release
• new organizational/tracking scheme for JIRA, with pointers to queries in folio-analytics repository
• New organizational structure for External Statistics reports
  • external statistics reports (e.g., ACRL) typically require running queries from different functional reporting areas
  • these reports will be captured in JIRA under one UXPROD-XXXX report cluster issue, then the descriptions will point to each of the queries required to run them on the folio-analytics repository
  • institutions will need to rank each of these 8 new UXPROD-XXXX report cluster issues
  • each reporting development team will take responsibility for the queries in their area for the external statistics clusters

For all recent work on FOLIO Reporting SQL development:

• https://github.com/folio-org/folio-analytics/commits/main

• Cornell (Sharon)
  • presented slides
  • will be using a virtualized environment with fixed IP to give LDP access (via DBeaver) to report builders
  • 120 reporting users, 40 of whom will build queries and will need to develop SQL skill set
  • excited about LDP query builder app
  • in early June, Voyager Report-a-thon (asking people to run reports in the old system as a form of backup)
  • reporting training starting in June
  • mid-June, taking a snapshot of Voyager (Postgres copy of old system); will continue to build queries against that data; will also be migrating much of the data to FOLIO, but not migrating everything; initially not going to be able to compare apples to apples the Voyager historical data and FOLIO data set, so some reports will need to be run separately on historical database and on FOLIO
  • CUL has had same system for last 20 years, so it's a big change
  • hard to explain in-app vs. LDP reports (previously didn't have any reporting in the ILS)
  • thankful to be able to use LDP, incorporating data sets from outside the ILS
  • would be interested in more details about the virtual desktop being used for DBeaver
• Duke (Angela)

• Jenn will present an abbreviated version of the presentation she made to MM SIG last week (slides)
  • public → two tables (srs_marc and srs_records) with SRS (here, MARC) in JSON format, with and without administrative data
  • folio_source_record → __marc is the "interpreted" or "deconstructed" MARC
    • breaks apart JSON blob into its different components 
    • there are tags that repeat, so there is an ordinality column to keep track of order
  • Data Import FAQ including info on administrative metadata
  • MM reports subgroup noticed inconsistencies between Data Import and QuickMARC
    • 2021-05-12 Data Import Subgroup meeting 
    • problems with 999 matching
    • don't know what's happening with generation (the versioning of the MARC SRS record); would expect that it starts at 0 and increments, but doesn't always work that way; QuickMARC and Data Import treat generation differently, including the id and matchedId; in QuickMARC, matchedId is masking the updated UUID, but in Data Import, everything is wiped out and given new IDs; still trying to figure all of this out
    • has impact on both reporting and tracking changes
    • hoping to work this out before people go live
    • Q: should we lean toward one app over the other for edits for now?
      • not really
      • problem with QuickMARC - if you edit any field in QuickMARC, the 006 field will update to be the leader position 6, so there's data loss there; hoping to get that fixed
  • updating FOLIO LDP takes all night; these are really large tables; also takes a long time to run the queries
    • Nassib is not aware of performance issues for queries, so send along anything that takes a long time
  • see slides for example queries, other helpful links

• SRS MARC database structure: might it be worth separating control fields from data fields so fixed fields can be indexed differently from free text fields, maybe would reduce self joins on the table
  • could include in proposal, coordinate with Jennifer
• Rollout plans from institutions
• Follow-up on Quickmarc/Data Import conflicts

Review and update Topics for Future Reporting SIG Meetings