2020-07-06 Reporting SIG Meeting notes

Andi Bihler

Attendance & Notes

• Today's attendance-taker: Linda Miller
• Today's note-takers:  Team Leads for project updates

Project updates

The Reporting SIG is using small working groups to address priorities and complete our work.  Each week, we will provide updates to the Reporting SIG from these various reporting-related groups and efforts. Please include updates on specific JIRA issues for prototype or query development workflow. Do we need to create any additional JIRA tickets?

Community & Coordination: 

• working on reporting survey

FOLIO Reporting development:

1. Reporting reference environment: folio_release is now deprecated because the FOLIO source does not appear to be maintained at present.  Please switch to using folio_snapshot instead when convenient (not urgent).
2. Reviewing approaches to query factoring
   
   1. Pre-computing joins
      SELECT ... FROM (r1 JOIN r2 JOIN r3 JOIN r4 JOIN r5);
-> CREATE TABLE r123 AS SELECT ... FROM (r1 JOIN r2 JOIN r3);
   SELECT ... FROM (r123 JOIN r4 JOIN r5);
   2. Pre-computing "with" clauses (aka CTEs or common table expressions)
      WITH r1 AS SELECT ...;  ->  CREATE TABLE r1 AS SELECT ...;
   3. Combining "with" clauses that use the same table
   4. Preparing intermediate/derived tables
      1. Adding the local schema to the search path (optional) (no longer recommended)
         SET search_path TO local, public;
      2. Creating indexes
         CREATE INDEX ON table_name (column_name);
      3. Vacuuming and analyzing
         VACUUM table_name;
ANALYZE table_name;
   5. Beginning to think about defining commonly useful derived tables (after Forum)
      1. Commonly used JSON fields not currently available in relational attributes
      2. Commonly used denormalizing joins
      3. Other repetition that can be factored
   6. Automatic regeneration of tables

Reporting Data Privacy Working Group:

• active borrowers count increment
• Ingolf Kuss met with Sven, the Data Privacy Officer for ZBW. ZBW is a library in the GBV consortium. ZBW is an early implementer. According to Ingolf, if Sven approves of the measures implemented to protect data privacy in the LDP, then they are likely to be acceptable to all ZBW and GBV libraries (assuming that is all German libraries that are affiliated with Folio at this point).  Detailed meeting notes are at 2020-06-30 Reporting Data Privacy Working Group Meeting Notes. A few key issues are:
  1. Create a list that contains personal data fields, regardless of storage and module. Maintenance of the list by developers of the respective modules in data base conception, i.e. think about data protection right from the start (privacy by design).
  2. For anonymized data fields in the LDP, the values should be recognizable as anonymized (use of '0' or 'null' may be confusing).
  3. Long-term solution for European libraries: As U.S. libraries use the LDP data for quality control and so need user data included (fees/fines, etc.), whereas anonymizing this information for European libraries makes annual reporting difficult, Sven proposes a database (QM) system for personal data where all the data must be deleted after a fixed time period; this system is completely separate from the LDP.

RA/UM Working Group

• no updates
• looking for query developers

MM Working Group

• preparing HathiTrust query for FOLIO Forum

ERM Working Group

• for latest updates, see ERM Prototype and Query Development Status
• Bjorn Muschall generating test data and Axel preparing data for reference LDP