2018-03-26 Reporting SIG Notes

Previous Notetaker: Doreen Herold

Today's Notetaker: Scott Perry

UXPROD-331 and UXPROD-332 JIRA Tx Review - Reporting SIG to review tickets entered in JIRA to support development of the external reporting environment (Data Lake/Store/Warehouse)

https://folio-org.atlassian.net/browse/UXPROD-331

https://folio-org.atlassian.net/browse/UXPROD-332

-need to decide what is to be included in V1

-consideration to develop reporting solution as a module and not a core piece of functionality for Okapi

-addition of an outbound filter which gets transaction and handles the writing to the data lake

Important for SIG to look at general features in v1. Are these essential?

HK: These represent tasks that will become clearer as developers work.

TO: Need to have features in V1 or 2 with enough time to be tested.

HK: Need all of the current list in V1. Individual SIGs will determine which data elements are essential.

External reporting environment has been relational, need to move towards data lake model.

Is Chicago confident that data will be correct after running test system through several iterations? PO link to correct item, inventory is real, our policies are being enforced. Would confirm all of the data from time to time. Referential consistency needs to be proved. Trust but verify. Basis for test cases that could be run periodically to prove referential integrity. This group, data migration and metadata management are all good areas for those with expertise in identifying data integrity tests.

Need ways to confirm migration as well as operational integrity. Tests can be documented so they can be run routinely.

Can we include referential data integrity in V1? Create another JIRA?

SB: This is a start, but the SIG will need to work with developers to flesh out the JIRAs. Need to discuss priority with PC. SIG agreed this will be a priority for V1. Generally speaking we all support the data lake, but need to refine the details.

Need to confirm data integrity as data migration occurs. Does data integrity belong elsewhere? It's part of everything. PC question?

 Anne will discuss the data elements definition work she is doing in the Data Migration Subgroup

Data Migration folder:

https://drive.google.com/drive/folders/1aUMIqc4SwRzOGR4yPzrRSRElwM2CQ3uY

Documentation of record structure (json) https://dev.folio.org/doc/api/

Working on User schema JSON so far. Which elements in current systems that aren't currently in FOLIO need to be migrated.

Have also done some of the work for items and inventory. This process highlights possible additional needs. Data elements are still fluid.

Is there a list of data elements to add? This is the first step in that process to the relevant SIG to review data elements. Can start to compare our SIG data elements in the spreadsheet.

Sharon will discuss strategies for translating reporting requirements into reporting deliverables