2018-10-22 Reporting SIG notes

Duke University

Nassib will discuss the https://glintcore.net project, which provides a foundation for the library data platform architecture for the FOLIO reference data warehouse. Glint is open source software for communicating, describing, and integrating data.

Notes

Glint and the LDP (slides by Nassib) (LDP = Library Data Platform)

Glint : An open source software for sharing, curating and integrating data.

Features:

• lightweight protocol
• interoperability layer
• new approach to managing scientfic data

Data curation = describing data. Data curation usually takes place at a late stage, after data collection, data pipelines and sharing with a research team. Glint takes care ot the data curation at an earlier stage. This helps to integrate and re-use the data.

Glint is kept simple (lightweight) to be able to run in many different contexts.

Glint has 2 interfaces:

• web-based interface
• command line interface

Glint could be configured to store the data in a repository. The data is posted to glint (e.g. by the command line interface) and will be assigned an URL.The data is user-modifiable. There is a simple language to modifiy the data.

Jason Skomorowski has written an UI-interface for integrating data in FOLIO (ui-datasets, a FOLIO client for Glint).

LDP - Library Data Platform that is build for Reporting. It consists of (classical) Batch-ETL and a database (on the left hand side, for a classicalReporting Tool), but also streaming ETL, distributed databases and other databases (on the right hand side, for other data analysis tools). One might use Glint as a data platform.

Questions that come up with LDP:

• How can requests be made available to people that don't have access to connect to the database ? → One can address that with Glint.
• How will data outside of the database be accessed ?
• Support for data analysis tools such as R ?
• What about privacy and permissions ?

How Glint can help:

Glint allows for a lot of flexibility how data are stored:

• file system
• network database

Glint can be used to offer an interface for accessing reports.

With Glint, one can access data that are within as well as outside of a database.

Integration with R and Python: Can use Python calls for Glint, a Glint library has been developed for Python.

LDP can be made accessible to FOLIO modules in form of Glient data sets, when direct database access is not feasible. I.e. if that would put to much load on the database.

What Glint doesn't do:

• Glint doesn't give you a full query language. It focuses on describing and sharing data. Assumes that querying will be done by database views.
• Less complex permissions model than in FOLIO.

Discussion

Q: Does Glint replace the database ? Will Glint be the primary source when writing reports ?

A: Glint would not replace the direct database connection.

Glint would share reports, share database views. Glint gives access to the data (to people who do not have direct access to the database).

Glint is for data scientists, not so much for data analysts (the latter ones would prefer to use SQL).

Glint can be used for data which do not fit in a database. It is also for sharing data to the public (can be views or generated reports).

A Reporting SIG Data Warehouse might be redundant for institutions which already have a data warehouse (e.g. needed for legal reasons or for national reporting). But most of us are not in the situation where we already have a data warehouse.

LDP is a hybrid solution for evolving needs for data analysis. The model (LDP and Glint) is superior in some ways but inferior in many ways. It would be a more modern system, but some things will have to be re-invented.

to be continued

In order to determine the order in which we will prototype our reports for the development of data models for the data warehouse, we need to prioritize them within each functional area (e.g., RM, RA, etc.). Please see the Prioritizing Reports page in the wiki for detailed instructions.

-This week, we will review our group's progress on prioritizing reports in the "import and export," "external statistics," and "resource mgmt" worksheets in the Reporting SIG Master Spreadsheet

-Monday November 12, 2018 is our target completion date

Holly walked us through the process of assigning ourselves to our reports in the FOLIO JIRA System. Instructions are provided on this wiki page, which will also contain other JIRA info:  Working on a JIRA Issue

-review of JIRA report assignments

-Monday November 12, 2018 is our target completion date

Topics for Future Meetings

Review and update Topics for Future Reporting SIG Meetings