| First presentation of LDP road map | Nassib | - Considerations:
- Many reports have been ranked by institutions for "go live (MVP)".
- Reporting SIG members have repeatedly communicated the need for access to all FOLIO data for reporting purposes.
- LDP critical dependencies on FOLIO core development which have been requested and flagged as critical beginning in late 2018 will likely not be addressed until mid-2020 or beyond, based on discussions with the FOLIO capacity planning and project management groups.
- FOLIO is requesting that "go live (MVP)" features be ready by January 2020, to be released in Summer 2020.
- Future developer resources appear to continue to be roughly around 1 FTE on average, consisting of several part-time developers.
- LDP road map is based on these considerations, and aims to release features according to FOLIO's projected time lines for addressing critical dependencies as well as available developer resources.
- LDP v11.0, for "go live (MVP)":
- LDP will support ad hoc, cross-domain queries for all, or a very large proportion, of FOLIO data extracted from storage modules. We would ask this working group to help us determine, in the near future, the definition of "all FOLIO data" required for inclusion in the LDP.
- LDP data will include MARC records extracted from FOLIO SRS and transformed for easier querying.
- The LDP developers and this working group will aim to implement all reports requested for "go live (MVP)" if possible.
- Historical data will be retained in the LDP but not transformed into a single schema.
- LDP data can be refreshed nightly from the FOLIO operational database.
- LDP will optionally anonymize personal data.
- LDP will include implementation guidelines (documentation) for local tables.Proposing a unique design that is pragmatic given the above considerations, but also very functional:
- Queries can be written using familiar relational attributes in the case of many common attributes (e.g.
loans.loanDate), but will require JSON paths to access some nested attributes (e.g. json_extract_path_text(loans.data, 'status', 'name') which refers to the "name" subfield located within the "status" field).
- In other words, a set of "common" attributes will be available as relational attributes, while other attributes will have to be retrieved from JSON data (in the same table) using the
json_extract_path_text() function. - Access to JSON data takes the form
json_extract_path_text(A, F1, F2, ...) where A is the attribute containing the JSON data, and F1, F2, ... is a list of up to five nested JSON fields which specifies a path into the JSON data. (If this is not clear, it helps a lot to look at an example.) The JSON data include the complete set of data provided by FOLIO, while the relational attributes are created as a convenience to make writing queries easierRefer to https://github.com/folio-org/ldp/blob/master/README.md for the proposed data model for LDP 1.0.
(1) Historical queries using a single schema, (2) ETL, and (3) full relational or star schema are planned for later releases of the LDP but are dependent on the identified critical dependencies and availability of developer resources.Data are available now in the test database sufficient to support at least the Circ Item Detail reportThe test database is currently using the proposed data model. Additional tables will soon be added to support writing the prioritized report queries. |