E-Resource usage data: Calculate and visualize statistics (UXPROD-1253)

[UXPROD-2859] Parse and store structured usage data Created: 05/Jan/21  Updated: 22/Nov/21  Resolved: 22/Nov/21

Status: Closed
Project: UX Product
Components: None
Affects versions: None
Fix versions: Kiwi (R3 2021)
Parent: E-Resource usage data: Calculate and visualize statistics

Type: New Feature Priority: P2
Reporter: Kristen Wilson Assignee: Kristen Wilson
Resolution: Done Votes: 0
Labels: eUsage, erm, reporting
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
is blocked by MODEUR-4 OpenAPI specs and JSON Schemas Closed
is blocked by MODEUR-5 Setting up DB schema separation Closed
is blocked by MODEUR-6 Connecting KB APIs and populating TE ... Closed
is blocked by MODEUR-7 Populate TD table with usage data fro... Closed
is blocked by MODEUR-20 Populate database with Package inform... Closed
Epic Link: E-Resource usage data: Calculate and visualize statistics
Development Team: Thor
Kiwi Planning Points (DO NOT CHANGE): 1
Rank: Chicago (MVP Sum 2020): R4
Rank: Cornell (Full Sum 2021): R4
Rank: 5Colleges (Full Jul 2021): R4
Rank: Lehigh (MVP Summer 2020): R3
Rank: U of AL (MVP Oct 2020): R5

 Description   

Purpose: The eUsage app currently stores raw COUNTER files, but does not convert the COUNTER data into structured data that can be used throughout FOLIO. We will need to extend mod-erm-usage to include tables that will store COUNTER data at the title level, as well as identifiers that will allow us to create connections to a local KB or E-holdings title.

Workflow

  1. A COUNTER report is uploaded to the eUsage app (either manually or through an automated harvest)
  2. Each title in the report is compared against the titles already stored in eUsage
  3. If a match occurs in eUsage, the new statistics are added to the existing title record
  4. If no match occurs in eUsage, the system attempts to match the title to a record in the local holdings KB and a create a new title record in eUsage
  5. If the title cannot be matched to the KB, it is reported for manual matching
  6. The reports of the harvest job are made available to the user for review and clean up

Questions

  1. What COUNTER report types are being stored in the eUsage app? COUNTER master reports, which contain all possible data
  2. Will all types of COUNTER reports need this treatment? Or only those that are specifically being used for visualizations and downloads? Only the master reports are currently stored and need to be parsed.
  3. Do we need to allow users to choose the local KB or the E-Holdings KB for this process? Or can we just focus on the local KB for now given that EBSCO is doing their own integration for E-Holdings? We will focus only on the local KB within ERM; E-holdings is out of scope.


 Comments   
Comment by Kristen Wilson [ 05/Jan/21 ]

Original analysis:

Add new data storage functions to existing mod-erm-usage
STATUS:
mod-erm-usage currently loads "counter-reports" (json) and csv files and delivers them back as is (there is no provision for pin-pointing stats for specific KB titles)

SUGGESTED:
Add API: /usage?period=<period> (request body with list of KB title ids) or similar

mod-erm-usage would require: mod-agreements (API tbd)
(note: mod-agreements has a reference in code to mod-erm-usage, so there's a risk of circular dependency, but mod-agreements does not declare that dependency and it's not obvious that it uses the API for anything)

INTERNALS:
Add two new tables for normalization, fast retrieval, and to hold persistent link (uuid) from KB titles.

Usage_Title table
Properties: usage title id, provider id, customerId, itemName, itemDataType, itemPlatform, itemPublisher, itemIdentifiers, KB title id

Usage_Statistics_Line table
Properties: title id, period, counters

Then extend the harvest logic to

Match title entries in the incoming counter report with existing records in the usage titles table (using match properties: provider, title, type, platform, publisher, identifiers)

If no match (we did not harvest usage stats for this title before):

create new usage title record with new usage title id
look up the title in mod-agreements / kb, set the KB resource id on the usage title record
(note: need to verify the search capabilities for this)
If match (already have some stats for that title)

just hold on to title id
and with that id:

Create title level statistic lines for that usage title id

if period already exists for the usage title id, update the entry with new counts
if period doesn't exist, create new entry with usage title id, period, counters

Generated at Fri Feb 09 00:27:26 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.