Page Comparison

Attendance

Attendance & Notes

• Today's attendance-taker: Linda (or substitute)

• See attendance list at bottom of page
  

Announcements and Reminders

Sharon

Announcements:

• About the Reporting SIG meeting schedule

  • Meetings are held on the first 4 Mondays of each month at 11:00 am Eastern U.S. time (see this time in your time zone). The first and third Mondays focus on Reporting Development topics, and the second and fourth Mondays focus on Reporting Business topics. 

  • "business" means topics like presentations on reporting functionality and new features, new reporting applications, surveys and studies on reporting, etc.

  • "development" means working on derived tables and report queries for the folio-analytics GitHub repository

  • "workshopping" queries could be scheduled during any of these meetings, and it would be great to have topics and/or questions in advance so we can prepare to walk through the answers/approaches, such as "how to I fix this inventory query to get rid of the duplicates?" or "what is the best way to calculate totals in this finance query?" 

• December 23 meeting is canceled
  

Ongoing Topics:

• Workshopping your queries

  • part of each Reporting SIG business meeting will be devoted to time to work through any query questions you may have

  • please reach out to Christie Thomas if you have a question you would like to "workshop" during an upcoming Reporting SIG meeting

• Impacts of New Fields and Features (Sharon)

  • Are there any new fields or features we should know about from any of the functional areas that may impact reporting?

  • Review and update the Impacts of New Fields and Features on Reporting grid 

• Upcoming Reporting SIG meeting topics (tentative)
  

  • Derived Tables

  • FUNCTIONS

• Any new members?

  • Welcome/introductions

• How to find our latest recordings

  • Recording details available on our Reporting SIG Meetings and Notes page and Reporting SIG Subteam Meetings page

SIG Recruitment:

We will need to be recruiting for a variety of roles in the coming months. Please consider whether you would be interested. Please reach out to Scott Perry or Sharon Markus with any questions.

• Representative for the Documentation Working Group 

  • urgently needed

  • work with reporting documentation team to  update Reporting section of FOLIO documentation for each FOLIO software release 

  • see https://docs.folio.org/docs/reporting/

• Query developers

  • ongoing need

  • work on derived tables and report queries for the FOLIO Analytics repository and to support Metadb

  • see https://github.com/folio-org/folio-analytics

Mapping Working Group

Mike Gorrell

The task:

1. We have over 80 Derived Tables scripts that currently extract JSON objects/arrays

2. Metadb 1.4 can do this for us automatically by creating appropriate MAPPING commands

3. We need to create the MAPPINGs

For today:

Review Metadb documentation which outlines how to create these mappings

Example of a Mapping created from a derived table creation script. From the po_lines_cost.sql script:

• FROM:

jsonb_extract_path_text(pol.jsonb, 'cost', 'listUnitPrice')::numeric(19,4) AS po_line_list_unit_price_phys,
jsonb_extract_path_text(pol.jsonb, 'cost', 'quantityPhysical') AS po_line_quant_phys,
jsonb_extract_path_text(pol.jsonb, 'cost', 'listUnitPriceElectronic')::numeric(19,4) AS po_line_list_unit_price_elec,
jsonb_extract_path_text(pol.jsonb, 'cost', 'quantityElectronic') AS po_line_quant_elec,
jsonb_extract_path_text(pol.jsonb, 'cost', 'additionalCost')::numeric(19,4) AS po_line_additional_cost,
jsonb_extract_path_text(pol.jsonb, 'cost', 'currency') AS po_line_currency,
jsonb_extract_path_text(pol.jsonb, 'cost', 'discount')::numeric(19,4) AS po_line_discount,
jsonb_extract_path_text(pol.jsonb, 'cost', 'discountType') AS po_line_discount_type,
jsonb_extract_path_text(pol.jsonb, 'cost', 'poLineEstimatedPrice')::numeric(19,4) AS po_line_estimated_price

TO this in the Mapping file

CREATE DATA MAPPING FOR json FROM TABLE folio_orders.po_line__ COLUMN jsonb PATH '$.cost' TO 'cost';

Meta analysis. I have written a script that tries to collect all Fields+Tables that are being extracted. This might speed up the analysis

See the Derived Table JSON Mappings Workgroup folder on the Reporting SIG google drive folder to contribute to this project

• mapping assignments to divvy up the work

• examples to see how to do the mappings

• questions about how to do the mappings

Notes:

-multiple derived tables may be extracting the same data fields in the data arrays, so need to make sure we do not have duplicates

-see create json mappings section of Metadb documentation

-What would be the new name of the table after mapping? Is the suffix taken from the value of TO and "pasted" behind the underscores? (They are going to the “__t” tables)I have clarity around which tables the transformed data will appear in. Let’s refer to the documentation https://metadb.dev/doc/#_create_data_mapping

There is an image of the data we are working with:

Image Added

And the CREATE DATA MAPPING commands:

CREATE DATA MAPPING FOR json
    FROM TABLE library.inventory__ COLUMN jsondata PATH '$'
    TO 't';

CREATE DATA MAPPING FOR json
    FROM TABLE library.inventory__ COLUMN jsondata PATH '$.metadata'
    TO 'metadata';

CREATE DATA MAPPING FOR json
    FROM TABLE library.inventory__ COLUMN jsondata PATH '$.tags'
    TO 'tags';

CREATE DATA MAPPING FOR json
    FROM TABLE library.inventory__ COLUMN jsondata PATH '$.tags.tagList'
    TO 'taglist';

Mapping a JSON object creates columns, and mapping a JSON array creates a table.  The only array in this example is tagList.  This results in an inventory__taglist table.  Everything else results in columns in the corresponding parent table, which in this case is inventory__t.The outermost object is a special case because there is no parent table.  So it results in the top-level inventory__t table.But if you imagine there is already a table inventory__t, the top-level columns and nested objects are added to it.