2024-09-03 Better Sample Data Meeting notes

1. Split the project in three:

   1. Golden copy for Bugfest environment.

   2. Work on a small data set for FOLIO Snapshot.

   3. write robust data anonymization scripts

Follow up on activities since last meeting:

Plan - who will take on what task:

a. Golden Copy / Yogesh Kumar

uChicago can provide data for inventory, but not sensitive data as circulation, users, acquisition etc.

Michigan State: some hesitation at management level.

A conversation also came up in the ARLEF group where a couple of large institutions said they could maybe provide data.

The QA team is working on fixing the current Bugfest Data set (September).

Focus work on b. and c. - while waiting for approval of data for the golden copy.

b. Snapshot environment / Charlotte Whitt

Charlotte has reached out to ID Devops re. adding data to monthly build environment: https://folio-quesnelia.dev.folio.org/ (the link is not up and running yet)

If this is to take too long, Charlotte will look into other options.

Recruit members from MM-SIG

Write up Data Import Job profiles

Data Import Job Profile which can import ~ 100 bibs in MARC 21 and create instance, holdings, item (corresponding to the locations we have set up).

Charlotte will create a reasonably file with anonymized MARC records.

Load MARC Authority data

Charlotte to fix the 36 instances with Instance source = FOLIO

Then later we can move on to Order/Order lines data. We will ask the MM-SIG, the Acquisition SIG, the ERM-SIG to review the entered data.

c. Status on writing robust data anonymization scripts / Lee Braginsky

• Replace PII with randomly generated data

• Scramble loan history

• Scramble orders, invoice amounts, fund codes

• Replace vendor names with randomized names

• Strip out staff notes with initials, etc.

• One set of data for the general environment, and perhaps a second sample set for the ECS environment

  • Get this from a consortia!

2. Review timeline document

• Draft - timeline document -

We will split it up the the three tracks as listed above.

Yogesh will update the Golden copy (Bugfest) section.

Charlotte will update the FOLIO Snapshot section

Lee will update the time line doc for his work on anonymization scripts.