Content Comparison

Organisational matters

Martina Schildt

• Quick intro: Why this subgroup and Wiki space intro

Questions to clarify:

• Do we need an own Slack channel?

  • we start with staying in #erm-team

  • eholdings subgroup did the same

  • if need be we can create an own channel at a later point

• Do we need an own mailing list to publish and distribute the agenda?

  • we start without

  • if need be we can create a mailing list for this subgroup

• Do we need a separate implementers' page?

  • we add our implementers topic to the ERM implementers page

  • in addition: we will have a topics list as the eholdings subgroup does

• Who would volunteer to take minutes?

  • Felix: If there were multiple people taking notes, I would volunteer.

  • Sabrina: In this case, I would volunteer too

  • Karina: I can also take notes next time

• Who would volunteer to note attendance?

  • Is not needed for this group

  • Everyone is welcome to add their names to the members list (will be created within the next days)

•  Martina Schildt create future topics page

https://folio-org.atlassian.net/wiki/x/MYFRC

•  Martina Schildt create members page and circulate

https://folio-org.atlassian.net/wiki/x/gwDcAg

Update on changes to Local KB

Owen Stephens

Link to slide deck:

• Upcoming changes March 2024 on Local KB Synchonization

• already increased speed of data ingest

• when doing testing in GBV loading GOKB took ~ 24 hours

• behaviour until now: titles are treated as universal -

• same titles from different sources (e.g. GOKB or other)are treated as the same title

• IDs used to match titles:

  • ISSN, ISBN, DOI, ZDB ID (unique ID used in Germany)

• this mechanism has not always worked well even if there was only one external source

• sometimes Local KB treated 2 titles as one where GOKB treated them as 2 > seemed like a title was missing when it was not

New approach

• Works will be treated as source specific

• incoming titles require a work source identifier

• if there are 2 different work source IDs for a single title > 2 separate titles will be created in local KB

• for GOKB: GOKB reference title UUID

• for tenants only using GOKB for data this means we mirror GOKB data much more precisely

• for tenants using multiple data sources (e.g. file imports as well as GOKB) titles will appear multiple times unless the same work source ID is used on import

Transition between old and new approach

• because existring systems will not have work source ID for each work, this will need to be added

• new approach does fall back matching on old IDs if incoming work source ID does not already exist > existing titles are used as correctly as possible

• titles will only get a work source ID added as they are matched

• for GOKB users this means re-harvesting the full GOKB

• Bernd: what happens if titles were merged wrongly?

• Owen: will work out and end up in the right state as soon as all the system has re-harvested all titles

• on harvest should do the job, but maybe it needs 2 harvests

• Bernd in chat: In the local KB, but what about the agreements? As a result the wrong title could be attached?

• Owen: if the title is directly attached to an AGL this can happen - system cannot know whteher that is the title that was meant to be matched - TO DO work on reports to inform users what happened / which title was matched

• title matching method is set by environment setting (system admin)

• Poppy defaults to old approach but can be switched to new

• Quesnelia defaults to new approach but can be switched to old

• not recommended to switch multiple times

• we have tested switching from old to new but not the other way around

• on Snapshot GOKB is harvested every day without errors

GOKB harvesting

Currently using an approach called “OAI-PMH”

• that is package based

• even a single and minor change to a package results in a complete update of the whole package

• if a package fails, we do not get any data from that package

Working on switching to a new approach called the “scrolling API”

• separate data feeds for package metadata (no titles) and titles

• this means we can partially ingest packages - failure of a single title does not affect rest of the package

• changes can be taken from title level

New approach

• moves work outside FOLIO

• New API called “pushKB”

• new external service also called “push KB”

• can be used for any data

Development status

• pushKB API has been added to FOLIO but is not active by default

• cannot be active at the same time as current OAI-PMH approach

• external service is work in progress

• most likely expect external service to be tested against Quesnelia release

• Felix in chat: Q for after Owen's presentation: Has logic changed for matching of GOKb package titles?

• if new titles are added or old titles are deleted from packages

• Owen: for package titles = PCIs is a combination of title record in FOLIO, package record in FOLIO and platform the title is on

• if we see a different title, this new title will be added - new PCI

• the old one would be treated as a deletion

• Christin in chat: Thank you Martina and Owen. I have to go to the next meeting. It would be great if the presentation could be made available. Then I'll take them with me to the rest of the GOKb team. Thank you 🙂 bye

•  Owen Stephens Create a ticket for working on reports to inform about AGL title matching

•  Owen Stephens Share presentation