[FOLIO-354] Integrate VuFind + FOLIO Created: 13/Oct/16  Updated: 10/Sep/19

Status: In Progress
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: P3
Reporter: Wolfram Schneider Assignee: Ian Hardy
Resolution: Unresolved Votes: 1
Labels: sysops_mgt
Σ Remaining Estimate: 2 days, 7 hours, 45 minutes Remaining Estimate: 2 days, 7 hours, 45 minutes
Σ Time Spent: 3 days, 6 hours, 45 minutes Time Spent: 0 minutes
Σ Original Estimate: Not Specified Original estimate: Not Specified

Issue links:
Relates
relates to UXPROD-993 OAI-PMH Support Closed
relates to UXPROD-351 Patron Portal Support (for Patron Fea... Closed
Sub-tasks:
Key
Summary
Type
Status
Assignee
FOLIO-355 VuFind community meeting, Freiburg Oc... Sub-task Closed Wolfram Schneider  
Sprint:

 Description   

VuFind is a library resource portal designed and developed for libraries by libraries. The goal of VuFind is to enable your users to search and browse through all of your library's resources by replacing the traditional OPAC

http://vufind-org.github.io/vufind/

This is an umbrella issue to cover the integration of VuFind and FOLIO



 Comments   
Comment by Hkaplanian [ 21/Nov/18 ]

Wolfram Schneider, I believe some of this work may already be completed. Have a look at the code that was created to integrate EDS. I believe much of that should work "as-is". It should look and work very much like what Vufind supports today. The one key area known to be missing are holds as it's waiting on some FOLIO functionality in that area to be completed.

Comment by Ian Hardy [ 22/May/19 ]

A couple comments on data loading. When records are in SRS the they can be easily indexed into vufind using solrmarc. It would be worth mapping instace records ontovufind's solr schema in order to support discovery of instances when MARC isn't being used. Here's a work-in-progress mapping.

https://docs.google.com/spreadsheets/d/16NPZm-b6kGcLIHrcIJud49Lmx3so_kXVxArh5Jkc9u8/edit?usp=sharing

For real time availability, holdings/item information, Vufind's Folio driver is realized here.

Comment by Demian Katz [ 28/May/19 ]

Ian Hardy, thanks for sharing the progress here – looks like a solid start. Let me know if there's anything I can do on the VuFind side to make the instance record ingest easier (is there a particular FOLIO export format that needs a better VuFind importer, for example?), or if there are any points in the schema you'd like clarified.

Comment by Ian Hardy [ 03/Jun/19 ]

Thanks Demian Katz, Rebeca and I had a try at indexing instance records. I put up a work-in-progress script to do that on the folio-tools repository here. Its not trying to map all the fields yet, just the basics to get the realtime holdings/item information working.

Right now, I'm not getting holdings information from the minimal records indexed. I'm not defining a record type in those records, but I noticed if I change the record type field to 'marc' and add some dummy data it starts working. Does the record type dictate how the getHoldings() function is invoked?

Comment by Demian Katz [ 03/Jun/19 ]

Ian Hardy, VuFInd only searches for real-time holdings information when an ILS connection object has been injected into the record driver. In the default configuration, this is only done for MARC records, because these are the only records that tend to come from an ILS. However, it's easy to add this functionality to any other driver by simply adding an IlsAwareDelegatorFactory to the configuration. You can see the default setup here:

https://github.com/vufind-org/vufind/blob/master/module/VuFind/src/VuFind/RecordDriver/PluginManager.php#L73

This can be overridden through your local module's module.config.php... or you can just edit the PluginManager itself as a short-term solution.

It's possible that we may want to create a new FolioInstance record driver – this would allow us to tie the IlsAwareDelegatorFactory to a narrower set of recods, and would also give us a place to hang custom methods specific to the FOLIO instance record, if it turns out that we need any. Let me know if you'd like any help setting any of this up!

Comment by Ian Hardy [ 04/Jun/19 ]

Thanks Demian Katz, that makes a lot of sense, I was able to get the availability information for instances locally. Maybe it does make sense to add a Folio instance record driver. One pattern in the instance data is to have the value of the field further classified by an ID:

Contributor (primary)

  "contributors": [
    {
      "name": "Pratchett, Terry",
      "contributorNameTypeId": "2b94c631-fca9-4892-a730-03ee529ffe2a"
    }
  ]

Since the id types can be extended, I think a custom record driver could help resolve these for human readable display.

Comment by Demian Katz [ 04/Jun/19 ]

Ian Hardy, it does sound like the ID classification may indeed justify building a custom record driver. Please feel free to open a pull request if you feel you have anything worth sharing... or if you'd like me to set up the skeleton of something you can build upon, let me know and I'm happy to do so.

Comment by Michelle Suranofsky [ 26/Aug/19 ]

Ian Hardy or Demian Katz - Do either of you know which FOLIO JIRA covers the 'export' from FOLIO - to be used for indexing in VuFind? I'm looking at https://folio-org.atlassian.net/browse/UXPROD-1911 ...wondering if this is it? thank you!

Comment by Demian Katz [ 26/Aug/19 ]

Michelle Suranofsky, UXPROD-1911 Closed certainly looks like it's related, but there also seems to be some confusion over there about exactly what it covers. Certainly, having a MARC export function would be helpful for VuFind work, though something like OAI-PMH support (which I believe is also in progress at some level) could also help. I'm a bit out of touch, though, so I haven't heard anything on the status of these things in a while. I welcome updates!

Comment by Hkaplanian [ 26/Aug/19 ]

Michelle Suranofsky and Demian Katz, ViewFind and FOLIO both support OAI-PMH. I believe that is the good path forward here.

Comment by Demian Katz [ 27/Aug/19 ]

Hkaplanian, at this point, is the OAI-PMH server a standard feature or a separate module? What metadata formats does it support? Is there a demo endpoint set up that we can test harvesting against?

Comment by Hkaplanian [ 27/Aug/19 ]

Demian Katz, the OAI-PMH server for FOLIO is a module that currently supports MARC and DC. I'll contact you in Slack for a demo end point.

Comment by Ian Hardy [ 09/Sep/19 ]

Hello, There are two oai-pmh modules on folio mod-oai-pmh and edge-oai-pmh. The edge-oai-pmh implements the oai-pmh protocol as it's api while mod-oai-pmh provides its own api. In the folio reference api environments edge-oai-pmh is running at /oai on port 8000. An API key is required to harvest records. In all the reference environments the api key is:

eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==

The key can either be used as a url parmaeter or included in the uri. For example:
as a param:

https://folio-snapshot.aws.indexdata.com:8000/oai?apikey=eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==&verb=ListRecords&metadataPrefix=marc21

as a part of the endpoint path:
https://folio-snapshot.aws.indexdata.com:8000/oai/eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==?verb=ListRecords&metadataPrefix=marc21

Doing a little testing today, I've had luck w/the oai_dc format using something like this in oai.ini:

[FOLIO]
url = "https://folio-testing.aws.indexdata.com:8000/oai/eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ=="
metadataPrefix = oai_dc
idSearch[] = "/^oai:folio.org:/"
set = all

For now, I'm getting an XML parsing error if I change the metadataPrefix to marc21

Comment by Demian Katz [ 09/Sep/19 ]

Thanks, Ian Hardy! Have you inspected the raw XML output from the server when requesting MARC21 to see if there's an obvious error there – i.e. do you get a parse error in your web browser as well, or is this something VuFind in particular is having trouble with? (I can try to investigate further in the next couple of days if you don't have time – just let me know).

Comment by Ian Hardy [ 09/Sep/19 ]

Hi Demian Katz, Thanks, the raw output (link below) looks valid, no errors that are at least obvious to me:

https://folio-snapshot.aws.indexdata.com:8000/oai/eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==?verb=ListRecords&metadataPrefix=marc21

The specific validation error from the harvest_oai.php script is:

Processing testmarc...
Problem loading XML: StartTag: invalid element name; Extra content at the end of the document

Looking over my own settings first as a likely source for errors.

Comment by Demian Katz [ 09/Sep/19 ]

Ian Hardy, I just tried a harvest with these settings:

[FOLIO]
url = "https://folio-testing.aws.indexdata.com:8000/oai/eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ=="
metadataPrefix = marc21
set = all

...and it worked fine, retrieving 17 records (was that the expected number?). I think the problem may have been the idSearch[] without a corresponding idReplace[] in your stanza.

Comment by Ian Hardy [ 10/Sep/19 ]

Sure enough, Thank you Demian Katz

Generated at Thu Feb 08 23:05:08 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.