[FOLIO-354] Integrate VuFind + FOLIO Created: 13/Oct/16 Updated: 10/Sep/19 |
|
| Status: | In Progress |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Umbrella | Priority: | P3 |
| Reporter: | Wolfram Schneider | Assignee: | Ian Hardy |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | sysops_mgt | ||
| Σ Remaining Estimate: | 2 days, 7 hours, 45 minutes | Remaining Estimate: | 2 days, 7 hours, 45 minutes |
| Σ Time Spent: | 3 days, 6 hours, 45 minutes | Time Spent: | 0 minutes |
| Σ Original Estimate: | Not Specified | Original estimate: | Not Specified |
| Issue links: |
|
||||||||||||
| Sub-tasks: |
|
||||||||||||
| Sprint: | |||||||||||||
| Description |
|
VuFind is a library resource portal designed and developed for libraries by libraries. The goal of VuFind is to enable your users to search and browse through all of your library's resources by replacing the traditional OPAC http://vufind-org.github.io/vufind/ This is an umbrella issue to cover the integration of VuFind and FOLIO |
| Comments |
| Comment by Hkaplanian [ 21/Nov/18 ] |
|
Wolfram Schneider, I believe some of this work may already be completed. Have a look at the code that was created to integrate EDS. I believe much of that should work "as-is". It should look and work very much like what Vufind supports today. The one key area known to be missing are holds as it's waiting on some FOLIO functionality in that area to be completed. |
| Comment by Ian Hardy [ 22/May/19 ] |
|
A couple comments on data loading. When records are in SRS the they can be easily indexed into vufind using solrmarc. It would be worth mapping instace records ontovufind's solr schema in order to support discovery of instances when MARC isn't being used. Here's a work-in-progress mapping. https://docs.google.com/spreadsheets/d/16NPZm-b6kGcLIHrcIJud49Lmx3so_kXVxArh5Jkc9u8/edit?usp=sharing For real time availability, holdings/item information, Vufind's Folio driver is realized here. |
| Comment by Demian Katz [ 28/May/19 ] |
|
Ian Hardy, thanks for sharing the progress here – looks like a solid start. Let me know if there's anything I can do on the VuFind side to make the instance record ingest easier (is there a particular FOLIO export format that needs a better VuFind importer, for example?), or if there are any points in the schema you'd like clarified. |
| Comment by Ian Hardy [ 03/Jun/19 ] |
|
Thanks Demian Katz, Rebeca and I had a try at indexing instance records. I put up a work-in-progress script to do that on the folio-tools repository here. Its not trying to map all the fields yet, just the basics to get the realtime holdings/item information working. Right now, I'm not getting holdings information from the minimal records indexed. I'm not defining a record type in those records, but I noticed if I change the record type field to 'marc' and add some dummy data it starts working. Does the record type dictate how the getHoldings() function is invoked? |
| Comment by Demian Katz [ 03/Jun/19 ] |
|
Ian Hardy, VuFInd only searches for real-time holdings information when an ILS connection object has been injected into the record driver. In the default configuration, this is only done for MARC records, because these are the only records that tend to come from an ILS. However, it's easy to add this functionality to any other driver by simply adding an IlsAwareDelegatorFactory to the configuration. You can see the default setup here: This can be overridden through your local module's module.config.php... or you can just edit the PluginManager itself as a short-term solution. It's possible that we may want to create a new FolioInstance record driver – this would allow us to tie the IlsAwareDelegatorFactory to a narrower set of recods, and would also give us a place to hang custom methods specific to the FOLIO instance record, if it turns out that we need any. Let me know if you'd like any help setting any of this up! |
| Comment by Ian Hardy [ 04/Jun/19 ] |
|
Thanks Demian Katz, that makes a lot of sense, I was able to get the availability information for instances locally. Maybe it does make sense to add a Folio instance record driver. One pattern in the instance data is to have the value of the field further classified by an ID: Contributor (primary) "contributors": [
{
"name": "Pratchett, Terry",
"contributorNameTypeId": "2b94c631-fca9-4892-a730-03ee529ffe2a"
}
]
Since the id types can be extended, I think a custom record driver could help resolve these for human readable display. |
| Comment by Demian Katz [ 04/Jun/19 ] |
|
Ian Hardy, it does sound like the ID classification may indeed justify building a custom record driver. Please feel free to open a pull request if you feel you have anything worth sharing... or if you'd like me to set up the skeleton of something you can build upon, let me know and I'm happy to do so. |
| Comment by Michelle Suranofsky [ 26/Aug/19 ] |
|
Ian Hardy or Demian Katz - Do either of you know which FOLIO JIRA covers the 'export' from FOLIO - to be used for indexing in VuFind? I'm looking at https://folio-org.atlassian.net/browse/UXPROD-1911 ...wondering if this is it? thank you! |
| Comment by Demian Katz [ 26/Aug/19 ] |
|
Michelle Suranofsky,
|
| Comment by Hkaplanian [ 26/Aug/19 ] |
|
Michelle Suranofsky and Demian Katz, ViewFind and FOLIO both support OAI-PMH. I believe that is the good path forward here. |
| Comment by Demian Katz [ 27/Aug/19 ] |
|
Hkaplanian, at this point, is the OAI-PMH server a standard feature or a separate module? What metadata formats does it support? Is there a demo endpoint set up that we can test harvesting against? |
| Comment by Hkaplanian [ 27/Aug/19 ] |
|
Demian Katz, the OAI-PMH server for FOLIO is a module that currently supports MARC and DC. I'll contact you in Slack for a demo end point. |
| Comment by Ian Hardy [ 09/Sep/19 ] |
|
Hello, There are two oai-pmh modules on folio mod-oai-pmh and edge-oai-pmh. The edge-oai-pmh implements the oai-pmh protocol as it's api while mod-oai-pmh provides its own api. In the folio reference api environments edge-oai-pmh is running at /oai on port 8000. An API key is required to harvest records. In all the reference environments the api key is: eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ== The key can either be used as a url parmaeter or included in the uri. For example: as a part of the endpoint path: Doing a little testing today, I've had luck w/the oai_dc format using something like this in oai.ini: [FOLIO] url = "https://folio-testing.aws.indexdata.com:8000/oai/eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ==" metadataPrefix = oai_dc idSearch[] = "/^oai:folio.org:/" set = all For now, I'm getting an XML parsing error if I change the metadataPrefix to marc21 |
| Comment by Demian Katz [ 09/Sep/19 ] |
|
Thanks, Ian Hardy! Have you inspected the raw XML output from the server when requesting MARC21 to see if there's an obvious error there – i.e. do you get a parse error in your web browser as well, or is this something VuFind in particular is having trouble with? (I can try to investigate further in the next couple of days if you don't have time – just let me know). |
| Comment by Ian Hardy [ 09/Sep/19 ] |
|
Hi Demian Katz, Thanks, the raw output (link below) looks valid, no errors that are at least obvious to me: The specific validation error from the harvest_oai.php script is: Processing testmarc... Problem loading XML: StartTag: invalid element name; Extra content at the end of the document Looking over my own settings first as a likely source for errors. |
| Comment by Demian Katz [ 09/Sep/19 ] |
|
Ian Hardy, I just tried a harvest with these settings:
[FOLIO]
url = "https://folio-testing.aws.indexdata.com:8000/oai/eyJzIjoiNXNlNGdnbXk1TiIsInQiOiJkaWt1IiwidSI6ImRpa3UifQ=="
metadataPrefix = marc21
set = all
...and it worked fine, retrieving 17 records (was that the expected number?). I think the problem may have been the idSearch[] without a corresponding idReplace[] in your stanza. |
| Comment by Ian Hardy [ 10/Sep/19 ] |
|
Sure enough, Thank you Demian Katz |