Course Reserves: 3rd party integrations such as the library discovery or OPAC layer
(UXPROD-1902)
|
|
| Status: | Closed |
| Project: | UX Product |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None | Parent: | Course Reserves: 3rd party integrations such as the library discovery or OPAC layer |
| Type: | Sub-task | Priority: | TBD |
| Reporter: | Mike Taylor | Assignee: | Mike Taylor |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
| Description |
|
This is the first and most important of the integrations required by
|
| Comments |
| Comment by Mike Taylor [ 31/Oct/19 ] |
|
A dummied-up version of how the integration should look can be seen at (We think Ian Hardy made this by hand-coding a file to import into VuFind.) |
| Comment by Ian Hardy [ 31/Oct/19 ] |
|
May be interesting to peruse the findReserves() and associated methods in vufind. https://vufind.org/wiki/development:plugins:ils_drivers#findreserves These aren't implemented for the FOLIO driver yet, but if it's easy to collect the information they'll need from the mod-courses APIs, that would help smooth the path. |
| Comment by Mike Taylor [ 01/Nov/19 ] |
|
Oh, interesting! I didn't realise VuFind had support for this kind of thing in ILS drivers. That being so, this may be the best approach to take, rather than the script-based approach I advocated over in
Who created and maintained the FOLIO driver for VuFind? Where is the source? |
| Comment by Ian Hardy [ 01/Nov/19 ] |
|
Hi Mike, the FOLIO driver was created by Chris Hallberg and Demian Katz, the source is here: https://github.com/vufind-org/vufind/blob/master/module/VuFind/src/VuFind/ILS/Driver/Folio.php |
| Comment by Mike Taylor [ 01/Nov/19 ] |
|
Thanks, that will be useful to know as we progress this. (Side-comment: ugh, PHP, I thought we were done with that |
| Comment by Mike Taylor [ 12/Nov/19 ] |
|
Demian Katz We'd like to start making progress on this. We think the FOLIO Course Reserves API is now approaching stability and it would not be unreasonable for someone to start work on supporting this API in the FOLIO VuFind plugin. Is there anyone with the inclination and availability to do this? Kurt Nordstrom and I are both available to provide information about the API. |
| Comment by Demian Katz [ 12/Nov/19 ] |
|
Mike Taylor, there is going to be a Zoom call this Friday at 12pm Eastern to discuss next steps in VuFind/FOLIO driver development. I'll be sure that this topic is raised there, and if you'd like to join the conversation, I can share the invitation with you. |
| Comment by Mike Taylor [ 13/Nov/19 ] |
|
Thanks, Demian Katz. All things being equal, I'd prefer not to get sucked into a meeting — unless you think I need to be there to advocate for the allocation of developer time to this work? I think the contract that needs developing to is pretty well specified and I'm more than happy to fill in whatever gaps might exist.
|
| Comment by Demian Katz [ 13/Nov/19 ] |
|
Thanks, Mike Taylor, we've definitely got the call covered, so no need for you to join us – just wanted to extend the invitation in case you had a desire to be there. The only other question is whether there's a test instance somewhere that could be used for testing/development of driver code (or a relatively straightforward process for setting up a local test environment). |
| Comment by Mike Taylor [ 13/Nov/19 ] |
|
Appreciated, Demian Katz, that sounds good to me. Test instance: well, yes, sort of. The FOLIO Snapshot service at https://folio-snapshot-okapi.aws.indexdata.com/ has mod-courses loaded, and you can see the (very very early) UI running on this service at https://folio-snapshot.aws.indexdata.com/cr/courses?sort=name The problem is that the only sample data we have so far is tiny: four courses (with no items reserved to them) and the necessary underlying courselistings, roles, terms, etc. These are records that I created out of thin air so I'd have something to develop the UI against. So they are going to be pretty inadequate. Worse, that data is not always present: it doesn't get auto-added when the system rebuilds. I do it myself by running the create-records script at https://github.com/folio-org/mod-courses/blob/master/src/main/resources/sample-data/create-records — which you can also do provided you have first done a successful okapi login as a user such as diku_admin that has the necessary permissions to add the records. We do have an issue filed for adding some more realistic data: see
I hope this is less than entirely unhelpful. |
| Comment by Demian Katz [ 18/Nov/19 ] |
|
Thanks, Mike Taylor, I guess the question now is what's the next step? Do you want to send me credentials for the FOLIO Snapshot (or are they available in public somewhere) so I can start putting something together? Would you like me to just code something up based on the spec and then pass it along for you to test? I'm flexible, since I don't think this will take too long to get started... just don't want to drop the ball. |
| Comment by Demian Katz [ 18/Nov/19 ] |
|
Super-quick update: I've just started a pull request at https://github.com/vufind-org/vufind/pull/1503 to give us a place to put work. I implemented the really low-hanging fruit of Department and Course retrieval, which if correctly implemented (currently untested) should at least drop those values into the appropriate VuFind search form, even though the code won't be able to retrieve anything yet. VuFind does expect that there should be a way to get a unified listing of all available instructors, which does not appear to be possible with this API. Not sure if there's a way to get that list that I'm missing, but given the way instructors are scoped to courses, I'm guessing not. This isn't a show-stopper, but it is a possible limitation on some functionality. Advice welcome! In any case, I'll be happy to put a little more time and effort into this once I've heard back regarding next steps... but this part was easy enough that it seemed worth doing now to establish a baseline. |
| Comment by Mike Taylor [ 18/Nov/19 ] |
|
Demian Katz Great that you've been able to start already! Many thanks. I have messaged you privately on Slack with suitable credentials, so as to avoid leaving them here on this public Jira. Please do feel free to push on with development: I have no VuFind installation here that I could use for development of my own, so while I could fix that it would suit me better to have someone who knows the software do the work! BTW., not sure if you've looked at the API docs recently (https://s3.amazonaws.com/foliodocs/api/mod-courses/p/courses.html) but I added an Overview section near the start to put all the detailed stuff into context. Hope it's of use.
Ha, interesting, you have landed on a part of the API that I have been wondering about this weekend. You are right that as things stand instructors are scoped to specific courselistings, and there is no simple way to obtain a master list. (Well, you could extract them from each courselisting, but we can all see how unattractive that is.) But Kurt Nordstrom what do you think about promoting instructors to a first-class object, like a rich controlled vocab? I was thinking of doing this from the perspective of being able to re-use instructors for multiple courses, but it would also help out with the VuFind integration. |
| Comment by Demian Katz [ 18/Nov/19 ] |
|
Thanks, Mike Taylor, that's all extremely helpful. I'll keep working on this as time permits, and I'll be very interested to hear more about the instructor issue as that develops. In the meantime, a couple of additional questions about the API: 1.) VuFind deals in bibliographic records, and it appears that this course reserves API deals in item records (which, of course, makes perfect sense). I just wonder whether there's a way I could easily get instance IDs and/or human readable IDs for the records associated with the items on reserve. Of course I realize that I can look everything up if I have to, but I imagine it would probably be significantly more performant if there were a way to get that data included in the API response in the first place. (And apologies if I'm missing something or making an incorrect assumption about which IDs are in play here). 2.) In the case of calls that accept CQL queries, how can I get a list of valid searchable fields? I assume that there's a way to fetch all of the reserves for a particular course or department by querying on the appropriate ID, but I'm not sure which field name to use. |
| Comment by Mike Taylor [ 18/Nov/19 ] |
|
On #1, I don't think I understand what it would mean to place an instance on reserve — how would you move it from its permanent location to its temporary location? Well, you couldn't ... could you? Anyway, I think there is probably no reason why the `reserve` record could not include an instanceId in the copiedItem — Kurt Nordstrom will be able to confirm or contradict, as he is the back-end guy on this module. But what would you want he instance ID for? Presumably to get information about the instance that item is an instance
On #2, you are right that this part of the API is underspecified — in all FOLIO modules. We don't have, and have never found time to create, a machine-readable specification for searchable indexes. In general, you can use any fieldname in the record as a CQL index, though: for example, `id`, `startDate` or `copiedItem.barcode` in a reserve record. You would fetch all reserves for a courselisting with ID 123 using /coursereserves/courselistings/123/reserves. All reserves for a given department — mathematics, say, would be more awkward. You would start by searching /coursereserves/departments?query=name==mathematics and extracting the departmentId from the resulting record — 456, say. Then you would find the courses run by that department using /coursereserves/courses?departmentId=456 and extracting the {{courseListingId}}s of those courses. You would then deduplicate those IDs, since the same department might conceivably offer the same underlying courselisting as more than one cross-listed course. Finally, you would get the reserves associated with those courselistings. Kurt Nordstrom I wonder if we could/should offer some special queries to make this less laborious? |
| Comment by Demian Katz [ 19/Nov/19 ] |
|
1.) I agree that internally it doesn't make sense to reserve instances – it's definitely an item-level operation. But because VuFind operates almost exclusively at the bibliographic level, it would be convenient if the returned information could include instance-level identifiers in addition to item-level ones (perhaps in response to a query parameter, if it doesn't make sense to pull that in all the time). VuFind doesn't display any title information coming from the reserves system; it just looks up a list of bibliographic identifiers, and uses those to retrieve all the details from its internal bibliographic index. If we only get item IDs, there's no way for VuFind to figure out what to do with them. Like I said, I can certainly try to do the item-to-instance lookups to make things work, especially if you don't think there's any other good reason to include that information in the response, but I just have a feeling it will be slow. 2.) VuFind's reserves lookup allows users to filter by instructor and/or course and/or department. The input is instructor, course and/or department IDs, the output is a list of bib IDs that have items on hold for the input filter set. Obviously, the most common scenario is to just pick one of those filters, but sometimes combinations are applied. It looked to me like CQL would be the most straightforward way to implement this, since if I could always talk to the same single API endpoint to get the data I need, then I could just construct an appropriate query for the incoming filter(s). We also need a way to dump out every single reserve (for the local indexing scenario). Thus, I was hoping there would be an API call that gives me everything, and that it could also accept CQL for filtering. I thought that was what /coursereserves/reserves was for, but I guess the data is more granular than I had expected. |
| Comment by Mike Taylor [ 19/Nov/19 ] |
|
1. Your explanation makes perfect sense. I have started a Slack discussion with Kurt Nordstrom about including the instanceId in the reserve record. |
| Comment by Mike Taylor [ 19/Nov/19 ] |
|
Looking again at the RAML and JSON Schema, it does look like there is a not presently a simpler way to get all the reserves associated with a department. The problem is that reserves are associated with a courselisting (i.e. a set of cross-listed courses) but it's the individual courses within that courselisting that are associated with a specific department. I think we could formulate a desired search here, and then think about what it would take to implement it. I guess the bottom line is that you want departmentName in the reserve record? |
| Comment by Demian Katz [ 19/Nov/19 ] |
|
2. "isn't that a searching facility that VuFind provides across the harvested course-reserves data?" – VuFind has two modes for dealing with reserves. One mode harvests the data into a local Solr index and uses it from there; the other mode connects to the ILS in real-time to do lookups. I think FOLIO should theoretically be able to support both of these modes (each of which is desirable under different circumstances), so I am looking to implement support for both options if possible. And regarding your "bottom line" – it would be helpful to have department name in the reserve record, but it would also be helpful to be able to use department as a filter when retrieving reserves. |
| Comment by Mike Taylor [ 19/Nov/19 ] |
|
"VuFind has two modes for dealing with reserves. One mode harvests the data into a local Solr index and uses it from there; the other mode connects to the ILS in real-time to do lookups". Sounds good, and I agree we should aim to support both. Glad to know we have the fallback if we need it, though! In practice, you can most easily use department name as a filter (i.e. a search criterion) only if it's a field in the record. That said, there is now a facility to configure the back-end so it supports joining queries of the kind that don't need data to be replicated. I don't know how to do this, but Kurt Nordstrom does — so that may be the most elegant approach here. |
| Comment by Demian Katz [ 19/Nov/19 ] |
|
Sounds good. For now, I'll concentrate on the "local index" solution, since I expect the current API will require slow-and-expensive operations to get all the data, but that doesn't really matter much if it's only the index-building that takes some time and not the end user experience. We can work through the real-time solution as the second step, to optimize the work I do for the index solution. |
| Comment by Demian Katz [ 19/Nov/19 ] |
|
Another update: my work-in-progress pull request is now feature complete ( see https://github.com/vufind-org/vufind/pull/1503 ) – it works correctly for both index-based and real-time searching. It is also extremely inefficient, and certainly not ready for prime time (for example, filtering of results currently works by loading all the reserves data into memory first, then filtering it), but hopefully it's a good start as a proof of concept, and we can evolve the code as the API develops. |
| Comment by Mike Taylor [ 19/Nov/19 ] |
|
Well, that is amazing! Can we see how the course-reserves data from folio-staging looks in VuFind? |
| Comment by Demian Katz [ 19/Nov/19 ] |
|
If you have a VuFind instance, it should be pretty easy – just set up Folio.ini to point to the folio-staging instance, and then go to the course reserves link in the footer in VuFind. You'll see drop-downs populated with course, department and instructor information. (This is using the default real-time mode). If you want to switch to Solr-based reserves (see https://vufind.org/wiki/configuration:course_reserves for details), the indexing script should correctly populate the Solr index. There is one catch, of course: I tested all of this without having any bib records loaded into my test instance... so I can confirm that VuFind is trying to retrieve the correct bib IDs from the index at the correct time, but my index didn't actually contain data for those bib IDs, so I didn't do an absolutely end-to-end test in that sense. If you're in a better position to do that than I am, I'm certainly happy to help troubleshoot. Right now, VuFind is using the first available hrId as the bibliographic identifier, on the assumption that those are what are being indexed. If you have a different scenario, some code will have to be adjusted (and perhaps some additional configuration options added to Folio.ini to cover different ID preferences). |
| Comment by Mike Taylor [ 19/Nov/19 ] |
|
Sorry, no, I have no VuFind instance — and admit to my shame I have never run one. For my present purposes, it would suffice to see a couple of screen-grabs. |
| Comment by Demian Katz [ 19/Nov/19 ] |
|
Here is a screen shot of the initial "real-time" screen, showing a populated drop-down: Here is a screen shot of the initial Solr-driven reserves screen, showing a populated Solr reserves index: I just can't easily demonstrate what you see when you click through to get to the bibliographic records. |
| Comment by Mike Taylor [ 19/Nov/19 ] |
|
Looking good! I think Kelly Drake is going to be excited by this. |
| Comment by Mike Taylor [ 19/Dec/19 ] |
|
Hi, Demian Katz. Just want to touch base on this, and make sure we're all seeing the situation in the same way. To the best of your understanding, is it the case that the work to import Course Reserves information into VuFind is essentially done, and you're just waiting on us to have a non-trivial data-set so you can exercise that import code in a non-trivial way? |
| Comment by Demian Katz [ 20/Dec/19 ] |
|
Mike Taylor, that's partially true – the code is feature-complete and ready for testing with real data. However, we also discussed some significant possible changes to the API that, if implemented, would require further work on the driver code to adopt. Right now, the code is EXTREMELY inefficient – it essentially pulls down all available data and then filters it in PHP-land, because there's no current way (that I know of) to filter the results being retrieved from the API; this obviously will not scale well. Additionally, because instructors are not currently "first-class citizens" in the API, retrieving the full instructor list requires an extra API call per course, which is also less than ideal. If we can do something to address these two problems, then I think we'll be truly done. |
| Comment by Mike Taylor [ 20/Dec/19 ] |
|
Demian Katz That all sounds familiar — thanks for the recapitulation. I should of course have just re-read this issue, but I was tricked by Jira's habit of "helpfully" hiding almost all the comments. The issue where you have to fetch each courselisting in order to obtain the instructors and build a unified list: am I rightly inferring that this happens only at ingest-time, not at run-time? If so, then I think we would realistically want to hold off on worrying about that: it's an infelicity, but not one that should materially affect the ingest time as you need to traverse all the courselistings anyway. And your other filtering concerns: do those, too, only arise when doung real-time lookups in the FOLIO course-reserves module, or will they also impede your ability to ingest? |
| Comment by Demian Katz [ 20/Dec/19 ] |
|
Mike Taylor, the answer to both questions depends on the configuration of VuFind. If users are configuring and maintaining the optional reserves index, then as you say, we have to look up all of the data at index time anyway, so this isn't really a problem. If they are using the real-time option, though (which is the default setting, because it requires less maintenance), then the inefficient lookups will happen at run-time. We can certainly recommend that users only use the reserves index method with FOLIO, at least for the moment, as a workaround, but in the long term, it would be nice to have greater flexibility and better support for both possibilities. |
| Comment by Mike Taylor [ 20/Dec/19 ] |
|
Thanks, Demian Katz, that's really helpful. My feeling, given the extreme time pressure we're under in building the Course Reserves module is that for now we should stick with the model where VuFind ingests and indexes the relevant information — and we can return to provide better support for the real-time approach further down the line. Kelly Drake, does that sound right to you? However, if taking the ingest-and-index approach means significantly more work for you, Demian Katz, we can revisit that strategy. Is it a matter of taking ten minutes to throw together a couple of scripts, or is it a couple of weeks' development? |
| Comment by Demian Katz [ 20/Dec/19 ] |
|
Mike Taylor, I'm happy to defer to your needs on this. No matter what happens, it's not a large amount of work on my end. If we go with the "recommend indexing" approach, I don't have to do anything – the code is already complete. It's just slightly more work for the end users setting up VuFind, because they have an extra index to maintain (but it's not a big deal). It's actually more work for me if the API is revised, because I'll have to revise my code to match – but that's not a big deal either, as the API is pretty straightforward. I think the only argument for making changes now is to avoid establishing practices that will be broken by future changes... but if you think everything I'm asking for can be implemented in the future through the addition of new calls without breaking existing calls, then there's really no significant risk. I should also note that VuFind 6.1 is now scheduled to be released in early February, 2020. If we decide to go with the existing code, I can merge it any time between now and February to make it into that release; otherwise, it will probably end up in 7.0 later in the year. Again, it makes no real difference to me – just letting you know our timeline in case it matters to you. |
| Comment by Mike Taylor [ 20/Dec/19 ] |
|
Thanks, Demian Katz, all of this is really helpful. Then, yes, let's go with the ingesting solution for now, and I will consider this stream of work "sufficiently finished" that I can concentrate on other areas. I do hope that, down the line, there will come a time when we can all step back, relax a bit, and think about what would be the best approach for end-users, and revise accordingly. But for now, let's just merge what you have now, and go with the forthcoming v6.1. Thanks again! |
| Comment by Demian Katz [ 20/Dec/19 ] |
|
Mike Taylor, is there any chance of richer test data between now and February? If so, it might be wise to wait on merging until we can perform more extensive testing with real data, just in case there are any bugs I missed when testing with the current more limited example data. If not, I'm reasonably confident that the existing code is good enough, and we can always deal with bugs as they crop up. |
| Comment by Mike Taylor [ 20/Dec/19 ] |
|
Yes, Kurt Nordstrom is working on importing the Simmons course data into FOLIO as we speak. I don't know if it'll get done today — if not, the probably not till after Christmas — but it should at any rate be done well before February. (If it's not, we have bigger problems!) |
| Comment by Demian Katz [ 20/Dec/19 ] |
|
Excellent, thanks Mike Taylor and Kurt Nordstrom! Let me know when it's ready (and whether I need to connect to a different instance than the standard test I've been using), and I'll take another look at everything and then (hopefully) merge. No huge rush, as today's my last day until the new year. Happy holidays in the meantime! |
| Comment by Mike Taylor [ 30/Dec/19 ] |
|
Kurt Nordstrom Do you have an issue for the data import work? I would like to block the present issue on it. |
| Comment by Mike Taylor [ 14/Feb/20 ] |
|
Pinging Kurt Nordstrom: Do you have an issue for the data import work? I would like to track its status so I know what if anyhing needs doing on the present issue. |
| Comment by Demian Katz [ 14/Feb/20 ] |
|
Just an update to say that the course reserves pull request has been merged to master in the VuFind project and will be included in the 7.0 release during Summer 2020 (and of course can be downloaded via GitHub and dropped into earlier VuFind releases in the meantime if anyone needs it). As discussed in earlier comments, this is extremely inefficient for use in VuFind's real-time lookup mode, so I really only recommend using it for the local-index method. If the API is revised in future to support the necessary improvements, I'd love to make that better... but it's at least an MVP for now. |
| Comment by Mike Taylor [ 14/Feb/20 ] |
|
Thanks, Demian Katz — so did Kurt Nordstrom's import work proceed to the point where you were able to test the import from FOLIO Course Reserves into VuFind to your satisfaction? |
| Comment by Demian Katz [ 14/Feb/20 ] |
|
Ian Hardy was able to help with some more realistic testing and found/fixed some minor bugs, which is what led me to the conclusion that we could merge the work in progress. I think he's the best person to comment on whether there are any edge cases that need to be added to Kurt Nordstrom's work, as he's a bit closer to it than I am. |
| Comment by Mike Taylor [ 14/Feb/20 ] |
|
All right, that sounds good enough for me! Many thanks for your work on this; and you, Ian Hardy Kelly Drake I think that this can now be closed, and if we run into specific problems with the VuFind integration down the line when we have more data, then we can file specific Jira issues for them. |
| Comment by Demian Katz [ 14/Feb/20 ] |
|
Sounds like a plan to me... and if there are any future VuFind-related bugs, please flag me in the description and/or cross-post the issue to the VuFind JIRA at https://vufind.org/jira to be sure that I see it! |
| Comment by Mike Taylor [ 14/Feb/20 ] |
|
Will do, Demian Katz! Kelly Drake, please close this issue if you are happy to do so. |
| Comment by Mike Taylor [ 28/Sep/20 ] |
|
Kelly Drake, are you happy for this to be closed? |
| Comment by Kelly Drake [ 28/Sep/20 ] |
|
Mike Taylor Done - and it is awesome! (Honestly thought I'd closed this one a while ago) |
| Comment by Mike Taylor [ 28/Sep/20 ] |
|
Excellent, thanks! |