Requirements for Ongoing Patron Loading

• FOLIO can accept a flat file of user information to be used to create, update, or delete patron records in bulk. 
• Libraries can designate the match point in the flat file to the match point on the user record in FOLIO.
• Libraries can designate how to match additional fields in the flat file to fields on the user record in FOLIO. We would not expect this to be in the GUI, but it should be able to be controlled programatically.
  • For example, libraries can designate that the flat field NetID maps to the FOLIO field username; the flat field Unique ID maps to the FOLIO field externalSystemID; the flat field Department maps to the custom FOLIO field Department; and so on.
• Libraries identify which action to take on each patron record through a field in the flat file. 
  • Examples - there could be three actions - create, update, delete; or two actions - create/update, and delete 
    • It may be difficult to know if a user record in the flat file doesn’t exist in FOLIO (in which case it should be created) or already exists in FOLIO (in which case it should be updated) when the flat file is being constructed.
    • It may also be the case that libraries will want to handle deletes separately from create/updates. For example, you may want to create/update daily, but only delete once a month. I don't know that that matters in terms of functional requirements.
• Using that flat file of user information:
• If a user record does not already exist with the identified match point, FOLIO creates a new user record.
• If a user record already exists with the identified match point, FOLIO should update the user record with the information in the flat file.
  • If a user is no longer entitled to library services, libraries may choose to use the update function to set/change an expiration date, or set the patron as inactive, rather than deleting. 
• If a user record has protected fields, the FOLIO update process does not change the value in those fields.
• If a user record exists in FOLIO with the identified match point, and the flat file indicates that the user record should be deleted:
  • If no dependencies exist, FOLIO should delete the patron record;
  • If dependencies exist, FOLIO should not delete the patron record. 
• Whenever this FOLIO user update process is run, it should provide output during / once complete to allow libraries to know if the process was successful and/or what errors occurred.
• Mod-user-import already does this.

At Duke, we do a nightly overlay of all our user records. It’s ~60,000 records, and it runs in ~one hour, at 9 PM every evening.

We would expect that level of performance at a minimum from a FOLIO bulk user update process.

We would like, in moving to FOLIO, to get to a process where we are only loading a changes file as opposed to updating all users. (Which is what the requirements listed are describing, generally.) But since we haven't been doing that, we don't know what our performance load would be like in our current environment.

• Up to now flat incremental CSV import (create, change) from Campus ID provider via ILS API but in the process of changing processes to a "import on demand", following requirements for latter "on demand" approach:
• Future Setup: Campus IDM → populates intermediate database including events and data for create, update, delete on demand (not only once a day)→ Middleware with live listener on intermediate DB processes own business logics → Import FOLIO User via API call
• Create/Update FOLIO user via API call (matching point is card reader number (barcode) or maybe another external ID)
  
  • UUID only matching point is not sufficient
  • if user exists = update
  • if user does not exist = create
  • custom fields have to be populated
• Deletions are not processed immediately but
  • Users flagged as "Exmatriculated" or "Deleted" by Campus IDM get first an "Exmatriculated" or "Deleted" Folio Patron Block and set as inactivate and/or change an expiration date
  • Requirement: Ability to set blocks via API as part of patron updating.
• User purge
  • we need a daily/some report on users with block XYZ have dependencies (open loans, fees/fines, ILL, maybe more) in order to clarify these incidents before deletion
  • real record deletions are processed in a time interval (1x year, January); data kept for statistical purposes until January

• Data changes are instantly

• Right now we are using OCLCs IDM Connector
• It can connect to different Targets (this can be LDAP, Sisis Sunrise or LBS4), to replicate user data between this Systems, this replication is live, not only once a day
• we need the possibility to define different Targets because we are importing our students user data from an LDAP System (one-way) and the public user data from the sisis sunrise System of the bavarian state library (both ways)
• this Targets need to support different mappings (Right now via xslt), to merge the data into our user profiles and back to the System of the bavarian state library
• only needed data fields are transported (GDPR)
• following Tasks are needed: add user, update user, block, unblock, delete user (is not deleting directly but putting a Special block on the user, the user is deleted once a day if no dependencies exisit)
  • generally - students who leave are expired and then deleted at the end of the semester. staff who have left get deleted.
• For the Task add user a duplicate check is performed (Name, Birthday, LMUID), if 2 or more duplicates are found the task stops and we are informed, if one duplicate is found the duplicate is updated.

• Data changes are instant
  • About 500 Tasks a day

Current process:

• Central IT / Identity Management provides a nightly file of about 54,000 records, derived from PeopleSoft and Workday, to feed Voyager's patron table
• Contains basic bio-demo data, EmplID ( = bar code), NetID
• The key columns in terms of determining circ privs are:
  • A status_code column with the values: Active, Retired, Admin WD, Completed, Deceased, End Study, LOA, Retired, Univ WD
  • And job_designation: staf, acad, fac, affil, ee, gm, gr, la, trst, ug, vm
• Some logic internal to Voyager maps the appropriate permissions to the user records
• The current job runs within 2 hours, early in the morning after the nightly Voyager backup is done.

Future state:

• Continuing with this feed would be the easiest route, not necessarily the most contemporary or cool.

• The job would need to complete within 2 hours due to our extensive Library hours, and presumably other FOLIO backup processes that would need to complete beforehand.

Current process:

1. Is the user record protected by the Library?
   1. If yes, don't overwrite any part of the record.
2. Does the user record belong to one of a specific list of classifications, affiliations, or student statuses?
   1. If yes, assign privileges accordingly.
3. Does the user record belong to one of a specific list of ancillary affiliations?
   1. If yes, assign privileges accordingly.
4. Has the user's "active" status changed?
   1. If yes, overwrite status field.
5. Does the user record have an e-mail address?
   1. If yes, does FOLIO already have an address?
      
      1. If  yes, do not overwrite the address.
   2. If no, assign a pre-determined address.
   3. If yes, overwrite status.

This is what I have regarding the logic of our user import.  What we would need:

• The file is ~32,000 records.  It would have to be processed in less than an hour.  It currently runs in a very few minutes, and we'd like to keep that timing as much as we could.
• We would like the option of full records, or certain fields in user records, to be protected from the overwrites that happen during daily imports.
• Item number 5 implies what we want from FOLIO rather than what we currently do in OLE.