Investigate LDP Hosting Support

• Organize conversation with hosting vendors to identify current offerings, concerns
• Organize conversation with implementers about reporting platforms and hosting plans (blocked by Recruit from new institutions not represented)

Summary of conversation in Sys Ops & Management SIG

• LDP1 Deployment and Operations (Slides from Wayne Schneider at Index Data) 
• Official notes from Sys Ops meeting
• Angela's notes from meeting

Best practices mentioned

• general tips
  • follow the admin guide
  • production database is standalone Postgres VM with dedicated storage, only use it for FOLIO reporting
  • run with network proximity between the FOLIO and LDP databases
  • run all of the LDP processes as cron jobs on Kubernetes (have our own container for FOLIO analytics)
  • for ldpmarc, incremental update is quick, but full update can be pretty long
  • for order of updates, might try LDP1, then ldpmarc, then derived tables
• disaster recovery:
  • some running on Amazon AWS, RDS Postgres for hosting, including disaster recovery (7 days of snapshots)
  • some have local snapshots, up to 28 days
• testing/upgrades
  • run two instances: staging and production
  • LDP1 and ldpmarc are both fairly independent of FOLIO releases, new versions come out on their own cadence
  • when there is a new version of LDP1 or ldpmarc, implement that in staging
  • if it's more than a point upgrade, might invite users to test
  • follow that with upgrade of production
  • review release notes for additional instructions about whether you need to run a full update
  • FOLIO Analytics is tied to FOLIO flower releases, so upgrade those together; include LDP1 in standard FOLIO testing
• logging
  • some use container log aggregation (CloudWatch, or Elastic), so the logs don't go away when the container goes away
  • some use FluentD/Rancher for logs, which then get pushed to Splunk; might need to log to standard error/out instead of to a file
  • probably a good idea to sent an alert for when the jobs fail, but could also check all manually each morning
• security
  • make sure the LDP IP address is only accessible from certain subnets
  • Kubernetes network is namespace isolated
  • set up users with read-only access, use those accounts for connections via tools like Cloud Beaver, then embed those tools in a VM that uses standard university permissions systems (e.g., SSO via Shibboleth); the different read-only user accounts can be granted permissions to just certain types of data, and different staff can be granted permission to use just the VM that matches the permissions they should have
  • "From a security standpoint, no, there isn't SSL all the way down to the schema, but we take care of that by controlling access, and FOLIO has the same issue."
• concerns
  • time it takes to update data is long, increases over time
  • no automatic recovery; when process fails need to re-run manually