2022-09-09 - Sys Ops & Management SIG Agenda and Meeting notes

Notes taken by Ian:

Operational Needs

link to Ingolf's presentation : WolfCon2022_FOLIO_SysOps_OperationalNeeds.pptx

Kiwi and Lotus docks for full single server install underway

Jason/TAMU maintaining K8s deployment docs; needs updates for Lotus as well

hbz demo is for public. Needs to move more (small) libraries over to FOLIO, or less they move to something else.

Upgrades can fail, as new modules can be deployed but not enabled, leading to system inconsistency.


link to Jeremy's presentation; WolfCon2022_FOLIO_SysOps_SUL_Jeremy.pptx

Stanford moving from Symphony to FOLIO, right in the middle. Speciality libraries (law and medical) also moving to FOLIO, single tenant. Building own Helm chart for deployment, and setting up a training group. see timeline.

If you have JIT (just-in-time) turned on in Postgres, turn it off for major performance improvement


Jason's presentation lost to broken USB

On Voyager for 25 years. FOLIO go-live moved forward due to proprietary software licensing issues at the university level. On premise data centers. Moving internal software development to a microservices model at this same time.

Also runs LDP. Get on reporting as early as possible.

FOLIO clusters are single-tenant (med and main merged). Has Qatar library in the mix as well, same tenant.

Live on Juniper HF #3 until 2 weeks ago -> now on Lotus HF #2. Morning Glory in Spring.

TAMU is reorg'ing, and like all of us, they need more sysadmin-type people.


Q&A

Mark V:
- when migrating at TAMU, HBZ, did you skip or do sequentially? Each major version to most recently HF at the time.
- how long did this take? 15 min for HBZ at 200K, TAMU took 1/2 day or about 4-5 hrs. Running ES reindex took longer after the fact. VuFind reharvest ~8 hrs. FOLIO has no 'offline' mode. let 'local' users into FOLIO for processes like fiscal rollover
- will HBZ be multi-tenant with new libraries? Yes. If more than 1million biblios, separate cluster required. TAMU spins off separate tenants for testing purposes. Keep in mind management of reference data...
- how are these operational needs communicated to dev teams so they can address them ahead of time? in practice, they aren't. Getting people to test is really hard. Getting messages to the right place is a lot of work, too. Pain Points doc is heading to TC to be listened to and hopefully addressed.
- do you see sysops be involved in release acceptance testing? TAMU does their own processes and participates in bugfest

Tod:
- robust pain points document collected... how do we get dev time, and agree on standards?

Mark V
- non-functional requirements should be first-class citizens. More focus on NFRs now that FOLIO is more feature-complete



- What levels of IT support folks do you need for FOLIO v previous systems? TAMU: >=2 people (K8s and DB Admin). Dennis: struggling with 2 people.


Whiteboard:

Operational Needs
- Upgrade Times
- No Offline Mode
- Multi-tenancy
- Get Development Time for Non-functional Requirements
- SysOps SIG <--> Release Acceptance Criteria
- Lack of Instrumentation is an Issue

WolfCon 2022 recordings page: OLF Recordings | Open Library Foundation

WolfCon 2022 session recordings: WolfCon Session Recordings - Google Docs

Links to class photos: IMG_20220831_154521.jpg IMG_20220831_154519.jpg IMG_20220831_154518[1].jpg

Discussion

• Pain Points came up also in other sessions
• App Store will help 3rd parties to publish their own apps. Sys Ops can have a role in it.
• Technical Debt session. A lot of talk about APIs. Inconsistencies in the APIs. Documentation issues. How do we make progress on that? Let's put in specific JIRAs "the documentation for this module is missing". We have to be more specific with the JIRAs. SysOps could work with DevOps teams on the Pain Points. Kitfox is the name of the DevOps team.
• A lot of interest in "platform minimal". Maybe SysOps can support that work. Maybe regression tests can be automated than in any other distribution. SysOps can contribute to this.
• Cyprus CI/CD Pipeline at Stanford. Jeremy will show a demo of that in a future SysOps meeting. FOLIO DevOps meeting would also be interested in that.

Dennis' Slides / Backups / CEPH - a type of file system which is compatible with Amazon S3. Jeremy: That is the way to go.

• scaling issues of a multi-tenant system. Problems can only be discovered if you really have such a system. Example: The container count needs to be enlarged in mod-agreements, otherwise it didn't scale properly. Mark Veksler is aware of that.
• do better integration testing. It gets better every year.

Another interesting session: About deletion. Charlottes's session.

• Tod, Ingolf and Mark Veksler should talk about multi-tenant issues. Florian: We should also talk to Dennis Benndorf then. Some things might make sense to approach at the project level.