Implementers' Guide to FOLIO Upgrade Testing

Owned by Tara Barnett
Last updated: Oct 19, 2024 by Darsi Rueda
5 min read

Introduction

What is a FOLIO upgrade?

FOLIO puts out several major upgrades every year. These are sometimes called “flower releases” because they are named after flowers (for example, “Poppy” or “Ramsons”). Additionally, FOLIO includes several Critical Service Patches for each release. These are often referred to with the abbreviation “CSP”, and numbered in the order in which they are released (for example, “Poppy CSP 3” or “Quesnelia CSP 2”). A flower release contains major functionality updates. A CSP contains crucial fixes for bugs. Formerly, FOLIO used “hotfixes” instead of CSPs. These are no longer used, but you may see references to them around the community and in documentation.

Upgrades are applied by hosting providers (whether that is a vendor or the institution itself). They are not individually applied on personal machines the way one might upgrade personal software, and individual users do not opt in or out. The whole institution upgrades at once.

How do libraries test upgrades?

The process of testing upgrades varies significantly from institution to institution. Below, you will find examples from live libraries outlining their testing processes.

In general, libraries prefer to upgrade a “testing” or “upgrade” environment that is an exact copy of their current production environment before attempting to upgrade their production environment directly. The library tests functionality in this testing environment, including workflows, data integrity, and any integrations. The library then upgrades their production environment and completes the same testing there. The strategy and depth of testing can differ significantly from institution to institution.

Testing Process: Examples From Live Libraries

5 Colleges

  • Negotiate scheduled upgrade with our hosting vendor. We make sure that the upgrade schedule avoids major campus events and maximizes staff availability. 

  • We always use the same dry run environment that is a snapshot of our production

  • Rename our Slack Channel from last flower to current release. When we get back to Massachusetts, this will be from #poppy-upgrade to #quesnelia-upgrade

  • Our FOLIO Systems Coordinator ensures environment is ready (upgrade scripts are run, permissions are set, etc.)

  • Prepare spreadsheet to log issues. This is where we track if the issue is

    • Can be fixed by hosting prior to upgrade

    • Staff should be aware or will require a workaround

    • blocks the upgrade

  • Committees review their UAT forms to make sure it covers new functionality

  • Committees also request if new functionality needs additional time to be reviewed before being rolled out to staff (Lists App Beta)

  • We have 1 week for all committees to do UAT

  • Each committee have their own process. The focus is End to End tasks and workflows that involve multiple FOLIO Apps or cross app interaction

  • Issues are tracked in our spreadsheet. Issues are determined to be fixable, blocking or “beware”. Issues with beware are often surprising functionality that we didn’t see or experience before (Call Number Browse). We report all issues that are not documented to hosting provider

  • When a test fails, user report the test the was performed and the error that occurred. The Systems Coordinator will attempt to replicate the test and identify the sources of the issue.

    • `If it is a configuration issue that can be set locally, it is added to the post-upgrade script.

    • If it is a known issue, it is highlighted to remind users to be mindful on upgrade

    • Otherwise it is reported to EBSCO with the error message, steps to replicate and screenshots

      • The EBSCO response to logged to assist in determining if this will be a blocking issue

  • Some committees work together in a meeting to do parts of the UAT. Others work asynchronously

  • We evaluate problems to determine if they should block the upgrade. Criteria include:

    • Is it new functionality? If it was not part of previous workflows, it is less likely to disrupt work

    • Is there a workaround and how onerous is the workaround?

    • How frequently does the process need to happen?

    • How many staff are affected?

  • At the end of the week, the chairs of the committees approve or don’t approve of the upgrade. And yes, there was a time when we didn’t approve an upgrade. We want “Can we live with it” rather than “Is everything perfect”.

Cornell

SPL

Main process:

  • Check for bug fixes, features (new bugs), or enhancements

  • Get the internal subject-matter experts to review tests and assemble a team to do the testing.

  • Thow away any spreadsheet tracking you have and use a Test Management System (see Must Haves below).

  • Schedule (in house or with your vendor) a fresh copy of Production to Test.

    • Having a second copy of your catalog pointed at your test environment is highly recommended.

  • As staff test, the IT Department tests workflow tools, scripts, or SIP services.

  • As issues are discovered during testing, other staff validate and document findings.

  • After testing, review with all team leads failures or new workarounds.

  • Update documentation and standard operating procedures, then communicate the changes to staff.

 

Must Haves!

  • Proper Prior Planning Prevents Poor Performance 

    • Check for bug fixes, features (new bugs), or enhancements

    • Schedule time with your staff.

  • Organizational buy-in and dedicated staff time

  • End user investigation by just “doing the regular job” in both systems.

  • NO SPREADSHEETS (never, ever again)

    • Use a Test Management Software – QaraTMS or Quack (we liked them both).

  • Review all findings and give everyone’s concerns equal weight when considering the go/no-go before upgrade.

  • Benefits of organized processes

    • No (fewer) surprises. Nope. None, never a surprise. 

    • Repetition brings smoother upgrades 

  • Complaints Feedback management

    • Ask testing staff how testing went before upgrading production and gather feedback for improvement next time.

    • Having staff volunteer to help test mitigates resentment if things weren’t “tested enough” before upgrades and offers a way to learn more about FOLIO and how JIRA works.

Stanford

  • Stanford is self-hosted!

  • We upgrade over a weekend between academic quarters or during a holiday break to minimize impact of downtime

  • Libsys team installs new flower version on dev system (for practice), then test system for staff testing, 3 months before anticipated production upgrade date

    • Assigns new permissions, configures new edge APIs etc.

  • Functional area subgroups organize their own testing, mostly use spreadsheets

    • Spreadsheets include new features to test, all workflows, bug fixes

  • Libsys tests all automated scripts and processes, coordinates integration testing (sip2, Caiasoft, Aeon lookup, etc.)

  • 4 week testing period

  • Slack channel for upgrade testing

  • Subgroups report problems as they find them, discuss on Folio slack, submit Jira tickets as needed

  • If any subgroup identifies a showstopper (some issue with enough negative impact that upgrading is truly problematic) then we will delay upgrade

    • This has happened! We delayed our Poppy upgrade 3 months to go live on CSP3 after originally testing the GA release (part of this delay was waiting for the next intersession)

TAMU

Tips and Takeaways

Sample Checklists and Documents

  • Stanford

    • Example Testing Checklists by area:

  • Five Colleges

FAQ

What integrations should I be checking at upgrades?

ALL OF THEM PLEASE.

Glossary

  • API

    • API stands for “Application programming interface.” Many integrations use APIs to connect to FOLIO.

  • Bugfest

  • CSP

    • CSP stands for Critical Service Patch. A CSP is “a small and targeted software patch to address CRITICAL issues, i.e. issues that significantly disrupt customers’ workflow”

  • Flower Release

    • A “Flower Release” is a major version upgrade in the FOLIO community. You will also sometimes see these referred to as “R1 2025” designating the year and the release. You can read more about releases here: Releases

  • Google Sheets

    • Google Sheets is a spreadsheet application similar to Excel used by many institutions to manage testing. It is part of Google Workspace (G Suite).

  • Integration

    • An integration is (more or less) anything that connects to FOLIO in any way. This may be a discovery layer, an external program connecting through APIs, a self-check machine, etc. Roughly speaking, if it touches FOLIO in any way but is not actually part of FOLIO, it is an integration.

  • Migration

    • A “migration” usually means moving from another ILS to FOLIO. Sometimes libraries refer to upgrades as “migrations”. It depends on the institution.

  • QaraTMS

  • QuAck

    • QuAck is “QuAck is an open-source test management service. It allows to store testcases and test suites and execute them.” You can read more about QuAck here: https://github.com/greatbit/quack

  • Release Digest

    • A release digest is a document put together by the developers explaining what is in each release. For an example, please see: . These are not always available for each release.

  • Release Notes

  • TestRails

    • TestRails is the program used by the FOLIO Testing Community to manage bugfest. In order to participate in Bugfest, users need individual TestRails log-ins. This is coordinated primarily on slack, in the #bug-fest channel.

  • Upgrade

    • A FOLIO upgrade is moving from one version of FOLIO to another.

See Also

FOLIO Implementers Presentation, WOLFcon 2024 <<link to recording here after our session>>. Slides: