Introduction

<<explanation of the testing process, generally>>

Testing Process: Examples From Live Libraries

5 Colleges

• Negotiate scheduled upgrade with our hosting vendor. We make sure that the upgrade schedule avoids major campus events and maximizes staff availability. 

• We always use the same dry run environment that is a snapshot of our production

• Rename our Slack Channel from last flower to current release. When we get back to Massachusetts, this will be from #poppy-upgrade to #quesnelia-upgrade

• Our FOLIO Systems Coordinator ensures environment is ready (upgrade scripts are run, permissions are set, etc.)

• Prepare spreadsheet to log issues. This is where we track if the issue is

  • Can be fixed by hosting prior to upgrade

  • Staff should be aware or will require a workaround

  • blocks the upgrade

• Committees review their UAT forms to make sure it covers new functionality

• Committees also request if new functionality needs additional time to be reviewed before being rolled out to staff (Lists App Beta)

• We have 1 week for all committees to do UAT

• Each committee have their own process. The focus is End to End tasks and workflows that involve multiple FOLIO Apps or cross app interaction

• Issues are tracked in our spreadsheet. Issues are determined to be fixable, blocking or “beware”. Issues with beware are often surprising functionality that we didn’t see or experience before (Call Number Browse). We report all issues that are not documented to hosting provider

• When a test fails, user report the test the was performed and the error that occurred. The Systems Coordinator will attempt to replicate the test and identify the sources of the issue.

  • `If it is a configuration issue that can be set locally, it is added to the post-upgrade script.

  • If it is a known issue, it is highlighted to remind users to be mindful on upgrade

  • Otherwise it is reported to EBSCO with the error message, steps to replicate and screenshots

    • The EBSCO response to logged to assist in determining if this will be a blocking issue

• Some committees work together in a meeting to do parts of the UAT. Others work asynchronously

• We evaluate problems to determine if they should block the upgrade. Criteria include:

  • Is it new functionality? If it was not part of previous workflows, it is less likely to disrupt work

  • Is there a workaround and how onerous is the workaround?

  • How frequently does the process need to happen?

  • How many staff are affected?

• At the end of the week, the chairs of the committees approve or don’t approve of the upgrade. And yes, there was a time when we didn’t approve an upgrade. We want “Can we live with it” rather than “Is everything perfect”.

Cornell

SPL

Main process:

• Check for bug fixes, features (new bugs), or enhancements

• Get the internal subject-matter experts to review tests and assemble a team to do the testing.

• Thow away any spreadsheet tracking you have and use a Test Management System (see Must Haves below).

• Schedule (in house or with your vendor) a fresh copy of Production to Test.

  • Having a second copy of your catalog pointed at your test environment is highly recommended.

• As staff test, the IT Department tests workflow tools, scripts, or SIP services.

• As issues are discovered during testing, other staff validate and document findings.

• After testing, review with all team leads failures or new workarounds.

• Update documentation and standard operating procedures, then communicate the changes to staff.

Must Haves!

• Proper Prior Planning Prevents Poor Performance 

  • Check for bug fixes, features (new bugs), or enhancements

  • Schedule time with your staff.

• Organizational buy-in and dedicated staff time

• End user investigation by just “doing the regular job” in both systems.

• NO SPREADSHEETS (never, ever again)

  • Use a Test Management Software – QaraTMS or Quack (we liked them both).

• Review all findings and give everyone’s concerns equal weight when considering the go/no-go before upgrade.

• Benefits of organized processes

  • No (fewer) surprises. Nope. None, never a surprise. 

  • Repetition brings smoother upgrades 
    

• Complaints Feedback management

  • Ask testing staff how testing went before upgrading production and gather feedback for improvement next time.

  • Having staff volunteer to help test mitigates resentment if things weren’t “tested enough” before upgrades and offers a way to learn more about FOLIO and how JIRA works.

Stanford

TAMU

Tips and Takeaways

<<maybe add some tips and takeaways that don’t fit anywhere else, or fold these into the examples from libraries>>

Sample Checklists and Documents

<<This is the most important part. Pull out links to documents here.>>

• The “HC” checklist

• Another google sheet?

• Five Colleges

  • Example of Metadata Management Committee’s UAT for Poppy

  • Example of ERM Committee’s UAT for Poppy

  • Google Form used for all FOLIO flower releases for Data Import/Data Export/Bulk Edit

  • Log of issues encountered with Poppy upgrade

FAQ

Glossary

• API

• Bugfest

• CSP

• Flower Release

• Google Sheets

• Integration

• Migration

• QaraTMS

• Quack

• Release Digest

• Release Notes

• TestRails

• Upgrade

See Also

FOLIO Implementers Presentation, WOLFcon 2024 <<link to recording here>>