2020-10-06 UI Testing Team Meeting Notes

E2E Testing capabilities

• Unsure about capabilities for removing a tenant via API
• Testrail tests are run once per release as a crowd-sourced event. Anton posts the link to the test rails instance. One website is running previous release, and other is running next release
• Environments are “bugfest-goldenrod.folio.ebsco.com” and bugfest-honeysuckle.folio.ebsco.com”
• Data is migrated from golden rod to honeysuckle
• In Q1 system is built from UChicago data set
• In Q2 (golden rod) we migrated the original dataset, then decommissioned 
• In Q3 we will migrate Q2 over to to the Q3 release
• Next year, we will have three releases instead of four, but in both cases this test rail is the final QA event.
• Data is not refreshed. Main objective was to have an enormous data set.
• You cannot have real user data in the system at all, and so there may be a different class of bugs that arise based on an accumulation of usage data in the production environment, but we cannot lift the dataset from a library.
• Tenant that being used for TestRail test is nowhere close to real tenant, so more like a unit test.

Nightmare testing

• platform-core/test/ui-testing/*.js contains most tests, 
• Most is navigating around from one place to the next making sure the UI is ready
• Helpers from stripes testing library 
• Not a great way to track whether something is successful well
• If you click on a button that triggers a drop-down, the drop-down not blurring, therefore not validating. Difficult to simulate subtle UI interactions and sequences https://github.com/folio-org/stripes-testing/blob/master/helpers.js#L722
• Hard to know when you’re in nightmare and when you’re in the browser
• Some cleanup happens, but not a whole lot
• Run as part of the nightly provisioning script, only run against nightly deploy (unless developing)

Questions / Solutions

• One of the challenges that QA world has is that it is difficult to improve the tools
• Symbiotic relationship between development teams and QA.
• Test Rail tests are very declarative and share a form with BigTest tests
• Maybe generate the human driven tests off of the UI tests

• Develop workflows that center around tests that allow collaboration between QA and ui testers

• What are the requirements for the backend for automating the QA setup. 
• The deployment aspect is not core to folio, so there’s very little we can assume
• We know that it’ll be a DOM app
• Want to assume the capabilities for deployment that we have now (EBSCO and AWS)
• Let’s look at 10 tests that are very valuable, and can we implement them with what we have today
• Take Aways
• Anton to give Frontside an account to Test Rails so that we can see the test plan
• Take section by section of tests to automate 
• Implement the nightmare tests and see how many are covered in Test Rail
• How much can be handled with a single set of base data. E.g. if all of that circulation data is effectively read-only, we can run much more of our tests on that data.