| Note | ||
|---|---|---|
| ||
Our goal was to test Q4 release in the similar fashion as a customer would conduct UAT and . We wanted to see if our current development/engineering process can provide desired quality level. After triage we had 67 open defects. This doesn't count 15 defects has been closed with "duplicate" or "won't do" resolutions. Needles to say that, if it would be a true UAT event we would have failed. We don't have enough quality gates to be built into out process that can protect releases from a large number of defects. Hopefully, quality improvement work that we're planning in Q1 and beyond will yield us better results in the futureBug Fest was very useful experience for the FOLIO team. In order to reduce defect leakage, we will need to improve our quality gates and to build additional layers of tests integrated into the CI pipeline. Specific recommendations will be published after the BugFest team's retrospective and will need to be included in the ongoing Q1 work. |
It was the first time we attempted to execute such an extensive manual product test run. That being said, it went relatively smooth for the first try:
- There were relatively few questions about the process in the bug-fest Slack channel at the beginning of the week.
- Participants did't require a lot of support in following the directions for Bug Fest.
- We incurred one performance issue on Tuesday that blocked all the testing activities. It was quickly resolved by Hongwei.
- We were not able to execute all test tasks in the first 2 days and event had to be extended for another 3 days.
- Pass/Fail results can be found here
- Some areas of Folio included into test plan didn't get executed. We have light coverage to the following areas:
- Inventory - Instance, Item, Holdings.
- Loans
- Fees and Fines
- Vendors
- Orders
- We need to do a better job planning task assignments next time.We didn't have good way to migrate reference data from Q3 to Q4
Here are the highlights of the results:
- 82 defect has been logged over the course of 5 days.
- 15 defects has been closed as "duplicate' or "won't do". We have 65 67 active defects remaining.
- 2 issues received immediate attention and are WIP.
- We had 14 participants from across FOLIO (admin, POs, regular testers, library staff)
- Testers logged the most number for defects against ui-users and ui-inventory modules (please see chart below.)
- Most defects against ui-requests has been dismissed because they were logged against incomplete features.
- Link to the live JIRA dashboard is here
