[FOLIO-1043] integrate ui-testing with PR Created: 30/Jan/18 Updated: 20/Mar/19 Resolved: 20/Mar/19 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Umbrella | Priority: | P3 |
| Reporter: | Jakub Skoczen | Assignee: | John Malconian |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | build-release, ci, devops, madrid, sprint31, sprint32, sprint33, sprint34, sprint36, sprint37, sprint38, sprint39, sprint40, sprint41, sprint42, sprint43, sprint44, sprint45 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | 4 weeks, 2 days | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Development Team: | Core: Platform | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
The idea is to speed up construction of the test system (maybe re-use the same system and upgrade selected components only) and the test run so it is possible to integrate it with the PR. 1. John Malconian Wayne Schneider Describe the approach |
| Comments |
| Comment by John Malconian [ 31/Jan/18 ] |
|
The idea, at least initially, is to run the UI regression test suite as part of a PR for any of the Stripes and UI-based repositories, This would be orchestrated by Jenkins. The UI test suite would be run against a pre-built FOLIO backend stack that is rebuilt daily. In order to set a known, good, baseline, the backend stack that is used will be an instance of 'folio-snapshot-stable'. For each PR, the CI will create a tenant on the backend, load a subset of data for that tenant, build a stripes platform which will include the changes for the UI component being testing. The UI regression test suite (or a portion of it) will be run against that stripes build. If all tests pass, the PR will be available to merge to master. If any tests fail, the developer will be notified and the pipeline stops until the issue is fixed and a subsequent PR is run and passes. |
| Comment by Wayne Schneider [ 07/Feb/18 ] |
|
Process document: https://docs.google.com/document/d/1YcHV7Dx-LMGckyLVPu3izLtgfrSR-0rFxNsLbsYuYfI |
| Comment by John Malconian [ 13/Mar/18 ] |
|
There are two outstanding issues that need to be added for this to be completed:
|
| Comment by Jakub Skoczen [ 14/Mar/18 ] |
|
We discussed that it would make sense to enable that for ui-users first, sometime next week. |
| Comment by John Malconian [ 30/Mar/18 ] |
|
Couple of things that are holding back this implementation:
|
| Comment by Jakub Skoczen [ 11/Apr/18 ] |
|
We have discussed two things:
|
| Comment by John Malconian [ 11/Apr/18 ] |
|
Test implementation exists on branch folio-1043-test of ui-users. There seems to be a discrepancy in the number of test failures compared to the nightly tests against folio-snapshot during roughly the same period. Some things to implement in an attempt to drill down into the cause of the discrepancy:
|
| Comment by John Malconian [ 12/Apr/18 ] |
|
After fiddling around more with 'yarn linking' and adding the mod-auth-demo-users role to the tenant configuration, the last run produced only three failures. |
| Comment by John Malconian [ 12/Apr/18 ] |
|
Niels Erik Nielsen has pointed out that a couple of the regression tests rely on the existence of the 'diku_admin' user. However, the test is run against a different tenant/tenant admin. |
| Comment by charles [ 12/Apr/18 ] |
|
It looks like the tests are expecting a certain title to be in the codex search, and it can't find any active users. Are we loading the same users and codex titles as in http://folio-testing.aws.indexdata.com/ ? |
| Comment by John Malconian [ 12/Apr/18 ] |
|
The codex search failure fails because the title, Arctic A-Z, exists only in a larger sample inventory data set that is not loaded for PR tests where only a minimal dataset is loaded. I'd rather not have to load the larger dataset for each tenant/PR. Perhaps we can change the behavior of this test somehow? |
| Comment by John Malconian [ 12/Apr/18 ] |
|
For example, the "New POPS" title is included in the minimal dataset so it would be found in all tenant instances (except it wouldn't be the first title found in instances with the larger dataset included. |
| Comment by charles [ 12/Apr/18 ] |
|
Ok-- this sounds good. |
| Comment by charles [ 12/Apr/18 ] |
|
I changed the search term in the codex search test to "POPs in Marine Mammals". |
| Comment by John Malconian [ 13/Apr/18 ] |
|
It appears all issues related to passing/failing UI tests have been worked out. I've also enabled publishing of yarn.lock files in Jenkins (they can be downloaded/viewed per job). I've been able to run the full UI regression suite without error in testing. I intend to implement full-stack PR-based regression testing on two repositories - ui-users and ui-inventory - on Monday morning. We'll see how it goes and make adjustments from there. |
| Comment by John Malconian [ 01/May/18 ] |
|
I've implemented a couple of new features into CI.
runRegression = 'partial' - Run module's UI regression tests only.
Currently, the 'ui-inventory' and 'ui-users' repositories are configured with "runRegression = 'partial'". I'll enable regression testing for additional repositories gradually. |
| Comment by Niels Erik Nielsen [ 01/May/18 ] |
|
Cool! |
| Comment by John Malconian [ 11/May/18 ] |
|
Currently, when a PR submitted for regression testing, the module that is tested is linked into 'folio-testing-platform' and the whole stripes platform is built and posted to Okapi. I'm thinking it would be more efficient to test the module with a more barebones stripes platform that consists only of the module being tested along with common stripes dependencies (stripes-core, stripes-components) and enabling only the backend modules that are explicitly specified as okapi interface dependencies. |
| Comment by Adam Dickmeiss [ 11/May/18 ] |
|
That would be great .. It would also check that the dependencies are right. |
| Comment by Matthew Jones [ 11/May/18 ] |
John Malconian, Stripes CLI does exactly this. From the app module's directory, run: stripes test nightmare |
| Comment by John Malconian [ 01/Jun/18 ] |
|
Implemented 'partial' regression tests for PRs on ui-requests and ui-checkout projects. In total, PR partial regression testing is enabled now for ui-users, ui-inventory, ui-requests, and ui-checkout. Correct me if I'm wrong Niels Erik Nielsen charles, but I don't think there are any other module specific test suites for any other UI module (ui-checkin, for example). So for all other UI modules, we either need to enable full regression tests for PRs or not enable any tests at all? |
| Comment by Niels Erik Nielsen [ 01/Jun/18 ] |
|
Right John Malconian Personally I think non-core module should only run their own tests on the PR. If they don't have any they will not run any. The developer should optimally still run relevant cross-module tests (typically 'test-exercise') or – if that's simpler – the entire suite, locally. |
| Comment by Niels Erik Nielsen [ 01/Jun/18 ] |
|
Regarding running a platform with only the module being tested: https://folio-org.atlassian.net/browse/FOLIO-1043?focusedCommentId=189585&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel I know I've said it elsewhere, but just a note for the record here as well:
|
| Comment by Jakub Skoczen [ 06/Jun/18 ] |
|
We have discussed an alternative approach to yarn linking whereby we specify the dependency of the package under test as the SHA1 of the Git commit. This can be done through a yarn command. The yarn.lock will be retrieved during the build process from a snapshot-stable URL (where it is already retain after each successful build). |
| Comment by John Malconian [ 26/Jun/18 ] |
|
Here is a summary of the latest changes:
I've also learned to never specify 'NODE_OPTIONS=--max_old_space_size...' globally. I spent about 3 days trying to figure out why the regression tests wouldn't run after making the above changes that was due to this. |
| Comment by Matthew Jones [ 27/Jun/18 ] |
|
FYI, I've updated stripe-cli's own jenkinsfile platform to 'none' as well. Technically the CLI doesn't have anything to build in app mode, so I'm wondering if we should introduce a platform value that does even less than 'none'. At this point, do you need
|
| Comment by John Malconian [ 23/Jul/18 ] |
|
Disable regression testing for PRs on ui-checkout, ui-requests, ui-users, and ui-inventory. Blocked by
|
| Comment by Marc Johnson [ 23/Jul/18 ] |
|
John Malconian Can you help me understand why running the tests has been completely disabled? |
| Comment by John Malconian [ 23/Jul/18 ] |
|
Marc Johnson - A combination of the systematic test failures in PRs that are not duplicated on the nightly tests as well as
|
| Comment by John Malconian [ 31/Jul/18 ] |
|
Re-enabled regression testing for PRs for ui-checkout, ui-requests, ui-users, and ui-inventory, In addition, added module tests for ui-organization. Will need to monitor for any signs of
|
| Comment by John Malconian [ 07/Aug/18 ] |
|
Disable regression testing for PRs for ui-checkout, ui-requests, ui-users, and ui-inventory, and ui-organization until
|