[FOLIO-1338] Regression tests fail consistently in PRs Created: 11/Jul/18 Updated: 12/Nov/18 Resolved: 31/Jul/18 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Bug | Priority: | P2 |
| Reporter: | John Malconian | Assignee: | John Malconian |
| Resolution: | Done | Votes: | 0 |
| Labels: | ci, sprint42, sprint43 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | |||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Even though we get repeated successful full-suite regression tests on daily builds, partial (module) tests fail consistently in PRs. This would seem to indicate that there may be an issue in the setup of the Stripes environment in the CI pipeline or perhaps even the set up of the PR tenant. |
| Comments |
| Comment by John Malconian [ 16/Jul/18 ] |
|
Made some changes to the tenant deployment piece of the UI regression tests so that they are more in-step with recent folio-ansible changes. Specifically, utilize the 'create-tenant-admin' role to add a new tenant admin user as well as sync up with the new 'module-sample-data' roles. |
| Comment by Jakub Skoczen [ 18/Jul/18 ] |
|
We see failures in ui-users and ui-inventory but those failures are different. Zak Burke will look into the ui-users side and try to figure out what's up. Niels Erik Nielsen will diagnose the issues in ui-inventory. John Malconian will provide steps on how to get the faulty builds out of the CI. |
| Comment by Zak Burke [ 18/Jul/18 ] |
|
From Niels Erik Nielsen: see also PRs in ui-requests that have consistent test failures. |
| Comment by Niels Erik Nielsen [ 18/Jul/18 ] |
|
Newly generated example for ui-requests: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-requests/job/PR-103/1/UI_20Regression_20Test_20Report/ |
| Comment by Niels Erik Nielsen [ 18/Jul/18 ] |
|
Similar example for ui-inventory: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-inventory/job/PR-223/1/UI_20Regression_20Test_20Report/ |
| Comment by Niels Erik Nielsen [ 18/Jul/18 ] |
|
As can be seen in the reports, the results are the same for all sub-suites for these two UI modules. The initial "should open module and find version tag" succeeds but then something happens, seemingly preventing the subsequent sub-suites from even clicking the module tab to open that module again. This pattern I can only remember having seen in the PR tests - but they've been consistent there for some weeks now. |
| Comment by Niels Erik Nielsen [ 18/Jul/18 ] |
|
This is ui-users, a PR from 13 days ago, same pattern (before other ui-user PR issues shadowed this), https://jenkins-aws.indexdata.com/job/folio-org/job/ui-users/job/PR-433/1/UI_20Regression_20Test_20Report/ |
| Comment by Niels Erik Nielsen [ 18/Jul/18 ] |
|
This is ui-checkout, a PR from 17 hours ago, same pattern: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-156/3/UI_20Regression_20Test_20Report/ The same over time (unless more fundamental issues occur): 20 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-153/3/UI_20Regression_20Test_20Report/ 12 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-153/5/UI_20Regression_20Test_20Report/ Same for different PRs: 20 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-151/2/UI_20Regression_20Test_20Report/ |
| Comment by John Malconian [ 18/Jul/18 ] |
|
I've disabled utilization of the yarn.lock file from folio-snapshot-stable when building the platform bundle. |
| Comment by John Malconian [ 18/Jul/18 ] |
|
Steps that simulate a PR build in Jenkins (using ui-users as example): Prep Project Repo:
Build Platform Bundle:
Deploy new tenant and enable modules:
Run UI Regression tests:
|
| Comment by John Malconian [ 18/Jul/18 ] |
|
I don't think 'yarn link' or 'yarn add' or 'stripes alias' has anything to do with this issue. I seem to get the same results (failures) without doing any of these things. I've also eliminated xvfb-run as a potential issue by alternatively invoking by setting DISPLAY manually. Running 'users' tests directly from platform-core also results in no changes. That is, the results are the same. Interestingly enough, I notice that setting NODE_OPTIONS=--max-old-space-size to any value results in the tests not running at all. That seems strange. |
| Comment by Matthew Jones [ 18/Jul/18 ] |
|
While attempting to reproduce the issue locally (specifically using ui-requests PR-103), prior to running the tests, I loaded my bundle a browser. After logging in and navigating to the Requests app, I receive the following error in the UI: Please refresh the page.
The following occurred resulting in the current page becoming unstable.
ERROR: Cannot read property 'connect' of undefined
in t
in WithStripes(t)
in WithModule(WithStripes(t))
in t
in withRouter(WithModule(WithStripes(t)))
in t
in n
in Connect(n)
in t
in t
in t
in t
in h
in t
in t
in div
in t
in t
in t
in main
in l
in div
in l
in t
in a
in t
in HotKeys
in t
in h
in t
in t
in InjectIntl(t)
in t
in IntlProvider
in t
in t
in t
in WithModules(t)
in Connect(WithModules(t))
|
| Comment by Matthew Jones [ 18/Jul/18 ] |
|
In addition to Requests, navigating to Users and Inventory apps will render the same error. Note: only Requests has been locally linked via yarn add described in the steps above. Check Out, Check In, Codex, and Settings each load fine for me. |
| Comment by Matthew Jones [ 18/Jul/18 ] |
|
The same error, as it reads from a development build: Please refresh the page.
The following occurred resulting in the current page becoming unstable.
ERROR: Cannot read property 'connect' of undefined
in SearchAndSort
in WithStripes(SearchAndSort)
in WithModule(WithStripes(SearchAndSort)) (created by Route)
in Route (created by withRouter(WithModule(WithStripes(SearchAndSort))))
in withRouter(WithModule(WithStripes(SearchAndSort))) (created by Requests)
in Requests (created by Wrapper)
in Wrapper (created by Connect(Wrapper))
in Connect(Wrapper) (created by Route)
in Route (created by RequestsRouting)
in Switch (created by RequestsRouting)
in RequestsRouting (created by Route)
in Titled
in GetContext (created by TitleManager)
in TitleManager (created by Route)
in ErrorBoundary (created by Route)
in div (created by Route)
in AddContext (created by Route)
in Route
in Switch (created by RootWithIntl)
in main (created by ModuleContainer)
in ModuleContainer (created by RootWithIntl)
in div (created by MainContainer)
in MainContainer (created by RootWithIntl)
in Router (created by RootWithIntl)
in Provider (created by RootWithIntl)
in FocusTrap (created by HotKeys)
in HotKeys (created by RootWithIntl)
in Titled
in GetContext (created by TitleManager)
in TitleManager (created by RootWithIntl)
in ModuleTranslator (created by InjectIntl(ModuleTranslator))
in InjectIntl(ModuleTranslator) (created by RootWithIntl)
in RootWithIntl (created by Root)
in IntlProvider (created by Root)
in ApolloProvider (created by Root)
in ErrorBoundary (created by Root)
in Root
in WithModules(Root) (created by Connect(WithModules(Root)))
in Connect(WithModules(Root))
|
| Comment by Matthew Jones [ 18/Jul/18 ] |
|
Also, I'm seeing errors in the console on the login page: checkPropTypes.js:19 Warning: Failed prop type: Invalid prop `component` supplied to `Field`.
in Field (created by Login)
in Login (created by Form(Login))
in Form(Login) (created by Connect(Form(Login)))
in Connect(Form(Login)) (created by ReduxForm)
in ReduxForm (created by Connect(ReduxForm))
in Connect(ReduxForm) (created by LoginCtrl)
in LoginCtrl (created by Connect(LoginCtrl))
in Connect(LoginCtrl) (created by RootWithIntl)
in ErrorBoundary (created by component)
in component (created by Route)
in Route (created by TitledRoute)
in TitledRoute
in WithStripes(TitledRoute) (created by RootWithIntl)
in Switch (created by RootWithIntl)
in Router (created by RootWithIntl)
in Provider (created by RootWithIntl)
in FocusTrap (created by HotKeys)
in HotKeys (created by RootWithIntl)
in Titled
in GetContext (created by TitleManager)
in TitleManager (created by RootWithIntl)
in ModuleTranslator (created by InjectIntl(ModuleTranslator))
in InjectIntl(ModuleTranslator) (created by RootWithIntl)
in RootWithIntl (created by Root)
in IntlProvider (created by Root)
in ApolloProvider (created by Root)
in ErrorBoundary (created by Root)
in Root
in WithModules(Root) (created by Connect(WithModules(Root)))
in Connect(WithModules(Root))
printWarning @ checkPropTypes.js:19
checkPropTypes @ checkPropTypes.js:82
validatePropTypes @ react.development.js:1300
createElementWithValidation @ react.development.js:1388
render @ Login.js:70
finishClassComponent @ react-dom.development.js:13193
updateClassComponent @ react-dom.development.js:13155
beginWork @ react-dom.development.js:13824
performUnitOfWork @ react-dom.development.js:15863
workLoop @ react-dom.development.js:15902
renderRoot @ react-dom.development.js:15942
performWorkOnRoot @ react-dom.development.js:16560
performWork @ react-dom.development.js:16482
performSyncWork @ react-dom.development.js:16454
requestWork @ react-dom.development.js:16354
scheduleWork$1 @ react-dom.development.js:16218
enqueueSetState @ react-dom.development.js:11299
./node_modules/react/cjs/react.development.js.Component.setState @ react.development.js:270
onStateChange @ connectAdvanced.js:205
dispatch @ createStore.js:173
(anonymous) @ createEpicMiddleware.js:59
(anonymous) @ redux-logger.js:1
(anonymous) @ index.js:11
(anonymous) @ loginServices.js:41
Promise.then (async)
(anonymous) @ loginServices.js:40
Promise.then (async)
loadTranslations @ loginServices.js:38
componentDidMount @ Root.js:51
commitLifeCycles @ react-dom.development.js:14361
commitAllLifeCycles @ react-dom.development.js:15462
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
commitRoot @ react-dom.development.js:15603
completeRoot @ react-dom.development.js:16618
performWorkOnRoot @ react-dom.development.js:16563
performWork @ react-dom.development.js:16482
performSyncWork @ react-dom.development.js:16454
requestWork @ react-dom.development.js:16354
scheduleWork$1 @ react-dom.development.js:16218
scheduleRootUpdate @ react-dom.development.js:16785
updateContainerAtExpirationTime @ react-dom.development.js:16812
updateContainer @ react-dom.development.js:16839
./node_modules/react-dom/cjs/react-dom.development.js.ReactRoot.render @ react-dom.development.js:17122
(anonymous) @ react-dom.development.js:17262
unbatchedUpdates @ react-dom.development.js:16679
legacyRenderSubtreeIntoContainer @ react-dom.development.js:17258
render @ react-dom.development.js:17317
./node_modules/@folio/stripes-core/src/index.js @ index.js:26
__webpack_require__ @ bootstrap:767
fn @ bootstrap:130
0 @ bundle.49e549967c2f30ff1a10.js:632252
__webpack_require__ @ bootstrap:767
(anonymous) @ bootstrap:903
(anonymous) @ bootstrap:903
checkPropTypes.js:19 Warning: Failed prop type: Invalid prop `component` supplied to `ConnectedField`.
in ConnectedField (created by Connect(ConnectedField))
in Connect(ConnectedField) (created by Field)
in Field (created by Login)
in div (created by Login)
in form (created by Form)
in Form (created by Login)
in div (created by Login)
in div (created by Login)
in Login (created by Form(Login))
in Form(Login) (created by Connect(Form(Login)))
in Connect(Form(Login)) (created by ReduxForm)
in ReduxForm (created by Connect(ReduxForm))
in Connect(ReduxForm) (created by LoginCtrl)
in LoginCtrl (created by Connect(LoginCtrl))
in Connect(LoginCtrl) (created by RootWithIntl)
in ErrorBoundary (created by component)
in component (created by Route)
in Route (created by TitledRoute)
in TitledRoute
in WithStripes(TitledRoute) (created by RootWithIntl)
in Switch (created by RootWithIntl)
in Router (created by RootWithIntl)
in Provider (created by RootWithIntl)
in FocusTrap (created by HotKeys)
in HotKeys (created by RootWithIntl)
in Titled
in GetContext (created by TitleManager)
in TitleManager (created by RootWithIntl)
in ModuleTranslator (created by InjectIntl(ModuleTranslator))
in InjectIntl(ModuleTranslator) (created by RootWithIntl)
in RootWithIntl (created by Root)
in IntlProvider (created by Root)
in ApolloProvider (created by Root)
in ErrorBoundary (created by Root)
in Root
in WithModules(Root) (created by Connect(WithModules(Root)))
in Connect(WithModules(Root))
|
| Comment by Niels Erik Nielsen [ 19/Jul/18 ] |
|
Definitely looks like you're on to something Matthew Jones, John Malconian. Thank you!
Okay, interesting that it's apparently not a generic problem across all scenarios. |
| Comment by John Malconian [ 19/Jul/18 ] |
|
Also interesting that these errors don't rear themselves on the nightly builds. |
| Comment by Matthew Jones [ 19/Jul/18 ] |
|
Reviewing my terminal output from yesterday, I also observed duplicate warnings on modules like stripes-core which seems suspect. This leads to believe "@folio/requests": "file:../ui-requests" (or other form of linking) may be part of the trouble. During today's stripes call Jason brought up using workspaces in CI. Previously I would not have recommended doing so given the resulting yarn.lock does not fully reflect all dependencies. However, when applied specifically to the context of validating a PR branch against a platform, the workspace approach may be ideal. We just won't get a suitable lock file out of it, and would need the branch/revision under test (which for a PR we have) to reproduce locally if necessary. I'm working on an outline to a proposed testing routine that can be more readily shared between dev/CI and will track that in
|
| Comment by John Malconian [ 26/Jul/18 ] |
|
Niels Erik Nielsen and I have determined the cause of the consistent PR regression test failures. Short story: Longer back story: https://folio-project.slack.com/archives/C210UCHQ9/p1527599753000111 So I rolled the CI back to yarn 1.6.0 and plans were made to refactor the build process to use 'yarn add' in place of 'yarn link' so that we could proceed with upgrading yarn in the CI at some later date. When 'yarn link' was used to link the project to the platform, the CI actually did remove the node_modules directory in the project directory that was created by a previous 'yarn install'. I had determined earlier on that this was necessary in order to get a good build. However, when I replaced 'yarn link' with 'yarn add', I removed this step thinking that it wasn't necessary. Re-implementing the step resolves the issue. Further complicating the troubleshootingof this whole thing is the version of yarn that is used. Niels and possibly others had been using a version of yarn < v1.6.0. Niels had been using v1.0.1. These versions seemed to work (passing tests) when the node_modules directory was present in the project directory. That is why (at least one of the reasons) there was a lot of "BUT the tests pass on my local box!" arm waving. Now that this particular issue is resolved, we probably should evaluate whether this testing process is valid as a best practice - particularly across all modules. So it just works now, but I do not have the NodeJS/yarn expertise to know if this is necessarily a best practice. I expect additional details about best practices will possibly emerge from
|
| Comment by John Malconian [ 31/Jul/18 ] |
|
Changes tested and merged. |