[FOLIO-1338] Regression tests fail consistently in PRs Created: 11/Jul/18  Updated: 12/Nov/18  Resolved: 31/Jul/18

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Bug Priority: P2
Reporter: John Malconian Assignee: John Malconian
Resolution: Done Votes: 0
Labels: ci, sprint42, sprint43
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File ui-requests-error.png    
Issue links:
Blocks
blocks FOLIO-1043 integrate ui-testing with PR Closed
Duplicate
duplicates FOLIO-1337 investigate the PR testing failures Closed
Relates
relates to FOLIO-1377 debug PR test failure issue in ui-inv... Closed
relates to STRIPES-543 Consider "stripes-framework" to wrap ... Closed
relates to UIU-569 stripes-componets 3 upgrade fails Closed
relates to FOLIO-1378 debug test issues on the vagrant box ... Closed
relates to FOLIO-1344 use stripes-cli in the build and test... Closed
relates to FOLIO-1043 integrate ui-testing with PR Closed
Sprint:

 Description   

Even though we get repeated successful full-suite regression tests on daily builds, partial (module) tests fail consistently in PRs. This would seem to indicate that there may be an issue in the setup of the Stripes environment in the CI pipeline or perhaps even the set up of the PR tenant.



 Comments   
Comment by John Malconian [ 16/Jul/18 ]

Made some changes to the tenant deployment piece of the UI regression tests so that they are more in-step with recent folio-ansible changes. Specifically, utilize the 'create-tenant-admin' role to add a new tenant admin user as well as sync up with the new 'module-sample-data' roles.

Comment by Jakub Skoczen [ 18/Jul/18 ]

We see failures in ui-users and ui-inventory but those failures are different. Zak Burke will look into the ui-users side and try to figure out what's up. Niels Erik Nielsen will diagnose the issues in ui-inventory. John Malconian will provide steps on how to get the faulty builds out of the CI.

Comment by Zak Burke [ 18/Jul/18 ]

From Niels Erik Nielsen: see also PRs in ui-requests that have consistent test failures.

Comment by Niels Erik Nielsen [ 18/Jul/18 ]

Newly generated example for ui-requests: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-requests/job/PR-103/1/UI_20Regression_20Test_20Report/

Comment by Niels Erik Nielsen [ 18/Jul/18 ]

Similar example for ui-inventory: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-inventory/job/PR-223/1/UI_20Regression_20Test_20Report/

Comment by Niels Erik Nielsen [ 18/Jul/18 ]

As can be seen in the reports, the results are the same for all sub-suites for these two UI modules. The initial "should open module and find version tag" succeeds but then something happens, seemingly preventing the subsequent sub-suites from even clicking the module tab to open that module again.

This pattern I can only remember having seen in the PR tests - but they've been consistent there for some weeks now.

Comment by Niels Erik Nielsen [ 18/Jul/18 ]

This is ui-users, a PR from 13 days ago, same pattern (before other ui-user PR issues shadowed this), https://jenkins-aws.indexdata.com/job/folio-org/job/ui-users/job/PR-433/1/UI_20Regression_20Test_20Report/

Comment by Niels Erik Nielsen [ 18/Jul/18 ]

This is ui-checkout, a PR from 17 hours ago, same pattern: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-156/3/UI_20Regression_20Test_20Report/

The same over time (unless more fundamental issues occur):

20 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-153/3/UI_20Regression_20Test_20Report/

12 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-153/5/UI_20Regression_20Test_20Report/

Same for different PRs:
15 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-152/7/UI_20Regression_20Test_20Report/

20 days ago: https://jenkins-aws.indexdata.com/job/folio-org/job/ui-checkout/job/PR-151/2/UI_20Regression_20Test_20Report/

Comment by John Malconian [ 18/Jul/18 ]

I've disabled utilization of the yarn.lock file from folio-snapshot-stable when building the platform bundle.

Comment by John Malconian [ 18/Jul/18 ]

Steps that simulate a PR build in Jenkins (using ui-users as example):

Prep Project Repo:

  • Checkout a "feature" branch of a UI module. Merge master branch into branch.
  • Update the version in the package.json to something unique. e.g. append '-pr-test-2'.
  • yarn install

Build Platform Bundle:

  • Clone 'platform-core' from Git.
  • Grab folio-snapshot-stable's yarn.lock file. e.g. 'wget http://folio-snapshot-stable.aws.indexdata.com/yarn.lock' (NOTE: I recently disabled this step)
  • Add UI module we checked out previously to the platform. For example, 'yarn add file:../ui-users && yarn upgrade @folio/users'
  • Build bundle utilizing Stripes CLI which is installed globally - 'stripes build --okapi http://folio-snapshot-stable.aws.indexdata.com:9130 --tenant pr_433_4 stripes.config.js bundle'. In this case, 'pr_433_4' is the name of the new tenant that will subsequently be created on folio-snapshot-stable. For purposes of testing, you may specify an existing tenant that has already been created from a previous PR or just specify 'diku'.
  • Jenkins uses the NPM package, 'http-server', to serve up the bundle. 'http-server -p 3000 ./bundle &'

Deploy new tenant and enable modules:

  • For the sake of manual testing, this step can be skipped. Use an existing tenant.

Run UI Regression tests:

  • git clone ui-testing repo
  • add the project we are testing. e.g. 'yarn add file:../ui-users && yarn upgrade @folio/users'
  • set env FOLIO_UI_USERNAME to the tenant admin which is always $ {TENANT}

    _admin

  • set env FOLIO_UI_PASSWORD to 'admin'
  • set env FOLIO_UI_URL to 'http://localhost:3000' (where our test bundle is served up)
  • Invoke partial test with 'xfvb-run' - 'xvfb-run --server-args="-screen 0 1024x768x24 yarn test-module -o --run=users'
Comment by John Malconian [ 18/Jul/18 ]

I don't think 'yarn link' or 'yarn add' or 'stripes alias' has anything to do with this issue. I seem to get the same results (failures) without doing any of these things.

I've also eliminated xvfb-run as a potential issue by alternatively invoking by setting DISPLAY manually.

Running 'users' tests directly from platform-core also results in no changes. That is, the results are the same.

Interestingly enough, I notice that setting NODE_OPTIONS=--max-old-space-size to any value results in the tests not running at all. That seems strange.

Comment by Matthew Jones [ 18/Jul/18 ]

While attempting to reproduce the issue locally (specifically using ui-requests PR-103), prior to running the tests, I loaded my bundle a browser. After logging in and navigating to the Requests app, I receive the following error in the UI:

Please refresh the page.
The following occurred resulting in the current page becoming unstable.
ERROR: Cannot read property 'connect' of undefined

    in t
    in WithStripes(t)
    in WithModule(WithStripes(t))
    in t
    in withRouter(WithModule(WithStripes(t)))
    in t
    in n
    in Connect(n)
    in t
    in t
    in t
    in t
    in h
    in t
    in t
    in div
    in t
    in t
    in t
    in main
    in l
    in div
    in l
    in t
    in a
    in t
    in HotKeys
    in t
    in h
    in t
    in t
    in InjectIntl(t)
    in t
    in IntlProvider
    in t
    in t
    in t
    in WithModules(t)
    in Connect(WithModules(t))
Comment by Matthew Jones [ 18/Jul/18 ]

In addition to Requests, navigating to Users and Inventory apps will render the same error. Note: only Requests has been locally linked via yarn add described in the steps above. Check Out, Check In, Codex, and Settings each load fine for me.

Comment by Matthew Jones [ 18/Jul/18 ]

The same error, as it reads from a development build:

Please refresh the page.
The following occurred resulting in the current page becoming unstable.
ERROR: Cannot read property 'connect' of undefined

    in SearchAndSort
    in WithStripes(SearchAndSort)
    in WithModule(WithStripes(SearchAndSort)) (created by Route)
    in Route (created by withRouter(WithModule(WithStripes(SearchAndSort))))
    in withRouter(WithModule(WithStripes(SearchAndSort))) (created by Requests)
    in Requests (created by Wrapper)
    in Wrapper (created by Connect(Wrapper))
    in Connect(Wrapper) (created by Route)
    in Route (created by RequestsRouting)
    in Switch (created by RequestsRouting)
    in RequestsRouting (created by Route)
    in Titled
    in GetContext (created by TitleManager)
    in TitleManager (created by Route)
    in ErrorBoundary (created by Route)
    in div (created by Route)
    in AddContext (created by Route)
    in Route
    in Switch (created by RootWithIntl)
    in main (created by ModuleContainer)
    in ModuleContainer (created by RootWithIntl)
    in div (created by MainContainer)
    in MainContainer (created by RootWithIntl)
    in Router (created by RootWithIntl)
    in Provider (created by RootWithIntl)
    in FocusTrap (created by HotKeys)
    in HotKeys (created by RootWithIntl)
    in Titled
    in GetContext (created by TitleManager)
    in TitleManager (created by RootWithIntl)
    in ModuleTranslator (created by InjectIntl(ModuleTranslator))
    in InjectIntl(ModuleTranslator) (created by RootWithIntl)
    in RootWithIntl (created by Root)
    in IntlProvider (created by Root)
    in ApolloProvider (created by Root)
    in ErrorBoundary (created by Root)
    in Root
    in WithModules(Root) (created by Connect(WithModules(Root)))
    in Connect(WithModules(Root))
Comment by Matthew Jones [ 18/Jul/18 ]

Also, I'm seeing errors in the console on the login page:

checkPropTypes.js:19 Warning: Failed prop type: Invalid prop `component` supplied to `Field`.
    in Field (created by Login)
    in Login (created by Form(Login))
    in Form(Login) (created by Connect(Form(Login)))
    in Connect(Form(Login)) (created by ReduxForm)
    in ReduxForm (created by Connect(ReduxForm))
    in Connect(ReduxForm) (created by LoginCtrl)
    in LoginCtrl (created by Connect(LoginCtrl))
    in Connect(LoginCtrl) (created by RootWithIntl)
    in ErrorBoundary (created by component)
    in component (created by Route)
    in Route (created by TitledRoute)
    in TitledRoute
    in WithStripes(TitledRoute) (created by RootWithIntl)
    in Switch (created by RootWithIntl)
    in Router (created by RootWithIntl)
    in Provider (created by RootWithIntl)
    in FocusTrap (created by HotKeys)
    in HotKeys (created by RootWithIntl)
    in Titled
    in GetContext (created by TitleManager)
    in TitleManager (created by RootWithIntl)
    in ModuleTranslator (created by InjectIntl(ModuleTranslator))
    in InjectIntl(ModuleTranslator) (created by RootWithIntl)
    in RootWithIntl (created by Root)
    in IntlProvider (created by Root)
    in ApolloProvider (created by Root)
    in ErrorBoundary (created by Root)
    in Root
    in WithModules(Root) (created by Connect(WithModules(Root)))
    in Connect(WithModules(Root))
printWarning @ checkPropTypes.js:19
checkPropTypes @ checkPropTypes.js:82
validatePropTypes @ react.development.js:1300
createElementWithValidation @ react.development.js:1388
render @ Login.js:70
finishClassComponent @ react-dom.development.js:13193
updateClassComponent @ react-dom.development.js:13155
beginWork @ react-dom.development.js:13824
performUnitOfWork @ react-dom.development.js:15863
workLoop @ react-dom.development.js:15902
renderRoot @ react-dom.development.js:15942
performWorkOnRoot @ react-dom.development.js:16560
performWork @ react-dom.development.js:16482
performSyncWork @ react-dom.development.js:16454
requestWork @ react-dom.development.js:16354
scheduleWork$1 @ react-dom.development.js:16218
enqueueSetState @ react-dom.development.js:11299
./node_modules/react/cjs/react.development.js.Component.setState @ react.development.js:270
onStateChange @ connectAdvanced.js:205
dispatch @ createStore.js:173
(anonymous) @ createEpicMiddleware.js:59
(anonymous) @ redux-logger.js:1
(anonymous) @ index.js:11
(anonymous) @ loginServices.js:41
Promise.then (async)
(anonymous) @ loginServices.js:40
Promise.then (async)
loadTranslations @ loginServices.js:38
componentDidMount @ Root.js:51
commitLifeCycles @ react-dom.development.js:14361
commitAllLifeCycles @ react-dom.development.js:15462
callCallback @ react-dom.development.js:100
invokeGuardedCallbackDev @ react-dom.development.js:138
invokeGuardedCallback @ react-dom.development.js:187
commitRoot @ react-dom.development.js:15603
completeRoot @ react-dom.development.js:16618
performWorkOnRoot @ react-dom.development.js:16563
performWork @ react-dom.development.js:16482
performSyncWork @ react-dom.development.js:16454
requestWork @ react-dom.development.js:16354
scheduleWork$1 @ react-dom.development.js:16218
scheduleRootUpdate @ react-dom.development.js:16785
updateContainerAtExpirationTime @ react-dom.development.js:16812
updateContainer @ react-dom.development.js:16839
./node_modules/react-dom/cjs/react-dom.development.js.ReactRoot.render @ react-dom.development.js:17122
(anonymous) @ react-dom.development.js:17262
unbatchedUpdates @ react-dom.development.js:16679
legacyRenderSubtreeIntoContainer @ react-dom.development.js:17258
render @ react-dom.development.js:17317
./node_modules/@folio/stripes-core/src/index.js @ index.js:26
__webpack_require__ @ bootstrap:767
fn @ bootstrap:130
0 @ bundle.49e549967c2f30ff1a10.js:632252
__webpack_require__ @ bootstrap:767
(anonymous) @ bootstrap:903
(anonymous) @ bootstrap:903


checkPropTypes.js:19 Warning: Failed prop type: Invalid prop `component` supplied to `ConnectedField`.
    in ConnectedField (created by Connect(ConnectedField))
    in Connect(ConnectedField) (created by Field)
    in Field (created by Login)
    in div (created by Login)
    in form (created by Form)
    in Form (created by Login)
    in div (created by Login)
    in div (created by Login)
    in Login (created by Form(Login))
    in Form(Login) (created by Connect(Form(Login)))
    in Connect(Form(Login)) (created by ReduxForm)
    in ReduxForm (created by Connect(ReduxForm))
    in Connect(ReduxForm) (created by LoginCtrl)
    in LoginCtrl (created by Connect(LoginCtrl))
    in Connect(LoginCtrl) (created by RootWithIntl)
    in ErrorBoundary (created by component)
    in component (created by Route)
    in Route (created by TitledRoute)
    in TitledRoute
    in WithStripes(TitledRoute) (created by RootWithIntl)
    in Switch (created by RootWithIntl)
    in Router (created by RootWithIntl)
    in Provider (created by RootWithIntl)
    in FocusTrap (created by HotKeys)
    in HotKeys (created by RootWithIntl)
    in Titled
    in GetContext (created by TitleManager)
    in TitleManager (created by RootWithIntl)
    in ModuleTranslator (created by InjectIntl(ModuleTranslator))
    in InjectIntl(ModuleTranslator) (created by RootWithIntl)
    in RootWithIntl (created by Root)
    in IntlProvider (created by Root)
    in ApolloProvider (created by Root)
    in ErrorBoundary (created by Root)
    in Root
    in WithModules(Root) (created by Connect(WithModules(Root)))
    in Connect(WithModules(Root))
Comment by Niels Erik Nielsen [ 19/Jul/18 ]

Definitely looks like you're on to something Matthew Jones, John Malconian. Thank you!

In addition to Requests, navigating to Users and Inventory apps will render the same error. Note: only Requests has been locally linked via yarn add described in the steps above. Check Out, Check In, Codex, and Settings each load fine for me.

Okay, interesting that it's apparently not a generic problem across all scenarios.

Comment by John Malconian [ 19/Jul/18 ]

Also interesting that these errors don't rear themselves on the nightly builds.

Comment by Matthew Jones [ 19/Jul/18 ]

Reviewing my terminal output from yesterday, I also observed duplicate warnings on modules like stripes-core which seems suspect. This leads to believe "@folio/requests": "file:../ui-requests" (or other form of linking) may be part of the trouble.

During today's stripes call Jason brought up using workspaces in CI. Previously I would not have recommended doing so given the resulting yarn.lock does not fully reflect all dependencies. However, when applied specifically to the context of validating a PR branch against a platform, the workspace approach may be ideal. We just won't get a suitable lock file out of it, and would need the branch/revision under test (which for a PR we have) to reproduce locally if necessary. I'm working on an outline to a proposed testing routine that can be more readily shared between dev/CI and will track that in FOLIO-1344 Closed .

Comment by John Malconian [ 26/Jul/18 ]

Niels Erik Nielsen and I have determined the cause of the consistent PR regression test failures.

Short story:
Previous to running the 'yarn add file:../project && yarn upgrade' steps inside the stripes platform. 'yarn install' is run in the project's repository, creating the node_modules directory with the project's dependencies. Removing this directory prior to running 'yarn add file:../project && yarn upgrade' results in a good build and regression tests that are expected to pass will pass. I've implemented the change in Jenkins in branch FOLIO-1338_yarn for now. I will test and merge the change when a new folio-snapshot-stable is available. Currently, PR regression tests will not run because folio-snapshot-stable is too old.

Longer back story:
At the end of May, yarn 1.7.0 was released and used by CI. However, this version introduced a bug that effectively broke 'yarn linking' and resulted in failed builds. Some details here:

https://folio-project.slack.com/archives/C210UCHQ9/p1527599753000111

So I rolled the CI back to yarn 1.6.0 and plans were made to refactor the build process to use 'yarn add' in place of 'yarn link' so that we could proceed with upgrading yarn in the CI at some later date. When 'yarn link' was used to link the project to the platform, the CI actually did remove the node_modules directory in the project directory that was created by a previous 'yarn install'. I had determined earlier on that this was necessary in order to get a good build. However, when I replaced 'yarn link' with 'yarn add', I removed this step thinking that it wasn't necessary. Re-implementing the step resolves the issue.

Further complicating the troubleshootingof this whole thing is the version of yarn that is used. Niels and possibly others had been using a version of yarn < v1.6.0. Niels had been using v1.0.1. These versions seemed to work (passing tests) when the node_modules directory was present in the project directory. That is why (at least one of the reasons) there was a lot of "BUT the tests pass on my local box!" arm waving.

Now that this particular issue is resolved, we probably should evaluate whether this testing process is valid as a best practice - particularly across all modules. So it just works now, but I do not have the NodeJS/yarn expertise to know if this is necessarily a best practice. I expect additional details about best practices will possibly emerge from FOLIO-1344 Closed .

Comment by John Malconian [ 31/Jul/18 ]

Changes tested and merged.

Generated at Thu Feb 08 23:12:37 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.