[FOLIO-1344] use stripes-cli in the build and test process Created: 13/Jul/18  Updated: 12/Nov/18  Resolved: 27/Sep/18

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: P2
Reporter: Matthew Jones Assignee: John Malconian
Resolution: Done Votes: 0
Labels: build-release, ci, sprint44, sprint45, sprint46, sprint47
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-1266 Reorganizing CI and nightly with plat... Closed
Relates
relates to FOLIO-1415 SPIKE: consider pre-release syntax fo... Open
relates to STCLI-93 Update app template's package scripts Open
relates to FOLIO-1335 document the differences between dev,... Closed
relates to FOLIO-1338 Regression tests fail consistently in... Closed
relates to UIU-604 Address data dependencies in Nightmar... Closed
relates to UIU-605 Address cross-module dependency in Ni... Closed
relates to FOLIO-1435 Define and document FOLIO platform te... Closed
relates to STCLI-80 Define okapi/tenant defaults Closed
relates to FOLIO-1450 Revise FOLIO release and deployment p... Open
relates to FOLIO-1444 resolve test problem with platform-co... Closed
relates to STRIPES-544 Understand all the stripes-building s... Closed
relates to FOLIO-1370 Platforms fail to install on Windows ... Closed
relates to STCLI-76 Support passing workingDirectory toke... Closed
Sprint:

 Description   

Document the procedure and logic for running tests via a platform rather than via a workspace. The goal is for a developer to run tests against a local environment that most closely resembles that of CI and eliminate dependency discrepancies. Update existing documentation, or create new if needed.

Also identify automation enhancements to the CLI that would make the testing process more efficient for developers.

Publish the documentation on the dev.folio.org



 Comments   
Comment by Matthew Jones [ 19/Jul/18 ]

Given the current issues around testing, I propose modifying this to document a new CLI-based test procedure for both CI and dev. It seems like the right opportunity to bring together a few things on the horizon with platform-core, ui-testing (framework-only), and likely workspaces. By having the CLI encompass as many steps as possible we can help minimize possible discrepancies.

Comment by Matthew Jones [ 19/Jul/18 ]

At first I thought much of this could be accomplished with existing commands, and some of it can. However, during initial testing I ran into the following issues trying to overlap the workspace and CI approach with the CLI:

  • --run option in the CLI is not as flexible as it is in ui-testing. I can specify test collections from an app-context, but not apps in the platform-context.
  • platform-complete's postinstall script fails when run from a workspace install. This is due to the script making assumptions with regards to the location of node_modules folder. This may be the opportunity to replace it entirely anyway as the CLI can generate the same kind of data.
Comment by Matthew Jones [ 03/Aug/18 ]

Update:

In drafting a proposed steps for running platform tests with Stripes-CLI, I couldn't help but draw similarities to that of running tests for an individual app. It makes sense that the procedure for each be as close as possible. Therefore I am drafting one set of steps to work in both cases.

Two branches are available for review:

  • App example: folio-org/ui-users#FOLIO-1344
  • Platform example: folio-org/platform-core#FOLIO-1344

Both have a package script named test-int for integration tests. I have reserved test for when unit tests are present. Please see the build/test sections of each readme.md.

Prerequisites: Each requires a running Okapi instance with a valid tenant.

Option 1: Build and test in one step (app or platform)

This is the quickest to just run tests.

$ yarn test-int

Option 2: Build and test in two steps (app or platform)

Quickest for developers that already have a dev instance running.

$ yarn start
$ yarn test-int --local

Option 3: Build and tests in separate steps (app or platform)

This option performs production builds and also retains build artifacts.

$ yarn build ./output
$ yarn stripes serve --existing-build ./output
$ yarn test-int --local

Regression tests (platform only)

This will run both the platform's own cross-module tests as well as each of the apps sets of tests. Replace test-int with test-regression in the options above.

$ yarn test-regression --local

Notes:

Stripes-CLI is a devDependency of the app and platform and therefore does not need to be globally installed.

For any of the build-related options above, an Okapi URL and tenant ID can be provided as --okapi and --tenant options. Alternatively, in the case of a platform, stripes.config.js can also be updated with an Okapi and tenant (command line option takes precedence).

The --local option is simply an alias for "--url http://localhost:3000" or (whatever host/port is defined to run the local dev server). When either --local or --url is present, the development build and serve steps are skipped.

Additional ui-testing options can be passed via --uiTest.optionName, such as setting username. These can also be set via environment variables that ui-testing knows about. I wonder if for consistency, it makes sense to pass everything on the command line.

$ yarn test-int --uiTest.username admin

One thing missing from this approach is running a platform's cross-module tests with an app build (say for the PR process). To keep things self-contained, I am proposing that instead we run an app's own unit tests via test first (when applicable) and then the app's own integration tests in isolation via test-int. For an app in isolation, build artifacts may not be necessary to maintain, meaning something like option 1 may be sufficient.

I am still looking into how best to handle the ui-app PR scenario w/cross module platform tests. I think that sort of mixed case may be worth running as a subsequent quality gate following the above app-in-isolation approach. The advantage to the app in isolation approach as the first quality gate is simplicity and consistency in setup between dev and CI environments.

Comment by Matthew Jones [ 03/Aug/18 ]

John Malconian, when you have time, I'm interested in your thoughts on the options described above. The following branches should be in a suitable state for experimenting with:

https://github.com/folio-org/ui-users/tree/FOLIO-1344
https://github.com/folio-org/platform-core/tree/FOLIO-1344

FYI, ui-users branch has a .stripesclirc.json file defining the default okapi and tenant settings (since users, by nature doesn't have stripes.config.js). This will take precedence over another .stripesclirc, if you have one, maintained in a higher level directory. The .stripesclirc in general, may be a good way to consolidate many options together, rather than having some on the command line, some in environment variables, etc.

Platform-core tests by now may have drifted from adjustments made to the original tests in ui-testing. Therefore I would not expect all to pass is-is. We can sync them up later once the procedure is looking good.

Comment by John Malconian [ 06/Aug/18 ]

Matthew Jones - Haven't had an opportunity to examine your test branches yet, however, I like the whole approach you've described above in the comments and don't for see any significant problems in terms of running in the CI. I'm a little confused by the presence of the .stripesclirc.json in the project directory. I'm assuming that the default settings will be geared toward developers and that the CI might inject its own .stripesclirc.json in place of the one that might exist in GitHub?

Comment by Matthew Jones [ 06/Aug/18 ]

In conversations, I have found the names "test-int" (integration) and "test-regression" to cause some confusion. These both invoke integration/end-to-end tests that require a live back-end. The main difference is that the former is a subset of the latter. For example, an app's "test-int" tests are also run during a platform's "test-regression". We may want to consider alternatives names for clarity.

Comment by Matthew Jones [ 06/Aug/18 ]

John Malconian Yeah the .stripesclirc in the repo was for developers. Platforms provide such defaults via stripes.config.js, but there is no equivalent in the app-context. However, I'm not too pleased about including a .stripesclirc in the repo as it could cause interference with local CLI preferences, etc. I think what I will do is include these two defaults in the CLI itself. Things like "localhost:9130" and "diku" are a large part of the sample documentation and vagrant boxes, so this may help in other aspects as well.

I also thought about separate .stripesclirc files CI. This might actually be a handy artifact of a Jenkins build, particularly if the .stripesclirc contained all the testing parameters used. The idea being a developer could just copy the CI's .stripesclirc locally and run the tests with all the same values for a PR without having to set environment variables, etc.

Comment by John Malconian [ 06/Aug/18 ]

Just for my own clarification, it appears 'yarn test-int' without 'run' option will run only cross-module tests. Use the 'run' option to run specific app/module-specific tests only in a platform context.

Comment by Matthew Jones [ 06/Aug/18 ]

Yes. The goal with "test-int" was to run the integration tests that come included with a given repo, be it an app or platform. If you want to run tests for a specific app (from within a platform) you can run "test-regression" for all of them, or pass just one app with --run.

--run users

However, if you're looking to just run tests for a given app, I would suggest running "test-int" from a clone of the app in question. At least that's the idea to simplify the test process.

Comment by Matthew Jones [ 06/Aug/18 ]

FYI, I created STCLI-80 Closed and will omit the .stripesclirc from the ui-users branch once complete.

Not sure what I was thinking... The CLI already does this! I've removed the .stripesclirc file.

Comment by Matthew Jones [ 06/Aug/18 ]

Also, regarding app and cross-module tests:

If you decide you want to run both cross-module and an app's tests from a platform at the same time you can do this:

yarn test-int --run WD/users

The "WD" is a token representing the working directory. In the case of platform-core, this says to run the (cross module) tests in the current platform, plus the tests for users.

Comment by John Malconian [ 06/Aug/18 ]

However, if you're looking to just run tests for a given app, I would suggest running "test-int" from a clone of the app in question. At least that's the idea to simplify the test process.

ok. This makes sense. In this case I suspect I would want to run 'yarn start' aka 'stripes serve' and then
'yarn test-int'. However, I notice something odd with this.

 'stripes serve --okapi http://folio-snapshot-latest.aws.indexdata.com --tenant foobar'

'yarn test-int --ui.Test.username diku --ui.Test.password admin --ui.Test.url http://folio-snapshot-latest.aws.indexdata.com --url  http://folio-snapshot-latest.aws.indexdata.com'

I get the following output:

yarn run v1.6.0
$ stripes test nightmare --ui.Test.username diku --ui.Test.password admin --ui.Test.url http://folio-snapshot-latest.aws.indexdata.com --url http://folio-snapshot-latest.aws.indexdata.com
Using URL http://folio-snapshot-latest.aws.indexdata.com
Starting Nightmare tests...
Host:          http://folio-snapshot-latest.aws.indexdata.com


  Module test: users:patron_group
    Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

      ✓ should login as diku_admin/admin (7954ms)
          Test suite   @folio/users:2.12.3
          Live module  @folio/users:2.12.3000375 (http://folio-snapshot-latest.aws.indexdata.com)

I wouldn't expect 'diku_admin' to be able to successfully login because I specified 'foobar' as the tenant. Does 'stripes serve' honor the tenant config?

Comment by John Malconian [ 06/Aug/18 ]

The other issue here from a CI perspective is not a deal breaker, but it's difficult for the CI to determine when to execute the next command after executing 'stripes serve' since it has to wait until the webpack build has been completed when run in daemon mode. That's why I've favored separate steps in the CI for building the bundle and then serving it up, but I don't think 'stripes build' works in an app context.

Comment by Matthew Jones [ 06/Aug/18 ]

In this case I suspect I would want to run 'yarn start' aka 'stripes serve' and then
'yarn test-int'.

If you want to build, serve, and test locally, "yarn test-int" will do this in one go. Only when you provide a "--url" option, will it bypass the build/serve steps, pointing to the build at the "--url" instead.

In fact, I think this is why your foobar tenant is ignored. Your yarn test-int line above includes a url, so your local build is not being used:

--url http://folio-snapshot-latest.aws.indexdata.com
--ui.Test.url http://folio-snapshot-latest.aws.indexdata.com  

Try this command instead:

yarn test-int --okapi http://folio-snapshot-320.aws.indexdata.com:9130 --tenant foobar --ui.Test.username diku --ui.Test.password admin

The underlying "stripes test nightmare" command has the same okapi and tenant options available for building a bundle as "stripes serve".

Comment by Matthew Jones [ 06/Aug/18 ]

it's difficult for the CI to determine when to execute the next command after executing 'stripes serve'

Option 1 should take care of this, invoking the tests only after the build is finished.

The catch with options 1 and 2 is that these both yield development builds (for now). Eventually we should be able to toggle prod settings on/off during both "stripes build" and "stripes serve" ( STCLI-10 Open ). Right now the only way to generate a build with production settings is with "stripes build".

Comment by Jakub Skoczen [ 08/Aug/18 ]

John Malconian will implement the changes proposed in the CI and close this when everything works ok.

Comment by John Malconian [ 08/Aug/18 ]
However, if you're looking to just run tests for a given app, I would suggest running "test-int" from a clone of the app in question. At least that's the idea to simplify the test process.

Some thoughts around this...I initially was thinking that would be "cheaper" and maybe more efficient to run the app-specific nightmare tests in the app context instead of the platform context in CI for PRs, however, there is still the issue of the ensuring there is a backend tenant that was been bootstrapped with the proper modules enabled. Plus we have identified many legitimate issues with code changes when building that code into a platform bundle. So if building the platform with the code that is under review acts a worthwhile quality gate in CI (which I believe it to be) then I'm thinking that all builds and tests in CI should be platform-based as it is currently. This will also give us the ability to run the cross-module tests in CI as well without extra setup.

So the CI process for PRs would essentially consist of the following steps "quality gates":

1. Ensure a successful 'yarn install' of the module.
2. Run Lint.
3. Run unit (non-integration) tests if they exist e,g 'yarn test'
4. Build stripes platform (with configurable platform to use) with the code under review (PR only)
5. Bootstrap test tenant (PR only)
6. Run appropriate module tests (and maybe cross-module tests) in platform context. e.g. 'yarn int-test etc...' (PR only)

This is roughly the same process used today. Just some of the mechanics will be changed.

Comment by John Malconian [ 08/Aug/18 ]

Additional thought regarding Step 4 above. If a specific platform has NOT been configured for the module in the project's Jenkinsfile, then the existing build steps in CI include building a bundle in the app context - 'PREFIX=/usr/local/share/.config/yarn stripes build --output=./bundle'. However, no further integration tests are run in this case currently. We should probably continue this model just to ensure that a bundle can be built successfully even if it is not a platform build. We could potentially take it a step further and generate a module descriptor via 'stripes mod descriptor --strict --full' and post it to okapi, bootstrap the backend tenant and run the module integration tests. Essentially, this gives us the option to run module integration tests for modules that are not yet part of a platform as well as those that are although I think it is preferable from a quality perspective to test as part of a platform.

Comment by John Malconian [ 09/Aug/18 ]

Matthew Jones Any additional thoughts on utilizing workspaces in the CI? Or is the current approach of just cloning projects and platforms sufficient?

Comment by Matthew Jones [ 09/Aug/18 ]

I think workspaces could be helpful for an added level of testing. However, they don't lend themselves to a simple self-contained "clone > install > test" workflow. Its not that workspace steps would be hard, its just that once we introduce a workspace, we are no longer self-contained to a single repo.

Establishing a consistent baseline workflow (for both apps and platforms) that is self-contained gives us a good foundation that is the most portable across dev and CI. In effect we should be able to get the same results from testing independent app and platform builds as we would in a workspace (or the link/alias/file methods). Therefore I think we should prove out the simpler baseline first. Then we can look to extend/optimize via Workspace.

Comment by Matthew Jones [ 09/Aug/18 ]

John Malconian, My apologies I completely overlooked your prior comments from yesterday when writing my reply to your question this morning. I took "current approach" to refer to the one proposed in this ticket, but after re-reading, I think you probably meant the one currently in use by Jenkins.

So the CI process for PRs would essentially consist of the following steps "quality gates":

1. Ensure a successful 'yarn install' of the module.
2. Run Lint.
3. Run unit (non-integration) tests if they exist e,g 'yarn test'
4. Build stripes platform (with configurable platform to use) with the code under review (PR only)
5. Bootstrap test tenant (PR only)
6. Run appropriate module tests (and maybe cross-module tests) in platform context. e.g. 'yarn int-test etc...' (PR only)

This is roughly the same process used today. Just some of the mechanics will be changed.

To me this poses a similar issue with the workspace approach in that the test procedure is not self-contained to the repo and readily duplicated across environments due to coupling with an external platform/repo. Could we address your concerns by breaking it apart and running the app's "test-int" followed by the platform's "test-int"?

there is still the issue of the ensuring there is a backend tenant that was been bootstrapped with the proper modules enabled.

Either way, we need to prep a backend tenant at least once. Can this tenant be reused across separate tests? I'll explain below...

we have identified many legitimate issues with code changes when building that code into a platform bundle

I agree, we should still run these tests. I'm proposing we consider the platform to be its own independently testable job/step/gate. Doing so may not be as efficient. There is a second bundle built, but hopefully the more expensive backend prep can be retained.

Starting with the app:

1. Ensure a successful 'yarn install' of the module.
2. Run Lint.
3. Run unit (non-integration) tests if they exist e,g 'yarn test'
4. Bootstrap test tenant (PR only) ...and retain it?
5. Run `yarn test-int`

At this point success would promote/publish the app and kick off a platform build:

1. Ensure a successful 'yarn install' of the platform.
2. Run Lint.
3. Run unit (non-integration) tests if they exist e,g 'yarn test'
4. Bootstrap test tenant (PR only) ...can this work be retained from prior app's PR?
5. Run `yarn test-int`

Notice these describe essentially the same routine for both apps and platforms. In addition, the above platform routine can also be triggered independently when the platform itself changes, for example, when we add apps to the platform or modify platform's own tests.

Likewise, the routine for an app is the same whether the app is part of a platform or not. This addresses your point about needing to run integration tests for modules that are not yet part of a platform. I believe achieving a level of consistency like this has some merit and will simplify understanding and reuse of tests in dev environments.

I realize the piece missing (for now) is that an app that could break a cross-module test gets promoted/published. We would, however, catch it shortly thereafter when the platform runs. Is there a way for the Nexus repo to more-or-less flag an artifact as private only for Jenkins? The idea being, we make it "public" once the next quality gate (the platform) has been passed.

I suppose in some respects, npm-folioci itself is Jenkin's own registry and this seems like a good example of something we want Jenkins to see but no one else. When we are more frequently publishing to npm-folio (or public npm), folks shouldn't have to directly depend on npm-folioci like they do today and perhaps it could serve as such.

Comment by John Malconian [ 13/Aug/18 ]

Matthew Jones After some thought last week, I think your approach above has considerable merit. I would like to take it one step further and remove platform testing with testing a single repo. I've broken it down like this with '"stages" representing the various quality gates:

Stage 1:

  • Linting, code quality, unit tests, and isolated (module) integration tests. The emphasis here is on the individual repo or project - not "the platform" and, with limited integration tests, we still can test things like module dependency resolution, permission checking, and so forth at this stage. PRs will use Stage 1 as the only quality gate. When these tests pass, an artifact is uploaded to the CI repository.

Stage 2:

  • Platform testing. Test the various platforms as a whole which consist of artifacts from the CI repository. This essentially becomes the basis for 'folio-testing'. Somewhat different than what folio-testing is today. Full suite integration tests and other automated integration tests that require a full environment are done at this stage. When these tests pass, a yarn.lock and a module list is committed to the platform repository.

Stage 3:

  • User acceptance/release candidate testing. A 'folio-snapshot' environment is built from platforms that have passed Stage 2.

Stage 4:

  • Individual modules are released to a production repository. Platforms are possibly tagged. Perhaps quarterly, at least initially.

That's sort of the simplified version of the CI/CD process as I envision it today. There is some details that needs to be examined in more depth. For instance, I'm thinking that the way we version UI components in the CI needs to change. It doesn't make any sense to version a component in testing as v1.2.3000123 and then release v1.3.0. It would make more sense for a developer to version the component in Github as something like 1.3.0-SNAPSHOT before development for that sprint or feature branch begins - similar to the process the backend Maven developers use - The CI can then iteratively append build numbers to that i.e 1.3.0-SNAPSHOT.1 until the release is actually made for 1.3.0 or whatever. I know that there are some limitations to how NPM dependency resolution occurs with snapshot releases, but they may work in our favor with this model - TBD.

I know I may be expanding the scope of the original issue, however, but I think it's worthwhile to re-evaluate everything we are doing at a higher level at this point. Curious specifically about your thoughts on versioning.

Comment by Matthew Jones [ 14/Aug/18 ]

John Malconian This looks like a good approach. I agree that its worthwhile revisiting the process as a whole.

With that in mind, I see stage one/two for apps/platforms drawing a parallel with how we might want to handle and publish the stripes-*/stripes-framework grouping of STRIPES-543 Closed .

I am very much in favor of using Semver's pre-release nomenclature for "1.3.0-SNAPSHOT". It seems the trouble in the past has been with pulling the latest version of a module when that module is pulled in as a dependency elsewhere. However, this should be minimized with the introduction of the stripes framework.

Comment by Matthew Jones [ 15/Aug/18 ]

John Malconian, Interesting development with running tests against ui-users:

When I run the following using a local folio/testing-backend vagrant box:

$ yarn test-int --show

I get 6 failing tests. Some of these are due to missing permissions and an endpoint. Presumably this vagrant box (installed today) doesn't have them. I verified my vagrant box was in fact out of date.

Access requires permission: users.item.get
Access requires permission: users.collection.get
No suitable module found for path /service-points-users

Next, I ran the tests against the folio snapshot-stable UI (and Okapi):

$ yarn test-int --show --url http://folio-snapshot-stable.aws.indexdata.com/

All 32 tests pass. Snapshot-stable is a platform build though.

Taking the middle road, using my local UI build of ui-users (in the app context) and pointing to the snapshot stable Okapi:

$ yarn test-int --show --okapi http://folio-snapshot-331.aws.indexdata.com:9130

Only 2 tests fail in new_proxy:

1) should add a proxy for user 1
2) should delete a sponsor of user 2

Looking into the source of these failures reveals that these new_proxy tests depend on ui-plugin-find-user which is not available when testing ui-users in isolation. In effect, this test is "cross-module" and therefore shouldn't exist inside of ui-users.

Comment by John Malconian [ 15/Aug/18 ]

Here are the results of running 'test-int' in ui-users branch FOLIO-1344 Closed (with latest updates from master merged). They are consistent with multiple runs.

$ stripes test nightmare --uiTest.username pr_463_10_admin --uiTest.password admin --show --okapi http://folio-snapshot-latest.aws.indexdata.com:9130 --tenant pr_463_10
Waiting for webpack to build...
Listening at http://localhost:3000
webpack built bf4c4323b43d7d2e0437 in 14604ms
⚠ 「wdm」:    2157 modules

WARNING in domelementtype
  Multiple versions of domelementtype found:
    1.1.3 ./~/dom-serializer/~/domelementtype
    1.3.0 ./~/domelementtype


WARNING in isarray
  Multiple versions of isarray found:
    0.0.1 ./~/path-to-regexp/~/isarray
    1.0.0 ./~/isarray


WARNING in warning
  Multiple versions of warning found:
    3.0.0 ./~/warning
    4.0.1 ./~/react-router-dom/~/warning

Check how you can resolve duplicate packages: 
https://github.com/darrenscerri/duplicate-package-checker-webpack-plugin#resolving-duplicate-packages-in-your-bundle

Starting Nightmare tests...
Host:          http://localhost:3000
Username:      pr_463_10_admin


  Module test: users:patron_group
    Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

      ✓ should login as pr_463_10_admin/admin (6861ms)
          Test suite   @folio/users:2.12.3
          Live module  @folio/users:2.12.3 (http://localhost:3000)
      ✓ should open app and find version tag (594ms)
      ✓ should create a patron group for "Alumni" (8541ms)
      1) should find an active user to edit
      2) should find patron group ID for "alumni_363"
      3) should edit user record using "alumni_363" group
      4) should find ID for "Staff" group
      5) should change patron group to "Staff" in user record
      ✓ should delete "alumni_363" patron group (5516ms)
      ✓ should confirm that "alumni_363" patron group has been deleted (1685ms)
      ✓ should logout (108ms)

  Module test: users:new_user
    Login > Create new user > Logout > Login as new user > Logout > Login > Edit new user and confirm changes
      ✓ should load login page (5353ms)
      ✓ should login as pr_463_10_admin/admin (1590ms)
          Test suite   @folio/users:2.12.3
          Live module  @folio/users:2.12.3 (http://localhost:3000)
      ✓ should open app and find version tag (745ms)
      ✓ should extract a patron group value (5473ms)
      ✓ should create a user: ijohnson1534362535422/ijohnson1534362535422 (5572ms)
      ✓ should logout (759ms)
      ✓ should login as ijohnson1534362535422/ijohnson1534362535422 (299ms)
      ✓ should logout (592ms)
      ✓ should login as pr_463_10_admin/admin (2329ms)
      ✓ should change username for ijohnson1534362535422 (7205ms)
      ✓ should logout (623ms)
      ✓ Should login as ijohnson1534362535422x/ijohnson1534362535422 (1207ms)

  Module test: users:new_permission_set
    Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

          Test suite   @folio/users:2.12.3
          Live module  @folio/users:2.12.3 (http://localhost:3000)
      ✓ should open app and find version tag (658ms)
      6) should create a new permission set
      7) should confirm creation of new permission set
      8) should delete new permission set
      ✓ should confirm deletion (596ms)

  Module test: users:new_proxy
    Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

          Test suite   @folio/users:2.12.3
          Live module  @folio/users:2.12.3 (http://localhost:3000)
      ✓ should open app and find version tag (738ms)
      9) should get active user barcodes
      10) should add a proxy for user 1
      11) should delete a sponsor of user 2


  21 passing (3m)
  11 failing

  1) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should find an active user to edit:
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  2) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should find patron group ID for "alumni_363":
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  3) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should edit user record using "alumni_363" group:
     Error: done() invoked with non-Error: Cannot set property 'value' of null
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  4) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should find ID for "Staff" group:
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  5) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should change patron group to "Staff" in user record:
     Error: done() invoked with non-Error: Cannot set property 'value' of null
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  6) Module test: users:new_permission_set
       Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

         should create a new permission set:
     Error: done() invoked with non-Error: Unable to find element by selector: //button[contains(.,"Check in")]
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  7) Module test: users:new_permission_set
       Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

         should confirm creation of new permission set:
     Error: done() invoked with non-Error: Unable to find element by selector: //a[.="Circulation employee"]
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  8) Module test: users:new_permission_set
       Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

         should delete new permission set:
     Error: done() invoked with non-Error: Unable to find element by selector: #clickable-edit-item
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  9) Module test: users:new_proxy
       Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

         should get active user barcodes:
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  10) Module test: users:new_proxy
       Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

         should add a proxy for user 1:
     TypeError: Cannot read property 'barcode' of undefined
      at Context.it (test/ui-testing/new_proxy.js:66:52)

  11) Module test: users:new_proxy
       Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

         should delete a sponsor of user 2:
     TypeError: Cannot read property 'barcode' of undefined
      at Context.it (test/ui-testing/new_proxy.js:97:50)



Some tests failed or something went wrong while attempting to run the tests.
11
exiting process
Done in 221.07s.

The tenant is 'pr_463_10' on folio-snapshot-stable (8/15). I posted the module descriptor for ui-users to okapi and enabled the module (folio_users-2.12.3-pr.463.10) along with the latest version of stripes-core on folio-snapshot-stable and mod-codex-inventory (this gets enabled always since nothing depends on it). Using Okapi's dependency resolution, the following modules were enabled for the tenant:

jenkins@8e6da5a15783:~/ui-users$ curl -w '\n' http://folio-snapshot-stable.aws.indexdata.com:9130/_/proxy/tenants/pr_463_10/modules
[

{ "id" : "folio_stripes-core-2.10.5000325" }

,

{ "id" : "folio_users-2.12.3-pr.463.10" }

,

{ "id" : "mod-authtoken-1.5.1-SNAPSHOT.25" }

,

{ "id" : "mod-circulation-11.0.1-SNAPSHOT.165" }

,

{ "id" : "mod-circulation-storage-5.6.0-SNAPSHOT.104" }

,

{ "id" : "mod-codex-inventory-1.2.0-SNAPSHOT.54" }

,

{ "id" : "mod-configuration-4.0.4-SNAPSHOT.37" }

,

{ "id" : "mod-feesfines-14.2.2-SNAPSHOT.10" }

,

{ "id" : "mod-inventory-storage-12.5.1-SNAPSHOT.144" }

,

{ "id" : "mod-login-4.2.1-SNAPSHOT.17" }

,

{ "id" : "mod-permissions-5.3.2-SNAPSHOT.27" }

,

{ "id" : "mod-users-15.1.0-SNAPSHOT.38" }

,

{ "id" : "mod-users-bl-3.0.0-SNAPSHOT.20" }

,

{ "id" : "okapi-2.17.0" }

]

I suspect the absence of 'ui-plugin-find-user' is responsible for at least some of these. I will enable that module next and re-test.

Comment by John Malconian [ 15/Aug/18 ]

Oddly enough, I get the same 11 test failures when the find-user plugin is enabled for the tenant:

[ {
  "id" : "folio_plugin-find-user-1.1.100024"
}, {
  "id" : "folio_stripes-core-2.10.5000325"
}, {
  "id" : "folio_users-2.12.3-pr.463.10"
}, {
  "id" : "mod-authtoken-1.5.1-SNAPSHOT.25"
}, {
  "id" : "mod-circulation-11.0.1-SNAPSHOT.165"
}, {
  "id" : "mod-circulation-storage-5.6.0-SNAPSHOT.104"
}, {
  "id" : "mod-codex-inventory-1.2.0-SNAPSHOT.54"
}, {
  "id" : "mod-configuration-4.0.4-SNAPSHOT.37"
}, {
  "id" : "mod-feesfines-14.2.2-SNAPSHOT.10"
}, {
  "id" : "mod-inventory-storage-12.5.1-SNAPSHOT.144"
}, {
  "id" : "mod-login-4.2.1-SNAPSHOT.17"
}, {
  "id" : "mod-permissions-5.3.2-SNAPSHOT.27"
}, {
  "id" : "mod-users-15.1.0-SNAPSHOT.38"
}, {
  "id" : "mod-users-bl-3.0.0-SNAPSHOT.20"
}, {
  "id" : "okapi-2.17.0"
} ]
jenkins@8e6da5a15783:~/ui-users$ xvfb-run '--server-args=-screen 0 1024x768x24' yarn test-int --uiTest.username pr_463_10_admin --uiTest.password admin --show --okapi http://folio-snapshot-latest.aws.indexdata.com:9130 --tenant pr_463_10
yarn run v1.9.4
$ stripes test nightmare --uiTest.username pr_463_10_admin --uiTest.password admin --show --okapi http://folio-snapshot-latest.aws.indexdata.com:9130 --tenant pr_463_10
Waiting for webpack to build...
Listening at http://localhost:3000
webpack built 2279adcf80465538f6f9 in 14594ms
⚠ 「wdm」:    2157 modules

WARNING in domelementtype
  Multiple versions of domelementtype found:
    1.1.3 ./~/dom-serializer/~/domelementtype
    1.3.0 ./~/domelementtype


WARNING in isarray
  Multiple versions of isarray found:
    0.0.1 ./~/path-to-regexp/~/isarray
    1.0.0 ./~/isarray


WARNING in warning
  Multiple versions of warning found:
    3.0.0 ./~/warning
    4.0.1 ./~/react-router-dom/~/warning

Check how you can resolve duplicate packages: 
https://github.com/darrenscerri/duplicate-package-checker-webpack-plugin#resolving-duplicate-packages-in-your-bundle

Starting Nightmare tests...
Host:          http://localhost:3000
Username:      pr_463_10_admin


  Module test: users:patron_group
    Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

      ✓ should login as pr_463_10_admin/admin (7576ms)
          Test suite   @folio/users:2.12.3-malc
          Live module  @folio/users:2.12.3-malc (http://localhost:3000)
      ✓ should open app and find version tag (548ms)
      ✓ should create a patron group for "Alumni" (8592ms)
      1) should find an active user to edit
      2) should find patron group ID for "alumni_4231"
      3) should edit user record using "alumni_4231" group
      4) should find ID for "Staff" group
      5) should change patron group to "Staff" in user record
      ✓ should delete "alumni_4231" patron group (5490ms)
      ✓ should confirm that "alumni_4231" patron group has been deleted (1677ms)
      ✓ should logout (107ms)

  Module test: users:new_user
    Login > Create new user > Logout > Login as new user > Logout > Login > Edit new user and confirm changes
      ✓ should load login page (5165ms)
      ✓ should login as pr_463_10_admin/admin (3235ms)
          Test suite   @folio/users:2.12.3-malc
          Live module  @folio/users:2.12.3-malc (http://localhost:3000)
      ✓ should open app and find version tag (547ms)
      ✓ should extract a patron group value (5373ms)
      ✓ should create a user: nbrown1534364484173/nbrown1534364484173 (5331ms)
      ✓ should logout (733ms)
      ✓ should login as nbrown1534364484173/nbrown1534364484173 (302ms)
      ✓ should logout (593ms)
      ✓ should login as pr_463_10_admin/admin (10257ms)
      ✓ should change username for nbrown1534364484173 (6566ms)
      ✓ should logout (622ms)
      ✓ Should login as nbrown1534364484173x/nbrown1534364484173 (1192ms)

  Module test: users:new_permission_set
    Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

          Test suite   @folio/users:2.12.3-malc
          Live module  @folio/users:2.12.3-malc (http://localhost:3000)
      ✓ should open app and find version tag (608ms)
      6) should create a new permission set
      7) should confirm creation of new permission set
      8) should delete new permission set
      ✓ should confirm deletion (601ms)

  Module test: users:new_proxy
    Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

          Test suite   @folio/users:2.12.3-malc
          Live module  @folio/users:2.12.3-malc (http://localhost:3000)
      ✓ should open app and find version tag (639ms)
      9) should get active user barcodes
      10) should add a proxy for user 1
      11) should delete a sponsor of user 2


  21 passing (3m)
  11 failing

  1) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should find an active user to edit:
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  2) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should find patron group ID for "alumni_4231":
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  3) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should edit user record using "alumni_4231" group:
     Error: done() invoked with non-Error: Cannot set property 'value' of null
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  4) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should find ID for "Staff" group:
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  5) Module test: users:patron_group
       Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout

         should change patron group to "Staff" in user record:
     Error: done() invoked with non-Error: Cannot set property 'value' of null
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  6) Module test: users:new_permission_set
       Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

         should create a new permission set:
     Error: done() invoked with non-Error: Unable to find element by selector: //button[contains(.,"Check in")]
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  7) Module test: users:new_permission_set
       Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

         should confirm creation of new permission set:
     Error: done() invoked with non-Error: Unable to find element by selector: //a[.="Circulation employee"]
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  8) Module test: users:new_permission_set
       Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout

         should delete new permission set:
     Error: done() invoked with non-Error: Unable to find element by selector: #clickable-edit-item
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

  9) Module test: users:new_proxy
       Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

         should get active user barcodes:
     Error: .wait() timed out after 30000msec
      at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)

  10) Module test: users:new_proxy
       Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

         should add a proxy for user 1:
     TypeError: Cannot read property 'barcode' of undefined
      at Context.it (test/ui-testing/new_proxy.js:66:52)

  11) Module test: users:new_proxy
       Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout

         should delete a sponsor of user 2:
     TypeError: Cannot read property 'barcode' of undefined
      at Context.it (test/ui-testing/new_proxy.js:97:50)



Some tests failed or something went wrong while attempting to run the tests.
11
exiting process
Done in 227.20s.

Will follow up tomorrow by testing against a wider set of tenant modules.

Comment by John Malconian [ 15/Aug/18 ]

Actually, those results should be expected. Enabling the module for the tenant with Okapi means nothing if it is not included in the bundle. Oversights like that mean its time to stop for the day.

Comment by Matthew Jones [ 15/Aug/18 ]

I can confirm the same 11 failures when using tenant pr_463_10. There are only 2 failures (known, for the plugin) when running against tenant diku. This suggest to me the initial tenant data does not meet the test's expectations.

The first failure occurs while looking for an active user. When not found, this has cascading effects on subsequent tests operating on that user. While there are active users in tenant pr_463_10, the test is looking for the 11th user in the list. However, this tenant only has 3 users to choose from. (Note: There are now more users in this tenant since I've run tests a couple of times, one of which creates a user.)

In patron_group.js:

.wait('#list-users div[role="listitem"]:nth-of-type(11) > a > div:nth-of-type(5)')
.evaluate(() => document.querySelector('#list-users div[role="listitem"]:nth-of-type(11) > a > div:nth-of-type(5)').title)

Changing both of these :nth-of-type() selectors locally from "11" to say "3" allows more tests to pass, but even selecting the 3rd user seem arbitrary. The test probably should be updated to select the first it finds, or even better, create n users before looking for then nth one.

Correctly finding a user will allow all these failing tests to pass:

      1) should find an active user to edit
      2) should find patron group ID for "alumni_1757"
      3) should edit user record using "alumni_1757" group
      4) should find ID for "Staff" group
      5) should change patron group to "Staff" in user record
Comment by Matthew Jones [ 15/Aug/18 ]

The next failure is also due to lack of data. The new_permission_set test is looking for existing permissions to assign to the permission set it creates.

.xclick('//button[contains(.,"Check in")]')
//...
.xclick('//button[contains(.,"Check out")]')

Tenant diku has the permissions "Check in: All permissions" and "Check out: All permissions", however, tenant pr_463_10 does not have these. It only has permissions related to users and settings, which makes sense given that's what we are dealing with here.

This test should be updated to select permissions pertaining only to the user's app to remove this cross-module data dependency, allowing the test to remain isolated.

This fix will allow these failing tests to pass:

      6) should create a new permission set
      7) should confirm creation of new permission set
      8) should delete new permission set
Comment by Matthew Jones [ 15/Aug/18 ]

The last failure (outside of the plugin ones) is also due to too few users. The new_proxy test is looking for the 9th user in the list when there aren't that many to choose from.

This is like the earlier :nth-of-type, but using the :nth-child selector:

.wait('#list-users div[role="listitem"]:nth-child(9)')

By correctly selecting a user from the list, this test will pass:

9) should get active user barcodes
Comment by Matthew Jones [ 15/Aug/18 ]

In summary, we have three ui-users tests that need updating due to data selections and one cross-module test that needs to be moved to platform-core (but probably not until we are ready to make the switch in CI). I will create appropriate UIU tickets for these in the morning.

  • patron_group (update)
  • new_permission_set (update)
  • new_proxy (update)
  • new_proxy (move)

Unfortunately I suspect testing other apps in isolation could yield similar findings. While it is more work now, decoupling these data and cross-module dependencies should have benefits in the longer term.

Also, while debugging these tests, it seems pretty clear we could benefit from an abstraction layer (or what we're calling an interactor in bigtest) over our search results. Hunting for the nth div of the nth list item to grab a barcode, for example, seems pretty fragile.

Comment by Matthew Jones [ 17/Aug/18 ]

I created UIU-604 Closed and UIU-605 Closed to address the test dependencies that cause failures when testing ui-users in isolation with the PR tenant.

Comment by John Malconian [ 27/Sep/18 ]

stripes-cli is now a devDependency of folio-testing-platform,platform-core,and plaform-complete. It is used to build webpacks via 'yarn build' and used to invoke local tests via 'yarn test' in Stripes and UI modules where either local unit tests or integration tests have been implemented. It is also used to invoke cross-platform and module tests in platform-core and platform-complete via 'yarn test-regression' and 'yarn test-int'.

Generated at Thu Feb 08 23:12:39 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.