[FOLIO-1344] use stripes-cli in the build and test process Created: 13/Jul/18 Updated: 12/Nov/18 Resolved: 27/Sep/18 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | Task | Priority: | P2 |
| Reporter: | Matthew Jones | Assignee: | John Malconian |
| Resolution: | Done | Votes: | 0 |
| Labels: | build-release, ci, sprint44, sprint45, sprint46, sprint47 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Document the procedure and logic for running tests via a platform rather than via a workspace. The goal is for a developer to run tests against a local environment that most closely resembles that of CI and eliminate dependency discrepancies. Update existing documentation, or create new if needed. Also identify automation enhancements to the CLI that would make the testing process more efficient for developers. Publish the documentation on the dev.folio.org |
| Comments |
| Comment by Matthew Jones [ 19/Jul/18 ] |
|
Given the current issues around testing, I propose modifying this to document a new CLI-based test procedure for both CI and dev. It seems like the right opportunity to bring together a few things on the horizon with platform-core, ui-testing (framework-only), and likely workspaces. By having the CLI encompass as many steps as possible we can help minimize possible discrepancies. |
| Comment by Matthew Jones [ 19/Jul/18 ] |
|
At first I thought much of this could be accomplished with existing commands, and some of it can. However, during initial testing I ran into the following issues trying to overlap the workspace and CI approach with the CLI:
|
| Comment by Matthew Jones [ 03/Aug/18 ] |
|
Update: In drafting a proposed steps for running platform tests with Stripes-CLI, I couldn't help but draw similarities to that of running tests for an individual app. It makes sense that the procedure for each be as close as possible. Therefore I am drafting one set of steps to work in both cases. Two branches are available for review:
Both have a package script named test-int for integration tests. I have reserved test for when unit tests are present. Please see the build/test sections of each readme.md. Prerequisites: Each requires a running Okapi instance with a valid tenant. Option 1: Build and test in one step (app or platform)This is the quickest to just run tests. $ yarn test-int Option 2: Build and test in two steps (app or platform)Quickest for developers that already have a dev instance running. $ yarn start $ yarn test-int --local Option 3: Build and tests in separate steps (app or platform)This option performs production builds and also retains build artifacts. $ yarn build ./output $ yarn stripes serve --existing-build ./output $ yarn test-int --local Regression tests (platform only)This will run both the platform's own cross-module tests as well as each of the apps sets of tests. Replace test-int with test-regression in the options above. $ yarn test-regression --local Notes:Stripes-CLI is a devDependency of the app and platform and therefore does not need to be globally installed. For any of the build-related options above, an Okapi URL and tenant ID can be provided as --okapi and --tenant options. Alternatively, in the case of a platform, stripes.config.js can also be updated with an Okapi and tenant (command line option takes precedence). The --local option is simply an alias for "--url http://localhost:3000" or (whatever host/port is defined to run the local dev server). When either --local or --url is present, the development build and serve steps are skipped. Additional ui-testing options can be passed via --uiTest.optionName, such as setting username. These can also be set via environment variables that ui-testing knows about. I wonder if for consistency, it makes sense to pass everything on the command line. $ yarn test-int --uiTest.username admin One thing missing from this approach is running a platform's cross-module tests with an app build (say for the PR process). To keep things self-contained, I am proposing that instead we run an app's own unit tests via test first (when applicable) and then the app's own integration tests in isolation via test-int. For an app in isolation, build artifacts may not be necessary to maintain, meaning something like option 1 may be sufficient. I am still looking into how best to handle the ui-app PR scenario w/cross module platform tests. I think that sort of mixed case may be worth running as a subsequent quality gate following the above app-in-isolation approach. The advantage to the app in isolation approach as the first quality gate is simplicity and consistency in setup between dev and CI environments. |
| Comment by Matthew Jones [ 03/Aug/18 ] |
|
John Malconian, when you have time, I'm interested in your thoughts on the options described above. The following branches should be in a suitable state for experimenting with: https://github.com/folio-org/ui-users/tree/FOLIO-1344 FYI, ui-users branch has a .stripesclirc.json file defining the default okapi and tenant settings (since users, by nature doesn't have stripes.config.js). This will take precedence over another .stripesclirc, if you have one, maintained in a higher level directory. The .stripesclirc in general, may be a good way to consolidate many options together, rather than having some on the command line, some in environment variables, etc. Platform-core tests by now may have drifted from adjustments made to the original tests in ui-testing. Therefore I would not expect all to pass is-is. We can sync them up later once the procedure is looking good. |
| Comment by John Malconian [ 06/Aug/18 ] |
|
Matthew Jones - Haven't had an opportunity to examine your test branches yet, however, I like the whole approach you've described above in the comments and don't for see any significant problems in terms of running in the CI. I'm a little confused by the presence of the .stripesclirc.json in the project directory. I'm assuming that the default settings will be geared toward developers and that the CI might inject its own .stripesclirc.json in place of the one that might exist in GitHub? |
| Comment by Matthew Jones [ 06/Aug/18 ] |
|
In conversations, I have found the names "test-int" (integration) and "test-regression" to cause some confusion. These both invoke integration/end-to-end tests that require a live back-end. The main difference is that the former is a subset of the latter. For example, an app's "test-int" tests are also run during a platform's "test-regression". We may want to consider alternatives names for clarity. |
| Comment by Matthew Jones [ 06/Aug/18 ] |
|
John Malconian Yeah the .stripesclirc in the repo was for developers. Platforms provide such defaults via stripes.config.js, but there is no equivalent in the app-context. However, I'm not too pleased about including a .stripesclirc in the repo as it could cause interference with local CLI preferences, etc. I think what I will do is include these two defaults in the CLI itself. Things like "localhost:9130" and "diku" are a large part of the sample documentation and vagrant boxes, so this may help in other aspects as well. I also thought about separate .stripesclirc files CI. This might actually be a handy artifact of a Jenkins build, particularly if the .stripesclirc contained all the testing parameters used. The idea being a developer could just copy the CI's .stripesclirc locally and run the tests with all the same values for a PR without having to set environment variables, etc. |
| Comment by John Malconian [ 06/Aug/18 ] |
|
Just for my own clarification, it appears 'yarn test-int' without 'run' option will run only cross-module tests. Use the 'run' option to run specific app/module-specific tests only in a platform context. |
| Comment by Matthew Jones [ 06/Aug/18 ] |
|
Yes. The goal with "test-int" was to run the integration tests that come included with a given repo, be it an app or platform. If you want to run tests for a specific app (from within a platform) you can run "test-regression" for all of them, or pass just one app with --run. --run users However, if you're looking to just run tests for a given app, I would suggest running "test-int" from a clone of the app in question. At least that's the idea to simplify the test process. |
| Comment by Matthew Jones [ 06/Aug/18 ] |
|
Not sure what I was thinking... The CLI already does this! I've removed the .stripesclirc file. |
| Comment by Matthew Jones [ 06/Aug/18 ] |
|
Also, regarding app and cross-module tests: If you decide you want to run both cross-module and an app's tests from a platform at the same time you can do this: yarn test-int --run WD/users The "WD" is a token representing the working directory. In the case of platform-core, this says to run the (cross module) tests in the current platform, plus the tests for users. |
| Comment by John Malconian [ 06/Aug/18 ] |
ok. This makes sense. In this case I suspect I would want to run 'yarn start' aka 'stripes serve' and then 'stripes serve --okapi http://folio-snapshot-latest.aws.indexdata.com --tenant foobar' 'yarn test-int --ui.Test.username diku --ui.Test.password admin --ui.Test.url http://folio-snapshot-latest.aws.indexdata.com --url http://folio-snapshot-latest.aws.indexdata.com' I get the following output: yarn run v1.6.0
$ stripes test nightmare --ui.Test.username diku --ui.Test.password admin --ui.Test.url http://folio-snapshot-latest.aws.indexdata.com --url http://folio-snapshot-latest.aws.indexdata.com
Using URL http://folio-snapshot-latest.aws.indexdata.com
Starting Nightmare tests...
Host: http://folio-snapshot-latest.aws.indexdata.com
Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
✓ should login as diku_admin/admin (7954ms)
Test suite @folio/users:2.12.3
Live module @folio/users:2.12.3000375 (http://folio-snapshot-latest.aws.indexdata.com)
I wouldn't expect 'diku_admin' to be able to successfully login because I specified 'foobar' as the tenant. Does 'stripes serve' honor the tenant config? |
| Comment by John Malconian [ 06/Aug/18 ] |
|
The other issue here from a CI perspective is not a deal breaker, but it's difficult for the CI to determine when to execute the next command after executing 'stripes serve' since it has to wait until the webpack build has been completed when run in daemon mode. That's why I've favored separate steps in the CI for building the bundle and then serving it up, but I don't think 'stripes build' works in an app context. |
| Comment by Matthew Jones [ 06/Aug/18 ] |
If you want to build, serve, and test locally, "yarn test-int" will do this in one go. Only when you provide a "--url" option, will it bypass the build/serve steps, pointing to the build at the "--url" instead. In fact, I think this is why your foobar tenant is ignored. Your yarn test-int line above includes a url, so your local build is not being used: --url http://folio-snapshot-latest.aws.indexdata.com --ui.Test.url http://folio-snapshot-latest.aws.indexdata.com Try this command instead: yarn test-int --okapi http://folio-snapshot-320.aws.indexdata.com:9130 --tenant foobar --ui.Test.username diku --ui.Test.password admin The underlying "stripes test nightmare" command has the same okapi and tenant options available for building a bundle as "stripes serve". |
| Comment by Matthew Jones [ 06/Aug/18 ] |
Option 1 should take care of this, invoking the tests only after the build is finished. The catch with options 1 and 2 is that these both yield development builds (for now). Eventually we should be able to toggle prod settings on/off during both "stripes build" and "stripes serve" (
|
| Comment by Jakub Skoczen [ 08/Aug/18 ] |
|
John Malconian will implement the changes proposed in the CI and close this when everything works ok. |
| Comment by John Malconian [ 08/Aug/18 ] |
However, if you're looking to just run tests for a given app, I would suggest running "test-int" from a clone of the app in question. At least that's the idea to simplify the test process. Some thoughts around this...I initially was thinking that would be "cheaper" and maybe more efficient to run the app-specific nightmare tests in the app context instead of the platform context in CI for PRs, however, there is still the issue of the ensuring there is a backend tenant that was been bootstrapped with the proper modules enabled. Plus we have identified many legitimate issues with code changes when building that code into a platform bundle. So if building the platform with the code that is under review acts a worthwhile quality gate in CI (which I believe it to be) then I'm thinking that all builds and tests in CI should be platform-based as it is currently. This will also give us the ability to run the cross-module tests in CI as well without extra setup. So the CI process for PRs would essentially consist of the following steps "quality gates": 1. Ensure a successful 'yarn install' of the module. This is roughly the same process used today. Just some of the mechanics will be changed. |
| Comment by John Malconian [ 08/Aug/18 ] |
|
Additional thought regarding Step 4 above. If a specific platform has NOT been configured for the module in the project's Jenkinsfile, then the existing build steps in CI include building a bundle in the app context - 'PREFIX=/usr/local/share/.config/yarn stripes build --output=./bundle'. However, no further integration tests are run in this case currently. We should probably continue this model just to ensure that a bundle can be built successfully even if it is not a platform build. We could potentially take it a step further and generate a module descriptor via 'stripes mod descriptor --strict --full' and post it to okapi, bootstrap the backend tenant and run the module integration tests. Essentially, this gives us the option to run module integration tests for modules that are not yet part of a platform as well as those that are although I think it is preferable from a quality perspective to test as part of a platform. |
| Comment by John Malconian [ 09/Aug/18 ] |
|
Matthew Jones Any additional thoughts on utilizing workspaces in the CI? Or is the current approach of just cloning projects and platforms sufficient? |
| Comment by Matthew Jones [ 09/Aug/18 ] |
|
I think workspaces could be helpful for an added level of testing. However, they don't lend themselves to a simple self-contained "clone > install > test" workflow. Its not that workspace steps would be hard, its just that once we introduce a workspace, we are no longer self-contained to a single repo. Establishing a consistent baseline workflow (for both apps and platforms) that is self-contained gives us a good foundation that is the most portable across dev and CI. In effect we should be able to get the same results from testing independent app and platform builds as we would in a workspace (or the link/alias/file methods). Therefore I think we should prove out the simpler baseline first. Then we can look to extend/optimize via Workspace. |
| Comment by Matthew Jones [ 09/Aug/18 ] |
|
John Malconian, My apologies I completely overlooked your prior comments from yesterday when writing my reply to your question this morning. I took "current approach" to refer to the one proposed in this ticket, but after re-reading, I think you probably meant the one currently in use by Jenkins.
To me this poses a similar issue with the workspace approach in that the test procedure is not self-contained to the repo and readily duplicated across environments due to coupling with an external platform/repo. Could we address your concerns by breaking it apart and running the app's "test-int" followed by the platform's "test-int"?
Either way, we need to prep a backend tenant at least once. Can this tenant be reused across separate tests? I'll explain below...
I agree, we should still run these tests. I'm proposing we consider the platform to be its own independently testable job/step/gate. Doing so may not be as efficient. There is a second bundle built, but hopefully the more expensive backend prep can be retained. Starting with the app: 1. Ensure a successful 'yarn install' of the module. At this point success would promote/publish the app and kick off a platform build: 1. Ensure a successful 'yarn install' of the platform. Notice these describe essentially the same routine for both apps and platforms. In addition, the above platform routine can also be triggered independently when the platform itself changes, for example, when we add apps to the platform or modify platform's own tests. Likewise, the routine for an app is the same whether the app is part of a platform or not. This addresses your point about needing to run integration tests for modules that are not yet part of a platform. I believe achieving a level of consistency like this has some merit and will simplify understanding and reuse of tests in dev environments. I realize the piece missing (for now) is that an app that could break a cross-module test gets promoted/published. We would, however, catch it shortly thereafter when the platform runs. Is there a way for the Nexus repo to more-or-less flag an artifact as private only for Jenkins? The idea being, we make it "public" once the next quality gate (the platform) has been passed. I suppose in some respects, npm-folioci itself is Jenkin's own registry and this seems like a good example of something we want Jenkins to see but no one else. When we are more frequently publishing to npm-folio (or public npm), folks shouldn't have to directly depend on npm-folioci like they do today and perhaps it could serve as such. |
| Comment by John Malconian [ 13/Aug/18 ] |
|
Matthew Jones After some thought last week, I think your approach above has considerable merit. I would like to take it one step further and remove platform testing with testing a single repo. I've broken it down like this with '"stages" representing the various quality gates: Stage 1:
Stage 2:
Stage 3:
Stage 4:
That's sort of the simplified version of the CI/CD process as I envision it today. There is some details that needs to be examined in more depth. For instance, I'm thinking that the way we version UI components in the CI needs to change. It doesn't make any sense to version a component in testing as v1.2.3000123 and then release v1.3.0. It would make more sense for a developer to version the component in Github as something like 1.3.0-SNAPSHOT before development for that sprint or feature branch begins - similar to the process the backend Maven developers use - The CI can then iteratively append build numbers to that i.e 1.3.0-SNAPSHOT.1 until the release is actually made for 1.3.0 or whatever. I know that there are some limitations to how NPM dependency resolution occurs with snapshot releases, but they may work in our favor with this model - TBD. I know I may be expanding the scope of the original issue, however, but I think it's worthwhile to re-evaluate everything we are doing at a higher level at this point. Curious specifically about your thoughts on versioning. |
| Comment by Matthew Jones [ 14/Aug/18 ] |
|
John Malconian This looks like a good approach. I agree that its worthwhile revisiting the process as a whole. With that in mind, I see stage one/two for apps/platforms drawing a parallel with how we might want to handle and publish the stripes-*/stripes-framework grouping of
I am very much in favor of using Semver's pre-release nomenclature for "1.3.0-SNAPSHOT". It seems the trouble in the past has been with pulling the latest version of a module when that module is pulled in as a dependency elsewhere. However, this should be minimized with the introduction of the stripes framework. |
| Comment by Matthew Jones [ 15/Aug/18 ] |
|
John Malconian, Interesting development with running tests against ui-users: When I run the following using a local folio/testing-backend vagrant box: $ yarn test-int --show I get 6 failing tests. Some of these are due to missing permissions and an endpoint. Access requires permission: users.item.get Access requires permission: users.collection.get No suitable module found for path /service-points-users Next, I ran the tests against the folio snapshot-stable UI (and Okapi): $ yarn test-int --show --url http://folio-snapshot-stable.aws.indexdata.com/ All 32 tests pass. Snapshot-stable is a platform build though. Taking the middle road, using my local UI build of ui-users (in the app context) and pointing to the snapshot stable Okapi: $ yarn test-int --show --okapi http://folio-snapshot-331.aws.indexdata.com:9130 Only 2 tests fail in new_proxy: 1) should add a proxy for user 1 2) should delete a sponsor of user 2 Looking into the source of these failures reveals that these new_proxy tests depend on ui-plugin-find-user which is not available when testing ui-users in isolation. In effect, this test is "cross-module" and therefore shouldn't exist inside of ui-users. |
| Comment by John Malconian [ 15/Aug/18 ] |
|
Here are the results of running 'test-int' in ui-users branch
$ stripes test nightmare --uiTest.username pr_463_10_admin --uiTest.password admin --show --okapi http://folio-snapshot-latest.aws.indexdata.com:9130 --tenant pr_463_10
Waiting for webpack to build...
Listening at http://localhost:3000
webpack built bf4c4323b43d7d2e0437 in 14604ms
⚠ 「wdm」: 2157 modules
WARNING in domelementtype
Multiple versions of domelementtype found:
1.1.3 ./~/dom-serializer/~/domelementtype
1.3.0 ./~/domelementtype
WARNING in isarray
Multiple versions of isarray found:
0.0.1 ./~/path-to-regexp/~/isarray
1.0.0 ./~/isarray
WARNING in warning
Multiple versions of warning found:
3.0.0 ./~/warning
4.0.1 ./~/react-router-dom/~/warning
Check how you can resolve duplicate packages:
https://github.com/darrenscerri/duplicate-package-checker-webpack-plugin#resolving-duplicate-packages-in-your-bundle
Starting Nightmare tests...
Host: http://localhost:3000
Username: pr_463_10_admin
Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
✓ should login as pr_463_10_admin/admin (6861ms)
Test suite @folio/users:2.12.3
Live module @folio/users:2.12.3 (http://localhost:3000)
✓ should open app and find version tag (594ms)
✓ should create a patron group for "Alumni" (8541ms)
1) should find an active user to edit
2) should find patron group ID for "alumni_363"
3) should edit user record using "alumni_363" group
4) should find ID for "Staff" group
5) should change patron group to "Staff" in user record
✓ should delete "alumni_363" patron group (5516ms)
✓ should confirm that "alumni_363" patron group has been deleted (1685ms)
✓ should logout (108ms)
Module test: users:new_user
Login > Create new user > Logout > Login as new user > Logout > Login > Edit new user and confirm changes
✓ should load login page (5353ms)
✓ should login as pr_463_10_admin/admin (1590ms)
Test suite @folio/users:2.12.3
Live module @folio/users:2.12.3 (http://localhost:3000)
✓ should open app and find version tag (745ms)
✓ should extract a patron group value (5473ms)
✓ should create a user: ijohnson1534362535422/ijohnson1534362535422 (5572ms)
✓ should logout (759ms)
✓ should login as ijohnson1534362535422/ijohnson1534362535422 (299ms)
✓ should logout (592ms)
✓ should login as pr_463_10_admin/admin (2329ms)
✓ should change username for ijohnson1534362535422 (7205ms)
✓ should logout (623ms)
✓ Should login as ijohnson1534362535422x/ijohnson1534362535422 (1207ms)
Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
Test suite @folio/users:2.12.3
Live module @folio/users:2.12.3 (http://localhost:3000)
✓ should open app and find version tag (658ms)
6) should create a new permission set
7) should confirm creation of new permission set
8) should delete new permission set
✓ should confirm deletion (596ms)
Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
Test suite @folio/users:2.12.3
Live module @folio/users:2.12.3 (http://localhost:3000)
✓ should open app and find version tag (738ms)
9) should get active user barcodes
10) should add a proxy for user 1
11) should delete a sponsor of user 2
21 passing (3m)
11 failing
1) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should find an active user to edit:
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
2) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should find patron group ID for "alumni_363":
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
3) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should edit user record using "alumni_363" group:
Error: done() invoked with non-Error: Cannot set property 'value' of null
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
4) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should find ID for "Staff" group:
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
5) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should change patron group to "Staff" in user record:
Error: done() invoked with non-Error: Cannot set property 'value' of null
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
6) Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
should create a new permission set:
Error: done() invoked with non-Error: Unable to find element by selector: //button[contains(.,"Check in")]
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
7) Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
should confirm creation of new permission set:
Error: done() invoked with non-Error: Unable to find element by selector: //a[.="Circulation employee"]
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
8) Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
should delete new permission set:
Error: done() invoked with non-Error: Unable to find element by selector: #clickable-edit-item
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
9) Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
should get active user barcodes:
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
10) Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
should add a proxy for user 1:
TypeError: Cannot read property 'barcode' of undefined
at Context.it (test/ui-testing/new_proxy.js:66:52)
11) Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
should delete a sponsor of user 2:
TypeError: Cannot read property 'barcode' of undefined
at Context.it (test/ui-testing/new_proxy.js:97:50)
Some tests failed or something went wrong while attempting to run the tests.
11
exiting process
Done in 221.07s.
The tenant is 'pr_463_10' on folio-snapshot-stable (8/15). I posted the module descriptor for ui-users to okapi and enabled the module (folio_users-2.12.3-pr.463.10) along with the latest version of stripes-core on folio-snapshot-stable and mod-codex-inventory (this gets enabled always since nothing depends on it). Using Okapi's dependency resolution, the following modules were enabled for the tenant: jenkins@8e6da5a15783:~/ui-users$ curl -w '\n' http://folio-snapshot-stable.aws.indexdata.com:9130/_/proxy/tenants/pr_463_10/modules , { "id" : "folio_users-2.12.3-pr.463.10" }, { "id" : "mod-authtoken-1.5.1-SNAPSHOT.25" }, { "id" : "mod-circulation-11.0.1-SNAPSHOT.165" }, { "id" : "mod-circulation-storage-5.6.0-SNAPSHOT.104" }, { "id" : "mod-codex-inventory-1.2.0-SNAPSHOT.54" }, { "id" : "mod-configuration-4.0.4-SNAPSHOT.37" }, { "id" : "mod-feesfines-14.2.2-SNAPSHOT.10" }, { "id" : "mod-inventory-storage-12.5.1-SNAPSHOT.144" }, { "id" : "mod-login-4.2.1-SNAPSHOT.17" }, { "id" : "mod-permissions-5.3.2-SNAPSHOT.27" }, { "id" : "mod-users-15.1.0-SNAPSHOT.38" }, { "id" : "mod-users-bl-3.0.0-SNAPSHOT.20" }, { "id" : "okapi-2.17.0" }] I suspect the absence of 'ui-plugin-find-user' is responsible for at least some of these. I will enable that module next and re-test. |
| Comment by John Malconian [ 15/Aug/18 ] |
|
Oddly enough, I get the same 11 test failures when the find-user plugin is enabled for the tenant: [ {
"id" : "folio_plugin-find-user-1.1.100024"
}, {
"id" : "folio_stripes-core-2.10.5000325"
}, {
"id" : "folio_users-2.12.3-pr.463.10"
}, {
"id" : "mod-authtoken-1.5.1-SNAPSHOT.25"
}, {
"id" : "mod-circulation-11.0.1-SNAPSHOT.165"
}, {
"id" : "mod-circulation-storage-5.6.0-SNAPSHOT.104"
}, {
"id" : "mod-codex-inventory-1.2.0-SNAPSHOT.54"
}, {
"id" : "mod-configuration-4.0.4-SNAPSHOT.37"
}, {
"id" : "mod-feesfines-14.2.2-SNAPSHOT.10"
}, {
"id" : "mod-inventory-storage-12.5.1-SNAPSHOT.144"
}, {
"id" : "mod-login-4.2.1-SNAPSHOT.17"
}, {
"id" : "mod-permissions-5.3.2-SNAPSHOT.27"
}, {
"id" : "mod-users-15.1.0-SNAPSHOT.38"
}, {
"id" : "mod-users-bl-3.0.0-SNAPSHOT.20"
}, {
"id" : "okapi-2.17.0"
} ]
jenkins@8e6da5a15783:~/ui-users$ xvfb-run '--server-args=-screen 0 1024x768x24' yarn test-int --uiTest.username pr_463_10_admin --uiTest.password admin --show --okapi http://folio-snapshot-latest.aws.indexdata.com:9130 --tenant pr_463_10
yarn run v1.9.4
$ stripes test nightmare --uiTest.username pr_463_10_admin --uiTest.password admin --show --okapi http://folio-snapshot-latest.aws.indexdata.com:9130 --tenant pr_463_10
Waiting for webpack to build...
Listening at http://localhost:3000
webpack built 2279adcf80465538f6f9 in 14594ms
⚠ 「wdm」: 2157 modules
WARNING in domelementtype
Multiple versions of domelementtype found:
1.1.3 ./~/dom-serializer/~/domelementtype
1.3.0 ./~/domelementtype
WARNING in isarray
Multiple versions of isarray found:
0.0.1 ./~/path-to-regexp/~/isarray
1.0.0 ./~/isarray
WARNING in warning
Multiple versions of warning found:
3.0.0 ./~/warning
4.0.1 ./~/react-router-dom/~/warning
Check how you can resolve duplicate packages:
https://github.com/darrenscerri/duplicate-package-checker-webpack-plugin#resolving-duplicate-packages-in-your-bundle
Starting Nightmare tests...
Host: http://localhost:3000
Username: pr_463_10_admin
Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
✓ should login as pr_463_10_admin/admin (7576ms)
Test suite @folio/users:2.12.3-malc
Live module @folio/users:2.12.3-malc (http://localhost:3000)
✓ should open app and find version tag (548ms)
✓ should create a patron group for "Alumni" (8592ms)
1) should find an active user to edit
2) should find patron group ID for "alumni_4231"
3) should edit user record using "alumni_4231" group
4) should find ID for "Staff" group
5) should change patron group to "Staff" in user record
✓ should delete "alumni_4231" patron group (5490ms)
✓ should confirm that "alumni_4231" patron group has been deleted (1677ms)
✓ should logout (107ms)
Module test: users:new_user
Login > Create new user > Logout > Login as new user > Logout > Login > Edit new user and confirm changes
✓ should load login page (5165ms)
✓ should login as pr_463_10_admin/admin (3235ms)
Test suite @folio/users:2.12.3-malc
Live module @folio/users:2.12.3-malc (http://localhost:3000)
✓ should open app and find version tag (547ms)
✓ should extract a patron group value (5373ms)
✓ should create a user: nbrown1534364484173/nbrown1534364484173 (5331ms)
✓ should logout (733ms)
✓ should login as nbrown1534364484173/nbrown1534364484173 (302ms)
✓ should logout (593ms)
✓ should login as pr_463_10_admin/admin (10257ms)
✓ should change username for nbrown1534364484173 (6566ms)
✓ should logout (622ms)
✓ Should login as nbrown1534364484173x/nbrown1534364484173 (1192ms)
Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
Test suite @folio/users:2.12.3-malc
Live module @folio/users:2.12.3-malc (http://localhost:3000)
✓ should open app and find version tag (608ms)
6) should create a new permission set
7) should confirm creation of new permission set
8) should delete new permission set
✓ should confirm deletion (601ms)
Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
Test suite @folio/users:2.12.3-malc
Live module @folio/users:2.12.3-malc (http://localhost:3000)
✓ should open app and find version tag (639ms)
9) should get active user barcodes
10) should add a proxy for user 1
11) should delete a sponsor of user 2
21 passing (3m)
11 failing
1) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should find an active user to edit:
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
2) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should find patron group ID for "alumni_4231":
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
3) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should edit user record using "alumni_4231" group:
Error: done() invoked with non-Error: Cannot set property 'value' of null
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
4) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should find ID for "Staff" group:
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
5) Module test: users:patron_group
Login > Add new patron group > Assign to user > Try to delete patron group > Unassign from user > Try to delete again > Logout
should change patron group to "Staff" in user record:
Error: done() invoked with non-Error: Cannot set property 'value' of null
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
6) Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
should create a new permission set:
Error: done() invoked with non-Error: Unable to find element by selector: //button[contains(.,"Check in")]
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
7) Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
should confirm creation of new permission set:
Error: done() invoked with non-Error: Unable to find element by selector: //a[.="Circulation employee"]
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
8) Module test: users:new_permission_set
Login > Create new permission set > Confirm creation > Delete permission set > Confirm deletion > Logout
should delete new permission set:
Error: done() invoked with non-Error: Unable to find element by selector: #clickable-edit-item
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
9) Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
should get active user barcodes:
Error: .wait() timed out after 30000msec
at Timeout._onTimeout (node_modules/nightmare/lib/actions.js:453:10)
10) Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
should add a proxy for user 1:
TypeError: Cannot read property 'barcode' of undefined
at Context.it (test/ui-testing/new_proxy.js:66:52)
11) Module test: users:new_proxy
Login > Find user two users > Add proxy to user 1 > Delete sponsor in user 2 > Logout
should delete a sponsor of user 2:
TypeError: Cannot read property 'barcode' of undefined
at Context.it (test/ui-testing/new_proxy.js:97:50)
Some tests failed or something went wrong while attempting to run the tests.
11
exiting process
Done in 227.20s.
Will follow up tomorrow by testing against a wider set of tenant modules. |
| Comment by John Malconian [ 15/Aug/18 ] |
|
Actually, those results should be expected. Enabling the module for the tenant with Okapi means nothing if it is not included in the bundle. Oversights like that mean its time to stop for the day. |
| Comment by Matthew Jones [ 15/Aug/18 ] |
|
I can confirm the same 11 failures when using tenant pr_463_10. There are only 2 failures (known, for the plugin) when running against tenant diku. This suggest to me the initial tenant data does not meet the test's expectations. The first failure occurs while looking for an active user. When not found, this has cascading effects on subsequent tests operating on that user. While there are active users in tenant pr_463_10, the test is looking for the 11th user in the list. However, this tenant only has 3 users to choose from. (Note: There are now more users in this tenant since I've run tests a couple of times, one of which creates a user.) In patron_group.js: .wait('#list-users div[role="listitem"]:nth-of-type(11) > a > div:nth-of-type(5)') .evaluate(() => document.querySelector('#list-users div[role="listitem"]:nth-of-type(11) > a > div:nth-of-type(5)').title) Changing both of these :nth-of-type() selectors locally from "11" to say "3" allows more tests to pass, but even selecting the 3rd user seem arbitrary. The test probably should be updated to select the first it finds, or even better, create n users before looking for then nth one. Correctly finding a user will allow all these failing tests to pass: 1) should find an active user to edit
2) should find patron group ID for "alumni_1757"
3) should edit user record using "alumni_1757" group
4) should find ID for "Staff" group
5) should change patron group to "Staff" in user record
|
| Comment by Matthew Jones [ 15/Aug/18 ] |
|
The next failure is also due to lack of data. The new_permission_set test is looking for existing permissions to assign to the permission set it creates. .xclick('//button[contains(.,"Check in")]') //... .xclick('//button[contains(.,"Check out")]') Tenant diku has the permissions "Check in: All permissions" and "Check out: All permissions", however, tenant pr_463_10 does not have these. It only has permissions related to users and settings, which makes sense given that's what we are dealing with here. This test should be updated to select permissions pertaining only to the user's app to remove this cross-module data dependency, allowing the test to remain isolated. This fix will allow these failing tests to pass: 6) should create a new permission set
7) should confirm creation of new permission set
8) should delete new permission set
|
| Comment by Matthew Jones [ 15/Aug/18 ] |
|
The last failure (outside of the plugin ones) is also due to too few users. The new_proxy test is looking for the 9th user in the list when there aren't that many to choose from. This is like the earlier :nth-of-type, but using the :nth-child selector:
.wait('#list-users div[role="listitem"]:nth-child(9)')
By correctly selecting a user from the list, this test will pass: 9) should get active user barcodes |
| Comment by Matthew Jones [ 15/Aug/18 ] |
|
In summary, we have three ui-users tests that need updating due to data selections and one cross-module test that needs to be moved to platform-core (but probably not until we are ready to make the switch in CI). I will create appropriate UIU tickets for these in the morning.
Unfortunately I suspect testing other apps in isolation could yield similar findings. While it is more work now, decoupling these data and cross-module dependencies should have benefits in the longer term. Also, while debugging these tests, it seems pretty clear we could benefit from an abstraction layer (or what we're calling an interactor in bigtest) over our search results. Hunting for the nth div of the nth list item to grab a barcode, for example, seems pretty fragile. |
| Comment by Matthew Jones [ 17/Aug/18 ] |
|
I created
|
| Comment by John Malconian [ 27/Sep/18 ] |
|
stripes-cli is now a devDependency of folio-testing-platform,platform-core,and plaform-complete. It is used to build webpacks via 'yarn build' and used to invoke local tests via 'yarn test' in Stripes and UI modules where either local unit tests or integration tests have been implemented. It is also used to invoke cross-platform and module tests in platform-core and platform-complete via 'yarn test-regression' and 'yarn test-int'. |