[FOLIO-1043] integrate ui-testing with PR Created: 30/Jan/18  Updated: 20/Mar/19  Resolved: 20/Mar/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Umbrella Priority: P3
Reporter: Jakub Skoczen Assignee: John Malconian
Resolution: Duplicate Votes: 0
Labels: build-release, ci, devops, madrid, sprint31, sprint32, sprint33, sprint34, sprint36, sprint37, sprint38, sprint39, sprint40, sprint41, sprint42, sprint43, sprint44, sprint45
Remaining Estimate: Not Specified
Time Spent: 4 weeks, 2 days
Original estimate: Not Specified

Issue links:
Blocks
is blocked by FOLIO-1111 Add Docker-based launch descriptors t... Closed
is blocked by FOLIO-1212 Missing records in UI Regression Tests Closed
is blocked by FOLIO-1266 Reorganizing CI and nightly with plat... Closed
is blocked by FOLIO-1338 Regression tests fail consistently in... Closed
is blocked by RMB-154 Investigate postgresql connection pool Closed
is blocked by UIU-604 Address data dependencies in Nightmar... Closed
is blocked by UIU-605 Address cross-module dependency in Ni... Closed
is blocked by FOLIO-1270 capture yarn lock for the last succes... Closed
is blocked by FOLIO-1312 Intermittant ansible deployment error... Closed
is blocked by UICHKOUT-414 Testing error_messages: Don't use 'di... Closed
Duplicate
duplicates FOLIO-1577 Automated builds for FOLIO 'release' Closed
Relates
relates to FOLIO-1212 Missing records in UI Regression Tests Closed
relates to FOLIO-1338 Regression tests fail consistently in... Closed
relates to FOLIO-1204 CircleCI for testing ui-* and stripes-* Closed
relates to OKAPI-603 define and implement Okapi tenant cle... Closed
relates to STCLI-59 Add ability to post module descriptor... Closed
relates to STCLI-60 Assigning perms throws error Closed
relates to FOLIO-1051 Refactor folio-ansible "data" roles ... Closed
relates to UIU-604 Address data dependencies in Nightmar... Closed
relates to UIU-605 Address cross-module dependency in Ni... Closed
relates to FOLIO-1022 Experment with module upgrade within ... Closed
relates to FOLIO-1050 Investigate Greenkeeper as possible a... Closed
relates to UITEST-16 work on maintaining the test suite so... Closed
relates to UIU-400 Users App: Status Filter Update Closed
relates to UXPROD-1055 Continuous Integration and Deployment Closed
relates to UXPROD-1059 Pull Request builds and integration t... Closed
Sprint:
Development Team: Core: Platform

 Description   

The idea is to speed up construction of the test system (maybe re-use the same system and upgrade selected components only) and the test run so it is possible to integrate it with the PR.

1. John Malconian Wayne Schneider Describe the approach



 Comments   
Comment by John Malconian [ 31/Jan/18 ]

The idea, at least initially, is to run the UI regression test suite as part of a PR for any of the Stripes and UI-based repositories, This would be orchestrated by Jenkins. The UI test suite would be run against a pre-built FOLIO backend stack that is rebuilt daily. In order to set a known, good, baseline, the backend stack that is used will be an instance of 'folio-snapshot-stable'. For each PR, the CI will create a tenant on the backend, load a subset of data for that tenant, build a stripes platform which will include the changes for the UI component being testing. The UI regression test suite (or a portion of it) will be run against that stripes build. If all tests pass, the PR will be available to merge to master. If any tests fail, the developer will be notified and the pipeline stops until the issue is fixed and a subsequent PR is run and passes.

Comment by Wayne Schneider [ 07/Feb/18 ]

Process document: https://docs.google.com/document/d/1YcHV7Dx-LMGckyLVPu3izLtgfrSR-0rFxNsLbsYuYfI

Comment by John Malconian [ 13/Mar/18 ]

There are two outstanding issues that need to be added for this to be completed:

  • add the ability to deploy new/updated modules via the install endpoint (see also folio-1111)
  • add sample user data to the tenant instance
Comment by Jakub Skoczen [ 14/Mar/18 ]

We discussed that it would make sense to enable that for ui-users first, sometime next week.

Comment by John Malconian [ 30/Mar/18 ]

Couple of things that are holding back this implementation:

  • We need a working folio-snapshot-stable. Last "stable" build (e.g passed regression tests was March 20.
  • RMB-154 Closed . This is causing issues with more than 3-4 tenants
Comment by Jakub Skoczen [ 11/Apr/18 ]

We have discussed two things:

  • putting the lin yarn.lock on the Jenkins build page
  • running the build in the DEBUG mode (for informative nightmare errors)
Comment by John Malconian [ 11/Apr/18 ]

Test implementation exists on branch folio-1043-test of ui-users. There seems to be a discrepancy in the number of test failures compared to the nightly tests against folio-snapshot during roughly the same period. Some things to implement in an attempt to drill down into the cause of the discrepancy:

  • 'link' in the repository code in the ui-testing platform
  • remove the node_modules directory in the repository code after the initial 'yarn install'
  • publish the yarn.lock file with the build that is generated in folio-testing-platform.
Comment by John Malconian [ 12/Apr/18 ]

After fiddling around more with 'yarn linking' and adding the mod-auth-demo-users role to the tenant configuration, the last run produced only three failures.

https://jenkins-aws.indexdata.com/job/folio-org/job/ui-users/job/folio-1043-test/88/UI_Regression_Test_Report/

Comment by John Malconian [ 12/Apr/18 ]

Niels Erik Nielsen has pointed out that a couple of the regression tests rely on the existence of the 'diku_admin' user. However, the test is run against a different tenant/tenant admin.

Comment by charles [ 12/Apr/18 ]

It looks like the tests are expecting a certain title to be in the codex search, and it can't find any active users. Are we loading the same users and codex titles as in http://folio-testing.aws.indexdata.com/ ?

Comment by John Malconian [ 12/Apr/18 ]

The codex search failure fails because the title, Arctic A-Z, exists only in a larger sample inventory data set that is not loaded for PR tests where only a minimal dataset is loaded. I'd rather not have to load the larger dataset for each tenant/PR. Perhaps we can change the behavior of this test somehow?

Comment by John Malconian [ 12/Apr/18 ]

For example, the "New POPS" title is included in the minimal dataset so it would be found in all tenant instances (except it wouldn't be the first title found in instances with the larger dataset included.

Comment by charles [ 12/Apr/18 ]

Ok-- this sounds good.

Comment by charles [ 12/Apr/18 ]

I changed the search term in the codex search test to "POPs in Marine Mammals".

Comment by John Malconian [ 13/Apr/18 ]

It appears all issues related to passing/failing UI tests have been worked out. I've also enabled publishing of yarn.lock files in Jenkins (they can be downloaded/viewed per job). I've been able to run the full UI regression suite without error in testing. I intend to implement full-stack PR-based regression testing on two repositories - ui-users and ui-inventory - on Monday morning. We'll see how it goes and make adjustments from there.

Comment by John Malconian [ 01/May/18 ]

I've implemented a couple of new features into CI.

  • PR-based regression tests are configurable in the project's Jenkinsfile:

runRegression = 'partial' - Run module's UI regression tests only.
runRegression = 'full' - Run full UI regression test suite.
runRegression = 'none' - Do not run UI regression tests (this is the default when not explicitly configured.

  • Regression testing is skipped if there is a module dependency mismatch.

Currently, the 'ui-inventory' and 'ui-users' repositories are configured with "runRegression = 'partial'". I'll enable regression testing for additional repositories gradually.

Comment by Niels Erik Nielsen [ 01/May/18 ]

Cool!

Comment by John Malconian [ 11/May/18 ]

Currently, when a PR submitted for regression testing, the module that is tested is linked into 'folio-testing-platform' and the whole stripes platform is built and posted to Okapi. I'm thinking it would be more efficient to test the module with a more barebones stripes platform that consists only of the module being tested along with common stripes dependencies (stripes-core, stripes-components) and enabling only the backend modules that are explicitly specified as okapi interface dependencies.

Comment by Adam Dickmeiss [ 11/May/18 ]

That would be great .. It would also check that the dependencies are right.

Comment by Matthew Jones [ 11/May/18 ]

I'm thinking it would be more efficient to test the module with a more barebones stripes platform that consists only of the module being tested

John Malconian, Stripes CLI does exactly this. From the app module's directory, run:

stripes test nightmare
Comment by John Malconian [ 01/Jun/18 ]

Implemented 'partial' regression tests for PRs on ui-requests and ui-checkout projects. In total, PR partial regression testing is enabled now for ui-users, ui-inventory, ui-requests, and ui-checkout. Correct me if I'm wrong Niels Erik Nielsen charles, but I don't think there are any other module specific test suites for any other UI module (ui-checkin, for example). So for all other UI modules, we either need to enable full regression tests for PRs or not enable any tests at all?

Comment by Niels Erik Nielsen [ 01/Jun/18 ]

Right John Malconian Personally I think non-core module should only run their own tests on the PR. If they don't have any they will not run any.

The developer should optimally still run relevant cross-module tests (typically 'test-exercise') or – if that's simpler – the entire suite, locally.

Comment by Niels Erik Nielsen [ 01/Jun/18 ]

Regarding running a platform with only the module being tested: https://folio-org.atlassian.net/browse/FOLIO-1043?focusedCommentId=189585&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel

I know I've said it elsewhere, but just a note for the record here as well:

  • Some tests go out to find sample data for testing in other modules.
  • Some modules invoke other modules.
Comment by Jakub Skoczen [ 06/Jun/18 ]

We have discussed an alternative approach to yarn linking whereby we specify the dependency of the package under test as the SHA1 of the Git commit. This can be done through a yarn command. The yarn.lock will be retrieved during the build process from a snapshot-stable URL (where it is already retain after each successful build).

Comment by John Malconian [ 26/Jun/18 ]

Here is a summary of the latest changes:

  • Instead of updating the package dependencies SHA1, I opted to use 'yard add file: ...' instead. The branch that is getting tested and merged has already been cloned and the project has already undergone all the CI machinations in regards to versioning, etc. It's effectively the same as using the git commit hash.
  • I've added the 'stripesPlatform' Jenkinsfile config option. 'stripesPlatform' specifies the Stripes platform to use when building the webpack. The default right now if this option is not specified is 'folio-testing-platform'. Other options currently are 'platform-core', 'folio-testing-platform' and 'none'. When 'none' is specified, stripes-cli will build an "app mode" minimal bundle. Currently, 'none' has been configured for the stripes-core, stripes-components, and stripes-smart-components repos. 'platform-core' has been configured for ui-users, ui-inventory, ui-requests, and ui-checkout. The rest are, therefore, still using 'folio-testing-plaform'. 'folio-testing-platform' probably shouldn't be the long-term default, but, for now, it was easiest in order to implement these changes.
  • Use of the current yarn.lock from folio-snapshot-stable when building Stripes platform.

I've also learned to never specify 'NODE_OPTIONS=--max_old_space_size...' globally. I spent about 3 days trying to figure out why the regression tests wouldn't run after making the above changes that was due to this.

Comment by Matthew Jones [ 27/Jun/18 ]

FYI, I've updated stripe-cli's own jenkinsfile platform to 'none' as well. Technically the CLI doesn't have anything to build in app mode, so I'm wondering if we should introduce a platform value that does even less than 'none'.

At this point, do you need STCLI-63 Closed complete to invoke the tests?

Comment by John Malconian [ 23/Jul/18 ]

Disable regression testing for PRs on ui-checkout, ui-requests, ui-users, and ui-inventory. Blocked by FOLIO-1338 Closed and FOLIO-1312 Closed .

Comment by Marc Johnson [ 23/Jul/18 ]

John Malconian Can you help me understand why running the tests has been completely disabled?

Comment by John Malconian [ 23/Jul/18 ]

Marc Johnson - A combination of the systematic test failures in PRs that are not duplicated on the nightly tests as well as FOLIO-1312 Closed which is producing far too many failures that have nothing to with the actual code being tested.

Comment by John Malconian [ 31/Jul/18 ]

Re-enabled regression testing for PRs for ui-checkout, ui-requests, ui-users, and ui-inventory, In addition, added module tests for ui-organization. Will need to monitor for any signs of FOLIO-1312 Closed .

Comment by John Malconian [ 07/Aug/18 ]

Disable regression testing for PRs for ui-checkout, ui-requests, ui-users, and ui-inventory, and ui-organization until FOLIO-1312 Closed is resolved.

Generated at Thu Feb 08 23:10:24 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.