[FOLIO-385] Integration tests for CI Created: 09/Nov/16  Updated: 12/Nov/18  Resolved: 11/Apr/17

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P3
Reporter: Wayne Schneider Assignee: Wolfram Schneider
Resolution: Done Votes: 0
Labels: sprint10, sprint11, sprint7, sprint8, sprint9
Remaining Estimate: Not Specified
Time Spent: 8 weeks, 2 days, 7 hours, 45 minutes
Original estimate: Not Specified

Issue links:
Blocks
blocks FOLIO-447 Build CI pipeline for integration tes... Closed
Relates
relates to FOLIO-555 write additional test-cases Open
relates to FOLIO-549 new git repo folio-testing-platform Closed
relates to STCOR-27 stripes-core won't run tests when dep... Closed
relates to STRIPES-15 UI tests Closed
relates to STRIPES-300 Assign unique IDs to UI elements as ... Closed
relates to FOLIO-539 CI UI test failure in FOLIO Demo Closed
Sprint:

 Comments   
Comment by Jakub Skoczen [ 26/Jan/17 ]

Guys, I would like to see whether we can move ahead on this issue. Yesterday, John, Wayne and I discussed creating a suite of integration tests that would be run automatically and periodically against the entire installation of the platform (Okapi, modules, UI). As such I was thinking Selenium might be a good tool for the job. Wolfram Schneider might have some other ideas as well.

Comment by Wolfram Schneider [ 26/Jan/17 ]

I found this tool: http://screenster.io/selenium-alternatives-for-testing-automation/

I think that Selenium is great, but we should also have a look at the alternatives.

Comment by Jakub Skoczen [ 31/Jan/17 ]

Here are some rough test cases/scenarios:

  • user logs in
  • user opens the Users app
  • user searches for a set of records
  • user changes sort order
  • user updates a record
  • user creates a record
  • user deletes a record
Comment by Jakub Skoczen [ 02/Feb/17 ]

Wolfram, what is the status of this issue? Have you made any progress?

Comment by Julian Ladisch [ 02/Feb/17 ]

Wolfram Schneider, just in case its not on your list of tools to evaluate: http://www.sikuli.org/

Comment by Wolfram Schneider [ 06/Feb/17 ]

I put my notes in a google docs:

https://docs.google.com/a/indexdata.com/document/d/1XaY9Z4HGgmEL40xSxovYyZPbIz1uCHUQj_Mk9lsVfk8/edit?usp=sharing

Comment by Julian Ladisch [ 06/Feb/17 ]

Wolfram Schneider, access to https://docs.google.com/a/indexdata.com/ is restricted, you may upload the document into this issue or create a page on wiki.folio.org.

Comment by Jakub Skoczen [ 07/Feb/17 ]

Wolfram Schneider you have missed the backend meeting today where we tried to discuss this issue. Can you please provide an update here?

Comment by Wolfram Schneider [ 09/Feb/17 ]

I worked with the wraith tool, to run tests. I setup some examples at http://mkws-dev.indexdata.com/wraith/

I used our most complicated tool “MKWS” for testing.

it works pretty well to detect small visual changes on a web site. e.g. on http://mkws-dev.indexdata.com/wraith/mkws-history/ you can see a small change (in red), which is the running time.

you can compare 2 sites, e.g. dev and production. Or you run a history (old, new).

This is a high level test. You can run it with different screen sizes. There is nothing to code, just to setup a config file. It is all open source, free, and run on the command line, on your local computer.

For JS developers, I think we still need code based testing frameworks as mocha or jest. I'm looking at this right now.

Comment by Wolfram Schneider [ 21/Feb/17 ]

For unkown reasons, the phantomjs tool cannot make screen shots of our folio trivial examples. Even simple images fails with a segfault. After some hours of debugging giving up now and will use an other demo page.

Something is wrong with the generated bundle, it cannot be parsed in a headless webkit.

SyntaxError: Parse error

I will discuss this later with Jason.

Comment by Wolfram Schneider [ 22/Feb/17 ]

The segfault of phantomjs is related to the older version of phantomjs 1.9.20 (which we are using for MKWS). The newer phantomjs 2.1.x seems to more stable. Also, casperjs seems to be a better tool for screenshots.

Wraith has a strange ruby error message for the screen shot history, it works only once and not again. It turns out that the framework has problem with absolute paths, and works only with a relative path . We can solve the issue with a symlink. I found github issues for similar errors, but none fixed our issue.

Comment by Wolfram Schneider [ 01/Mar/17 ]

The error message "SyntaxError: Parse error" is the reason why phantomjs fails to take any screenshots.

See also
https://github.com/ariya/phantomjs/issues/11640
https://stackoverflow.com/questions/14905664/getting-more-information-from-phantomjs-syntaxerror-parse-error-message
https://stackoverflow.com/questions/28165420/getting-more-information-from-syntaxerror-parse-error-message-in-phantomjs-ca/32212962

Comment by Wolfram Schneider [ 01/Mar/17 ]

Vagrant runs fine on my mac at home. In the office, I got a HTTP 404, very strange. But good to know it works at home on a modern mac.

Comment by Wolfram Schneider [ 02/Mar/17 ]

The latest version of folio/folio-demo v0.7.0 runs fine on my old mac now.

Comment by Wolfram Schneider [ 02/Mar/17 ]

The bundle.js file in folio/folio-demo is 9.2MB huge. Wow! It is amazing that he browser can parse this. I see sometimes a warning that it took 400ms to parse.

All major browsers (Firefox, Safari, Chrome) can parse the file. However, phantomjs fails.

The github bug reports of phantomjs suggest to run the JS in node.js, to get the error message. But this doesn't work if a browser "document" object is required. The bug reports also says that the issue should be fixed in phantomjs 2.1.x, which is apparently not true for folio-ui.

I run a test with the JavaScript syntax checker "esprima" and the bundle.js looks fine.

https://www.npmjs.com/package/esprima
Esprima (esprima.org, BSD license) is a high performance, standard-compliant ECMAScript parser written in ECMAScript (also popularly known as JavaScript)

Comment by Wolfram Schneider [ 02/Mar/17 ]

uglify cannot parse the bundle. But I don't see an error in the code, maybe this is related to newer ES6 syntax:
http://exploringjs.com/es6/ch_arrow-functions.html

$ uglifyjs bundle.js 
Parse error at bundle.js:25120,183
ial',module:'@folio/trivial',getModule:() => __webpack_require__(714).default,mo
                                        ^
SyntaxError: Unexpected token: punc ())
    at JS_Parse_Error.get (eval at <anonymous> (/usr/local/lib/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:86:23)
    at /usr/local/lib/node_modules/uglify-js/bin/uglifyjs:382:39
    at time_it (/usr/local/lib/node_modules/uglify-js/bin/uglifyjs:616:15)
    at /usr/local/lib/node_modules/uglify-js/bin/uglifyjs:343:9
    at tryToString (fs.js:447:3)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:434:12)
Comment by Jakub Skoczen [ 02/Mar/17 ]

Discussed on the call: Wolfram to take Jest for a spin before the end of the week, if this is not promising he will turn over to Jasmine.

Comment by Jakub Skoczen [ 20/Mar/17 ]

Wolfram, do we have a Jenkins job running? Have you successfully implemented test scenarios with nightmare.js? If so, where can we see test result/log?

Comment by Jakub Skoczen [ 21/Mar/17 ]

Nigthmare.js and mokka in use for tests. The jenkins job is still to be set up.

Comment by John Malconian [ 21/Mar/17 ]

Just a couple of comments previously discussed with Wolfram:

  • In the CI environment, run the testing tool inside a Docker container so that we don't clutter the Jenkins host with development tools as well as easily deploy the tool to run on other hosts.
Comment by Wolfram Schneider [ 23/Mar/17 ]

To run a test, you have a running UI installation, e.g.

git clone ssh://git@github.com/folio-org/stripes-sample-platform.git
cd stripes-sample-platform
yarn install
yarn start

Note: auth must be enabled. The url can be on your local machine http://localhost:9130 or the demo site
http://folio-c10.aws.indexdata.com:9130

cat stripes.config.js 
module.exports = {
  okapi: { 'url':'http://localhost:9130', 'tenant':'diku' },
  config: {
    reduxLog: true,
    disableAuth: false
    // candidate logCategories: 'core,redux,path,mpath,action,perm,xhr'
  },
  modules: {
    '@folio/trivial': {},
    '@folio/users': {},
    '@folio/items': {},
    '@folio/scan': {}
  }
};

The nighmare/mocca tests setup is in my git account https://github.com/wschnrid/folio-ui-test

git clone git@github.com:wschnrid/folio-ui-test.git
yarn install
yarn test-simple

This should work.

yarn test
yarn test v0.21.3
$ ./node_modules/.bin/mocha 


  Load a Page / 100-startpage
    / (Home Page)
      ✓ should load without error (1577ms)

A full test runs as:

yarn test
Comment by Wolfram Schneider [ 23/Mar/17 ]

There are 2 ways to debug the tests. With nightmare, you can use the
DEBUG=nightmare or DEBUG=nightmare*. This runs on the command line:

DEBUG=nightmare yarn test-simple
yarn test-simple v0.21.3
$ ./node_modules/.bin/mocha test/100-startpage.js 


  Load a Page / 100-startpage
    / (Home Page)
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://localhost:3000 +4ms
  nightmare queueing action "wait" +1ms
  nightmare running +0ms
  nightmare electron child process exited with code 0: success! +1s
      ✓ should load without error (1350ms)


  1 passing (1s)

✨  Done in 1.88s.

on a desktop machine, you can view the browser window (FOLIO_UI_DEBUG=1) and the developer console (FOLIO_UI_DEBUG=2):

FOLIO_UI_DEBUG=2 yarn test-simple
yarn test-simple v0.21.3
$ ./node_modules/.bin/mocha test/100-startpage.js 


  Load a Page / 100-startpage
    / (Home Page)
      ✓ should load without error (1818ms)


  1 passing (2s)

✨  Done in 2.32s.
Comment by Wolfram Schneider [ 23/Mar/17 ]

The regression tests works fine on MacOS and Linux (desktop). To run it on a server, you need a running X11 server. This could be implemented with a docker npm images with Xvfb installed.

Comment by John Malconian [ 23/Mar/17 ]

Wolfram Schneider

Try this Dockerfile:

FROM ubuntu:xenial

RUN apt-get -q update && \
    DEBIAN_FRONTEND="noninteractive" apt-get -q install -y \
    -o Dpkg::Options::="--force-confnew"  --no-install-recommends \
    git wget unzip xvfb && \
    apt-get -q clean -y && rm -rf /var/lib/apt/lists/* && \
    rm -f /var/cache/apt/*.bin 

ENV NODEJS_VERSION 6

RUN wget --no-check-certificate --no-cookies https://deb.nodesource.com/setup_${NODEJS_VERSION}.x -O /tmp/node.sh  && \
    chmod +x /tmp/node.sh && \
    sh -c "/tmp/node.sh" && \
    rm -f /tmp/node.sh && \
    npm install -g yarn && \
    cd /usr/src && \
    git clone https://github.com/wschnrid/folio-ui-test && \
    yarn install

WORKDIR /usr/src/folio-ui-test
     
CMD ["yarn", "test"]

Pass your environment variables in during 'docker run' with the '-e' flag. Example:

-e "FOLIO_UI_URL=http://folio-ci10.aws.indexdata.com:80"

Let me know if this works.

Comment by Jakub Skoczen [ 30/Mar/17 ]

Wolfram Schneider is this integrated with our CI?

Comment by Wolfram Schneider [ 30/Mar/17 ]

I'm working on the docker/jenkins integration. It is not done yet, sorry.

Comment by Wolfram Schneider [ 31/Mar/17 ]

The system testing repo moved to
https://github.com/folio-org/ui-testing

There is docker instance
https://hub.docker.com/r/folioci/ui-testing/

The tests pass fine inside docker.

Comment by Wolfram Schneider [ 31/Mar/17 ]

John setup a jenkins job to re-build the docker images on demand
https://jenkins-aws.indexdata.com/job/ui-testing

Comment by John Malconian [ 31/Mar/17 ]

Just some details:

  • I set up a Jenkin's job called ui-testing which looks for changes to the folio-org/ui-testing project and build and publishes docker images of the test tool to our folioci repo on Docker Hub.
  • I added a build step to the FOLIO_UI_Demo_1 and FOLIO_UI_Demo_2 Jenkin's jobs (the jobs that build the FOLIO Demo at http://folio-uidemo.aws.indexdata.com. The build step runs the UI tests against the demo. If there is a failure, you will be notified via email.
  • I modified the ui-testing Dockerfile so that it's easier to run. e.g:

docker run -it --rm -e "FOLIO_UI_URL=http://folio-uidemo.aws.indexdata.com" folioci/ui-testing

Let me know if there's anything else.

Comment by Wolfram Schneider [ 03/Apr/17 ]

I enabled regression testing with Travis CI

See https://travis-ci.org/wschnrid/ui-testing

It will run every time after a git push, or a github pull request.

The config file is .travis.yml , see
https://github.com/folio-org/ui-testing/blob/master/.travis.yml

Generated at Thu Feb 08 23:05:22 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.