[FOLIO-385] Integration tests for CI Created: 09/Nov/16 Updated: 12/Nov/18 Resolved: 11/Apr/17 |
|
| Status: | Closed |
| Project: | FOLIO |
| Components: | None |
| Affects versions: | None |
| Fix versions: | None |
| Type: | New Feature | Priority: | P3 |
| Reporter: | Wayne Schneider | Assignee: | Wolfram Schneider |
| Resolution: | Done | Votes: | 0 |
| Labels: | sprint10, sprint11, sprint7, sprint8, sprint9 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | 8 weeks, 2 days, 7 hours, 45 minutes | ||
| Original estimate: | Not Specified | ||
| Issue links: |
|
||||||||||||||||||||||||||||||||||||
| Sprint: | |||||||||||||||||||||||||||||||||||||
| Comments |
| Comment by Jakub Skoczen [ 26/Jan/17 ] |
|
Guys, I would like to see whether we can move ahead on this issue. Yesterday, John, Wayne and I discussed creating a suite of integration tests that would be run automatically and periodically against the entire installation of the platform (Okapi, modules, UI). As such I was thinking Selenium might be a good tool for the job. Wolfram Schneider might have some other ideas as well. |
| Comment by Wolfram Schneider [ 26/Jan/17 ] |
|
I found this tool: http://screenster.io/selenium-alternatives-for-testing-automation/ I think that Selenium is great, but we should also have a look at the alternatives. |
| Comment by Jakub Skoczen [ 31/Jan/17 ] |
|
Here are some rough test cases/scenarios:
|
| Comment by Jakub Skoczen [ 02/Feb/17 ] |
|
Wolfram, what is the status of this issue? Have you made any progress? |
| Comment by Julian Ladisch [ 02/Feb/17 ] |
|
Wolfram Schneider, just in case its not on your list of tools to evaluate: http://www.sikuli.org/ |
| Comment by Wolfram Schneider [ 06/Feb/17 ] |
|
I put my notes in a google docs: |
| Comment by Julian Ladisch [ 06/Feb/17 ] |
|
Wolfram Schneider, access to https://docs.google.com/a/indexdata.com/ is restricted, you may upload the document into this issue or create a page on wiki.folio.org. |
| Comment by Jakub Skoczen [ 07/Feb/17 ] |
|
Wolfram Schneider you have missed the backend meeting today where we tried to discuss this issue. Can you please provide an update here? |
| Comment by Wolfram Schneider [ 09/Feb/17 ] |
|
I worked with the wraith tool, to run tests. I setup some examples at http://mkws-dev.indexdata.com/wraith/ I used our most complicated tool “MKWS” for testing. it works pretty well to detect small visual changes on a web site. e.g. on http://mkws-dev.indexdata.com/wraith/mkws-history/ you can see a small change (in red), which is the running time. you can compare 2 sites, e.g. dev and production. Or you run a history (old, new). This is a high level test. You can run it with different screen sizes. There is nothing to code, just to setup a config file. It is all open source, free, and run on the command line, on your local computer. For JS developers, I think we still need code based testing frameworks as mocha or jest. I'm looking at this right now. |
| Comment by Wolfram Schneider [ 21/Feb/17 ] |
|
For unkown reasons, the phantomjs tool cannot make screen shots of our folio trivial examples. Even simple images fails with a segfault. After some hours of debugging giving up now and will use an other demo page. Something is wrong with the generated bundle, it cannot be parsed in a headless webkit. SyntaxError: Parse error I will discuss this later with Jason. |
| Comment by Wolfram Schneider [ 22/Feb/17 ] |
|
The segfault of phantomjs is related to the older version of phantomjs 1.9.20 (which we are using for MKWS). The newer phantomjs 2.1.x seems to more stable. Also, casperjs seems to be a better tool for screenshots. Wraith has a strange ruby error message for the screen shot history, it works only once and not again. It turns out that the framework has problem with absolute paths, and works only with a relative path |
| Comment by Wolfram Schneider [ 01/Mar/17 ] |
|
The error message "SyntaxError: Parse error" is the reason why phantomjs fails to take any screenshots. See also |
| Comment by Wolfram Schneider [ 01/Mar/17 ] |
|
Vagrant runs fine on my mac at home. In the office, I got a HTTP 404, very strange. But good to know it works at home on a modern mac. |
| Comment by Wolfram Schneider [ 02/Mar/17 ] |
|
The latest version of folio/folio-demo v0.7.0 runs fine on my old mac now. |
| Comment by Wolfram Schneider [ 02/Mar/17 ] |
|
The bundle.js file in folio/folio-demo is 9.2MB huge. Wow! It is amazing that he browser can parse this. I see sometimes a warning that it took 400ms to parse. All major browsers (Firefox, Safari, Chrome) can parse the file. However, phantomjs fails. The github bug reports of phantomjs suggest to run the JS in node.js, to get the error message. But this doesn't work if a browser "document" object is required. The bug reports also says that the issue should be fixed in phantomjs 2.1.x, which is apparently not true for folio-ui. I run a test with the JavaScript syntax checker "esprima" and the bundle.js looks fine. https://www.npmjs.com/package/esprima |
| Comment by Wolfram Schneider [ 02/Mar/17 ] |
|
uglify cannot parse the bundle. But I don't see an error in the code, maybe this is related to newer ES6 syntax: $ uglifyjs bundle.js Parse error at bundle.js:25120,183 ial',module:'@folio/trivial',getModule:() => __webpack_require__(714).default,mo ^ SyntaxError: Unexpected token: punc ()) at JS_Parse_Error.get (eval at <anonymous> (/usr/local/lib/node_modules/uglify-js/tools/node.js:28:1), <anonymous>:86:23) at /usr/local/lib/node_modules/uglify-js/bin/uglifyjs:382:39 at time_it (/usr/local/lib/node_modules/uglify-js/bin/uglifyjs:616:15) at /usr/local/lib/node_modules/uglify-js/bin/uglifyjs:343:9 at tryToString (fs.js:447:3) at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:434:12) |
| Comment by Jakub Skoczen [ 02/Mar/17 ] |
|
Discussed on the call: Wolfram to take Jest for a spin before the end of the week, if this is not promising he will turn over to Jasmine. |
| Comment by Jakub Skoczen [ 20/Mar/17 ] |
|
Wolfram, do we have a Jenkins job running? Have you successfully implemented test scenarios with nightmare.js? If so, where can we see test result/log? |
| Comment by Jakub Skoczen [ 21/Mar/17 ] |
|
Nigthmare.js and mokka in use for tests. The jenkins job is still to be set up. |
| Comment by John Malconian [ 21/Mar/17 ] |
|
Just a couple of comments previously discussed with Wolfram:
|
| Comment by Wolfram Schneider [ 23/Mar/17 ] |
|
To run a test, you have a running UI installation, e.g.
git clone ssh://git@github.com/folio-org/stripes-sample-platform.git
cd stripes-sample-platform
yarn install
yarn start
Note: auth must be enabled. The url can be on your local machine http://localhost:9130 or the demo site
cat stripes.config.js
module.exports = {
okapi: { 'url':'http://localhost:9130', 'tenant':'diku' },
config: {
reduxLog: true,
disableAuth: false
// candidate logCategories: 'core,redux,path,mpath,action,perm,xhr'
},
modules: {
'@folio/trivial': {},
'@folio/users': {},
'@folio/items': {},
'@folio/scan': {}
}
};
The nighmare/mocca tests setup is in my git account https://github.com/wschnrid/folio-ui-test git clone git@github.com:wschnrid/folio-ui-test.git yarn install yarn test-simple This should work.
yarn test
yarn test v0.21.3
$ ./node_modules/.bin/mocha
Load a Page / 100-startpage
/ (Home Page)
✓ should load without error (1577ms)
A full test runs as: yarn test |
| Comment by Wolfram Schneider [ 23/Mar/17 ] |
|
There are 2 ways to debug the tests. With nightmare, you can use the
DEBUG=nightmare yarn test-simple
yarn test-simple v0.21.3
$ ./node_modules/.bin/mocha test/100-startpage.js
Load a Page / 100-startpage
/ (Home Page)
nightmare queuing process start +0ms
nightmare queueing action "goto" for http://localhost:3000 +4ms
nightmare queueing action "wait" +1ms
nightmare running +0ms
nightmare electron child process exited with code 0: success! +1s
✓ should load without error (1350ms)
1 passing (1s)
✨ Done in 1.88s.
on a desktop machine, you can view the browser window (FOLIO_UI_DEBUG=1) and the developer console (FOLIO_UI_DEBUG=2):
FOLIO_UI_DEBUG=2 yarn test-simple
yarn test-simple v0.21.3
$ ./node_modules/.bin/mocha test/100-startpage.js
Load a Page / 100-startpage
/ (Home Page)
✓ should load without error (1818ms)
1 passing (2s)
✨ Done in 2.32s.
|
| Comment by Wolfram Schneider [ 23/Mar/17 ] |
|
The regression tests works fine on MacOS and Linux (desktop). To run it on a server, you need a running X11 server. This could be implemented with a docker npm images with Xvfb installed. |
| Comment by John Malconian [ 23/Mar/17 ] |
|
Try this Dockerfile: FROM ubuntu:xenial
RUN apt-get -q update && \
DEBIAN_FRONTEND="noninteractive" apt-get -q install -y \
-o Dpkg::Options::="--force-confnew" --no-install-recommends \
git wget unzip xvfb && \
apt-get -q clean -y && rm -rf /var/lib/apt/lists/* && \
rm -f /var/cache/apt/*.bin
ENV NODEJS_VERSION 6
RUN wget --no-check-certificate --no-cookies https://deb.nodesource.com/setup_${NODEJS_VERSION}.x -O /tmp/node.sh && \
chmod +x /tmp/node.sh && \
sh -c "/tmp/node.sh" && \
rm -f /tmp/node.sh && \
npm install -g yarn && \
cd /usr/src && \
git clone https://github.com/wschnrid/folio-ui-test && \
yarn install
WORKDIR /usr/src/folio-ui-test
CMD ["yarn", "test"]
Pass your environment variables in during 'docker run' with the '-e' flag. Example: -e "FOLIO_UI_URL=http://folio-ci10.aws.indexdata.com:80" Let me know if this works. |
| Comment by Jakub Skoczen [ 30/Mar/17 ] |
|
Wolfram Schneider is this integrated with our CI? |
| Comment by Wolfram Schneider [ 30/Mar/17 ] |
|
I'm working on the docker/jenkins integration. It is not done yet, sorry. |
| Comment by Wolfram Schneider [ 31/Mar/17 ] |
|
The system testing repo moved to There is docker instance The tests pass fine inside docker. |
| Comment by Wolfram Schneider [ 31/Mar/17 ] |
|
John setup a jenkins job to re-build the docker images on demand |
| Comment by John Malconian [ 31/Mar/17 ] |
|
Just some details:
docker run -it --rm -e "FOLIO_UI_URL=http://folio-uidemo.aws.indexdata.com" folioci/ui-testing Let me know if there's anything else. |
| Comment by Wolfram Schneider [ 03/Apr/17 ] |
|
I enabled regression testing with Travis CI See https://travis-ci.org/wschnrid/ui-testing It will run every time after a git push, or a github pull request. The config file is .travis.yml , see |