[FOLIO-2674] Enable Jenkins jobs to save artifacts of Cypress for front-end tests Created: 02/Jul/20  Updated: 13/Jul/20  Resolved: 10/Jul/20

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: Task Priority: TBD
Reporter: David Crossley Assignee: David Crossley
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Attachments: PNG File Screen Shot 2020-07-03 at 14.19.37.png     PNG File build-7-20200704.png     PNG File expose-build-1-20200708.png    
Issue links:
Relates
relates to UICR-21 Add testing framework Closed
Sprint: DevOps: sprint 92
Development Team: FOLIO DevOps

 Description   

The testing framework cypress.io is being investigated.

For example see UICR-21 Closed , and its linked document for "testing-notes", and the "cypress-testing" branch of ui-courses.

Cypress generates artifacts (screenshots and videos). Enable Jenkins to save these.

Jenkins already does save other artifacts (e.g. yarn.lock). See jenkins-pipeline-libs/vars/buildNPM.groovy



 Comments   
Comment by Mike Taylor [ 02/Jul/20 ]

Thanks, David! Specifically, the two directories cypress/vidoes and cypress/screenshots would be useful to save.

Comment by David Crossley [ 03/Jul/20 ]

Done in jenkins-pipeline-libs/pull/87

The tar archive file just contains everything in the cypress output directory. Too difficult to be selective, e.g. there might not be a screenshots directory.

So there are a couple of minor extra directories packaged with it.

Comment by Mike Taylor [ 03/Jul/20 ]

Excellent, thank you! How do I access these saved files?

The other directories in cypress are the tests themselves, plus whatever plugins and whatnot I have written to support them. It does feel wrong to save these, so what I could do is reconfigure so that there is a single cypress/artifacts directory which will contain videos and screenshots if either of them is generated. I assume it would be trivial to change this PR accordingly?

Comment by David Crossley [ 03/Jul/20 ]

Yes that would be good. Nothing is easy of course - always takes time to test. But yes, please do that.

How to access – follow the "orange dot / green tick" from the most recent commit: https://github.com/folio-org/ui-courses/tree/cypress-testing and then the "Artifacts" tab at the top. Or go across to the "Classic interface" from that top-right panel. It is at the top of that page, e.g. https://jenkins-aws.indexdata.com/job/folio-org/job/ui-courses/job/cypress-testing/13/

Comment by Mike Taylor [ 03/Jul/20 ]

Ah, I see it at the top of the Jenkins report — nice!

Does the artifact have to be a single file (tar archive) or could it be the directory in browsable format, so you could go straight to the videos and screenshots without a download-and-unpack stage?

Comment by Mike Taylor [ 03/Jul/20 ]

Anyway, I have made the change to the Cypress configuration, so that everything we need to save is in one place: ui-courses/cypress/artifacts.

Comment by David Crossley [ 03/Jul/20 ]

This is applied to all UI module builds. So if a module has many tests (e.g. ui-users) then that would be unwieldy.

Comment by Mike Taylor [ 03/Jul/20 ]

OK.

Comment by David Crossley [ 04/Jul/20 ]

Improved the tar archive stucture in jenkins-pipeline-libs/pull/88

Comment by David Crossley [ 05/Jul/20 ]

Enabled a demonstration of the alternative view, for Jenkins to directly expose all Cypress artifacts.

See branch build #7 and the front page of that branch and the attached screenshot: build-7-20200704.png

So if a UI module has many tests then the normal view is pushed a long way down the page.

Unless someone (e.g. John Malconian or Ian Hardy) knows otherwise, then i gather that this is the normal behaviour – to display each individual file, not a directory folder.

Comment by Mike Taylor [ 06/Jul/20 ]

Many thanks, David, this looks great!

Let's go ahead and close this as complete.

Comment by Mike Taylor [ 07/Jul/20 ]

Sorry David Crossley, maybe I am still not quite there on this. Does something need to be done to configure individual branches of ui-courses to include the file-at-a-time listing of build artifacts as well as the single `cypress.tar.gz` bundle? Because while I see these individual-file links at the links you give above, I don't see them at master-branch build pages like https://jenkins-aws.indexdata.com/job/folio-org/job/ui-courses/job/master/234/ or at the master-branch summary page https://jenkins-aws.indexdata.com/job/folio-org/job/ui-courses/job/master/

Comment by David Crossley [ 07/Jul/20 ]

The current implementation is not listing the individual files. That was just a demonstration to show why listing every individual file would seriously clutter the Jenkins interface for a UI repository with lots of tests. (We are trying to plan ahead to the use of Cypress for other modules.)

Comment by David Crossley [ 07/Jul/20 ]

We need to address the concern of storing artifacts on the Jenkins server, especially when Cypress is utilised in more UI projects.

Now artifacts are only stored when one of the tests fail.

See jenkins-pipeline-libs/pull/89

As before, at GitHub select the coloured checkmark of the most recent commit. If the tests fail then see the "Artifacts" tab in the top panel. Or go across to the "Classic interface" from that top-right panel. The tar archive is at the top of that page.

Comment by Mike Taylor [ 07/Jul/20 ]

That was just a demonstration to show why listing every individual file would seriously clutter the Jenkins interface for a UI repository with lots of tests.

Oh! That's a real shame, it was super-helpful having the files right that!

What if you did list the files, but capped the list at (say) ten entries? Or even three?

Storing artifacts only on failure is fine, BTW.

Comment by David Crossley [ 07/Jul/20 ]

The "archiveArtifacts" does not have that abilty. It can select the files based on an Apache Ant fileSet, but not to filter them any further.

Comment by Mike Taylor [ 07/Jul/20 ]

So you're saying you can list all the file, or none of them, but that's it?

Comment by David Crossley [ 08/Jul/20 ]

Here is some documentation for archiveArtifacts which briefly explains, and also links to Apache Ant FileSet doc. So it can use a comma-separated list of filename patterns to be included. There is also an "excludes" option which can be applied in addition to the "includes" mask.

There is no ability to further filter, e.g. to limit to a specific number of entries.

I have created a demonstration branch of ui-courses, which uses a demonstration branch of jenkins-pipeline-libs.

The ui-courses branch deliberately evokes two test failures. I noticed that when such a failure happens, then Cypress generates screenshots. Their filename includes the term "...(failed).png". So the demonstration Ant pattern being used is "cypress/artifacts/**/*(failed).png" to select and expose only those particular files.

Is the pattern "*(failed).png" reliable?

Is just that set of archived files all that is needed?

Is the "cypress.tar.gz" still needed?

When a module such as ui-users (with many tests) uses Cypress, then this might flood Jenkins with failed screenshots.

The demonstration:
https://jenkins-aws.indexdata.com/job/folio-org/job/ui-courses/job/folio-2674-cypress-artifacts-expose/1/
and the attached screenshot of that:

Comment by David Crossley [ 08/Jul/20 ]

I went ahead and merged the branch referred to in the previous comment. See jenkins-pipeline-libs/pull/91

When there are failed tests, then it shows the Cypress screenshots for those, together with the cypress.tar.gz

Comment by Mike Taylor [ 09/Jul/20 ]

David Crossley Sorry I rather lost track of this Jira, but I think everything is now working as intended and we can go ahead and close it.

Comment by David Crossley [ 10/Jul/20 ]

Okay. However do note the comments above, that screenshots and archives are only available for builds that have "failed tests". The document in ui-courses needs adjustment.

Comment by Mike Taylor [ 10/Jul/20 ]

Good point, thanks for the reminder. I'll make that change.

Comment by Mike Taylor [ 10/Jul/20 ]

Done. https://github.com/folio-org/ui-courses/blob/master/doc/testing-with-cypress.md#jenkins-integration

Comment by David Crossley [ 13/Jul/20 ]

Mike Taylor I did fine-tuning for Jenkins so that we can use the default Cypress configuration, with no need to declare special folders.

(Done in jenkins-pipeline-libs/pull/93 and 94, and in ui-courses/pull/69)

That section in the ui-courses/doc/testing-with-cypress.md#jenkins-integration can now be reduced.

The demonstration there of failed tests does not need any modification, as it is still the same result.

Comment by Mike Taylor [ 13/Jul/20 ]

Thanks, David — updated the document accordingly.
https://github.com/folio-org/ui-courses/blob/master/doc/testing-with-cypress.md#jenkins-integration

Generated at Thu Feb 08 23:22:25 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.