[FOLIO-1771] clean up and isolate tests Created: 01/Feb/19  Updated: 03/Jun/20  Resolved: 21/Feb/19

Status: Closed
Project: FOLIO
Components: None
Affects versions: None
Fix versions: None

Type: New Feature Priority: P2
Reporter: Jakub Skoczen Assignee: Eric Valuk
Resolution: Done Votes: 0
Labels: platform-backlog, q1-performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original estimate: Not Specified

Issue links:
Blocks
blocks MODINVSTOR-238 Performance: GET 200 - instance-stora... Closed
blocks MODINVSTOR-182 analyze and optimize identifiers arra... Closed
Relates
relates to MODINVSTOR-247 filtering /item-storage/items by 'que... Closed
relates to FOLIO-1829 set up a Jenkins job to run 'platform... Closed
relates to FOLIO-1830 review 'platform-work-performance' tests Closed
relates to MODINVSTOR-246 filtering /instance-storage/instances... Closed
relates to MODINVSTOR-248 filtering /item-storage/items when us... Closed
relates to MODINVSTOR-256 Improve performance of /instance-stor... Closed
relates to FOLIO-1757 Add a test case for instance identifi... Closed
relates to FOLIO-1744 SPIKE: establish expected response ti... Closed
relates to FOLIO-1815 SPIKE: "profile" checkin/out-by-barco... Closed
Sprint: Core: Platform - Sprint 56, Core: Platform - Sprint 57
Story Points: 3
Development Team: Core: Platform

 Description   

As Eric Valuk tries to establish an expected response time baseline for a simple performance tests we have decided that the test suite needs to be reorganized to allow us to understand the results:

  • the tests need to be isolated – e.g by introducing a cooldown and warmup times between tests. Currently the tests directly influence each other which makes it impossible to understand perf regressions in particular APIs
  • we need smaller number of vetted test cases (workflows) – many of the tests are testing raw APIs with artificial parameters. This is too large surface area and too many conditions to optimize for. We should focus on tests that represent real-life load on the system.

Tasks to be peformed:

  • Cross check existing tests against 2 real world work flows.
  • Examine all large queries from standpoint of real world usage
  • set up a new Jenkins job to include only verified tests
  • Include only tests listed below
  • Delete/disable other tests
  • Add cool-down time between tests to isolate them
  • set up test parameters to match the number of threads used to configure our baseline test in FOLIO-1744 Closed

Test to be included in the initial job



 Comments   
Comment by Jakub Skoczen [ 06/Feb/19 ]

Eric Valuk Thanks for providing acceptance criteria. Should we clarify the initial workflows that are going to be included? I understand this is Check-in and RTAC workflows?

Comment by Jakub Skoczen [ 06/Feb/19 ]

Eric Valuk Let's also make sure to include the following tests:

  • the identifier/isbn filtering tests that Hongwei added recently
  • a look up of instance by id ( MODINVSTOR-246 Closed )
  • a look up of item by id ( MODINVSTOR-248 Closed )
  • a look up of item by barcode (Not sure if there was an issue for it?)
  • a look up of instance by location ( MODINVSTOR-256 Closed )
Comment by Jakub Skoczen [ 07/Feb/19 ]

Eric Valuk Hongwei Ji Guys, as we discussed yesterday on the call it is OK to set this as a new performance job in Jenkins, independent from the existing "big" performance job to avoid compromising the quality dashboard.

Comment by Hongwei Ji [ 07/Feb/19 ]

I propose we put new tests under core-platform folder (which I have done for the new test I created for FOLIO-1757 Closed identifier search). Then we can adjust the existing Jenkins job pipeline to run all tests under core-platform folder first, so we will have both new tests and old tests. The benefits are 1) reduced complexity 2) reused infrastructure 3) side by side comparison. If the same API shows dramatically different response time between new and old tests, we will know the old tests were not created/executed properly (whether it was caused by other tests in the same workflow or system under load due to previous tests).

Comment by Eric Valuk [ 21/Feb/19 ]

merged today

Comment by Jakub Skoczen [ 27/Feb/19 ]

Eric Valuk Hongwei Ji Guys, If I recall correctly we talked about creating a seperate issue for creating a dedicated Jenkins job for the aforementioned tests?

Update: I have chatted with Hongwei Ji and created FOLIO-1829 Closed for this.

Generated at Thu Feb 08 23:15:47 UTC 2024 using Jira 1001.0.0-SNAPSHOT#100246-sha1:7a5c50119eb0633d306e14180817ddef5e80c75d.