Overview

Currently there are existing workflow scripts that test the workflows individually.  Sometimes these workflows are tested in combination with other workflows by invoking a second workflow manually, such as an automated Check In Check Out test with a manual Data Import.  In order to simulate production usage of FOLIO, it is necessary to have a "master" JMeter script which executes all of these workflows so that all of them can be run concurrently in an automated fashion. This page outlines various aspects and considerations of designing such a script and the final decisions that will lead to creating the master script.

Script Requirements

1. Script that runs the workflows described in PERF-457 - Getting issue details... STATUS
2. The workflows are configurable before running the tests. Some scripts or workflows require specific configurations. While the Check In Check Out workflow needs to have the number of users and ramp up time configured, Data Import scripts may require the job profile and MARC files that are being loaded at runtime set. 
3. Script should run from Carrier-io.  Currently Carrier-io kicks off a test using Jenkins job and the job is configurable via the job's forms with text boxes and a Jenkins file.  The master script may want to follow this model. 
4. Script should be able to distribute the virtual users to mimic live usage.  In real life there will be a percentage of librarians checking in books for their patrons, a handful of librarians ordering books, a few of them exporting the items' bibliographic data, etc.. The script should be configured to match real life's distribution of usage on the system.
5. Either built into the master script or Jenkins job as a step or implemented as a manual step, there should be a process to restore the database by way(s) of cleaning up the records that were created or altered so that the test can be rerun from the same starting point. 

Considerations

1. Variables
   1. Distribute variables into classes
      1. class: Main: Okapi Host, Username, Tenant
      2. class: Main (workflows specific): edge-host, API Keys
      3. class: Load: User, duration, ramp up <-- workflow specific
         1. Long vs. short durations 
         2. Big jobs vs. time-bound jobs.
      4. class: Flow-based: profile, file  <-- workflow specific
   2. Passing in variables from Jenkins job. 
      1. Group the like workflows' variables into separate groups and assigning probabilities in each group
      2. JSON configuration that has configurations for each workflow
         1. Pros: Does not need to create a long list of parameters in the Jenkins job. 
         2. Cons: Long file, hard to work with.
      3. Store the workflow configuration file on Github (pull it down when building the job)
         1. Pros:  Changes to the config are documented as Github as commit comment each time the file is checked in.
         2. Cons: The configuration still needs to be in some form (JSON, XML, plaint text/csv, etc..)
      4. Store the workflow config file in Artifact package
         1. Pros:
         2. Cons: 
            1. If a slight change to one of the parameters is made, a new artifact package will have to be recreated. 
            2. Need to keep record of the versions of the packages that have whichever changes. 
      5. Expose the parameters directly in Jenkins as text boxes and other controls (drop down lists, check boxes, etc..) 
         1. Pros:
            1. Can be configurable with JSON file in one field
            2. Jenkins could store default values for parameters.
         2. Cons: 
            1. A very long Jenkins job that may have well over 200 parameters (approximately 4-5 parameters for each workflow + up to 15 general parameters).
2. Probability of calls (TBD - with small POCs)
   1. Flow based?
   2. tenant/cluster based - to control the distribution of calls to the workflows. 
   3. Implementation:
      1. Smaller thread groups easier to manage, debug
                 How: New thread group for each workflow
                 Pros: 
                 Cons:
      2. One thread group
                 How: Extremely hard to combine them in one thread
                 Pros:
                 Cons:
         1. 1. Hard to manage all in one thread
            2. There are too many lines with transaction controllers in one thread.
            3. We have different workflows for some of them load is configured with number of users and for others with the size of processed file, thus it is extremely hard to combine them in one thread.
3. How to retrieve secured variables like username, password?
4. Automation: Any modifications to the current Jenkins job needed?  
5. Script size: How big is too big? Can it reasonably accommodate 30-40 workflows?
6. Test results monitoring:
   1. Do we need to create new Grafana dashboard?
   2. Will we have automated reporting?
   3. How will we take workflow process duration like Data Import/Export duration from the database?