Jira Project for AWS Environments

• Project: AWS Costs (Key: COSTS)
• Issue Type: AWS Environment

Requesting a New Environment

• Anyone can create a Draft ticket in the Jira project.
• A Dev Team Representative (up to the team – for example the Dev Lead, PO, Scrum Master or QA Lead) can change status from Draft to Submitted.
  • The Dev Team Representative should be the person who will be available to answer questions.
  • Should be submitted at least two weeks before it’s needed.  Ideally four.
  • See required information.
• AWS Cost Review Group can change status from Submitted to Approved or Denied
  • See approval process.
• Dev team can change status from Approved to Active
  • See activation process.

AWS Environment Request: Required Information in the Jira ticket

As with the New Module Evaluation Process, the onus us on development teams to fill out a template with required information, hopefully fixing any issues & addressing concerns before the request is actually submitted.  The benefit then to a team is that then the actual approval should be as easy & quick as possible.   The required information should be sufficient, but no more than necessary, to make that (easy, quick) decision.

Basic Info (one-liners)

1. Dev Team.  Who will be using the environment?
2. Purpose.  Specific feature(s) being tested?
3. Modules.  Primary module(s) related to the purpose of the environment.
4. Expected Start Date
5. Expected End Date
6. Data Set / Size

Justification

1. Expand on the purpose as necessary.  Why is this new environment needed?
2. Are there any existing environments serving the same team and/or modules?  If so, explain why they are not sufficient for this need.
3. Cost estimates: a) monthly and b) total for the life of the environment.  
   1. Reference Namespace configurations costs estimation as a guide.
4. Impact / risks if the request is declined.

Cost Management Plan

1. Has the dev team reviewed the Off-hour Environment Downscale policy?  Will the dev team be requesting a shorter weekend shutdown period than indicated by the Shutdown Schedule?
   1. Note: If the dev team wants a different (same length) weekend shut down period, they should submit a RANCHER ticket, as this does not affect the budget.

Approval Process

AWS Cost Review Group is notified when Jira items moved to Submitted

The ACRG should then:

• Within two weeks: either approve, deny, or ask questions on the Jira.
  • PO and Dev Lead expected to respond to any questions, meet to discuss synchronously if requested.
• Approval Checklist
  • All Basic Info fields submitted?
  • All Justification questions addressed?
  • All Cost Management Plan questions addressed?
  • All answers reasonable?
• Determine whether the purchase of reserved instances is warranted based on the length of time the environment is expected to exist. Reserved instances can be purchased for dedicated RDS and OpenSearch instances.
• Once approved, notify Treasurer (how?) to be aware of potential impact.
• If denying the request, explain in writing and notify Community Council.

Activation Process

• Kitfox creates the environment.
• Dev Lead / PO changes status to Active before (or as soon as) the environment is actually used, i.e. incurring non-trivial costs.
• Within one week of changing to Active, dev team is expected to have Cost Management Plan implemented / active.

Dev Team Review / Extending an Environment

Dev Teams should regularly review (at least monthly, or ideally during each sprint planning) whether their currently Active environments are still needed.

Still Needed

If an environment is still needed, the Dev Team should review whether it is currently exceeding (or likely to soon exceed) any of the details submitted in the original request:

• Expected End Date
• Monthly Cost Estimate
• Total Cost Estimate

If so:

• The Dev Team Representative should submit an updated version of the AWS Environment Request, within the same Jira as the initial request, and change the status to In Review.
• The AWS Cost Review Group should then repeat the Approval Process, again within two weeks, including any necessary discussion.
  • Once re-approved, the AWS Cost Review Group should change status back to Active.
  • Before denying a renewal request, escalate the issue to a Community Council discussion / decision.  (This should require more oversight than denying an initial request, since it would interrupt existing work to shut down an environment.)  If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.

No Longer Needed

If an environment is not need longer needed permanently

• The Dev Team should request that Kitfox shut down the environment

If an environment should be shut down on a certain upcoming date,on

• The Dev team designated member can stop the environment with the self-serve Jenkins job documented at How to start and activate Rancher environment

Monitoring Existing Environments

Two weeks after each Flower Release, the AWS Cost Review Group should review all Active environments to identify any that have exceeded their latest updated Environment Request, i.e.

• The End Date is past, or
• The Total Cost is over the estimate by > 10%, or
• The Monthly Cost for any recent month is over the estimate by > 10%.

Ideally this should never identify any environments, since the Dev Team would have submitted an updated request and changed the status to In Review.

However if it does:

• AWS Cost Review Group should request the dev team submit an updated request within two weeks.
• In the unlikely event that the Dev Team is unresponsive after that time period, the AWS Cost Review Group should escalate the issue to Community Council.  If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.

Environments that Predate this Process

When the ACRG first forms, it should ask the dev teams responsible for existing environments to submit an Environment Request for each environment by the next Flower Release.

Shutting Down an Environment

Kitfox can use its technical discretion on how to "shut down" an environment, such as just setting the number of tasks to zero, as long as it's no longer incurring non-trivial costs.

When Kitfox shuts down an environment Kitfox should also:

• Change the Jira status to Shut Down.