...

• Project: AWS Costs (Key: COSTS)
• Issue Type: AWS Environment
• Status values:
  • Draft
  • Submitted
  • Approved
  • Denied
  • Active
  • In Review (periodic?)
  • Shut Down

...

• Anyone can create a Draft.
• A Dev Team Representative (up to the team – for example the Dev Lead, PO, Scrum Master or QA Lead) can change status from Draft to Submitted.
  • The Dev Team Representative should be the person who will be available to answer questions.
  • Should be submitted at least two weeks before it’s needed.  Ideally four.
  • See required information.
• AWS Cost Review Group can change status from Submitted to Approved or Denied
  • See approval process.
• Kitfox or dev team (?) can change status from Approved to Active
  • See activation process.

Monitoring Existing Environments

• TBD, but perhaps use the In Review cycle periodically, and eventually Shut Down.

Required Information in a New Environment Request

Basic Info (one-liners)

1. Dev Team.  Who will be using the environment?
2. Purpose.  Specific feature(s) being tested?
3. Modules.  Primary module(s) related to the purpose of the environment.
4. Expected Start Date
5. Expected End Date
6. Data Set / Size. 

Justification

1. Expand on the purpose as necessary.  Why is this new environment needed?
2. Are there any existing environments serving the same team and/or modules?  If so, explain why they are not sufficient for this need.
3. Cost estimates: a) monthly and b) total for the life of the environment.  
   1. Are there any past (or still existing) environments that were used in a similar manner, such that this environment would likely have similar monthly costs?
4. Impact / risks if the request is declined.

Cost Management Plan

1. Will the dev team be following the “pausing/stopping” guidelines?  Explain any planned divergence from those guidelines.
   • Refers to  “D4: Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.”
   • Specifically what will be the expected operating hours?  Or will it need to run 24/7?
2. What budgets, budget alerts, cost anomaly detection will be used?
   • Refers to “D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates” and “D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations”

Approval Process

• AWS Cost Review Group notified when Jira items moved to Submitted (how?)
• Within two weeks: either approve, deny, or ask questions on the Jira.
• PO and Dev Lead expected to respond to any questions, meet to discuss synchronously if requested.
• Approval Checklist
  • Reasonable answers?
  • Reasonable answers?
• All required fields submitted?
• All Justification questions addressed?
• All Cost Management Plan questions addressed?
• Once approved, notify Treasurer (how?) to be aware of potential impact.
• If denying the request, explain in writing and notify Community Council.

Activation Process

• Dev Lead / PO changes status to Active before (or as soon as) the environment is actually used, i.e. incurring non-trivial costs.
• Within one week of changing to Active, dev team is expected to have Cost Management Plan implemented / active.

Dev Team Review / Extending an Environment

• Dev Teams should regularly review (at least monthly, or ideally during each sprint planning) whether their currently Active environments are still needed

Still Needed

If an environment is still needed, the Dev Team should review whether it is currently exceeding (or likely to soon exceed) any of the details submitted in the original request:

• Expected End Date
• Monthly Cost Estimate
• Total Cost Estimate

If so:

• The Dev Team Representative should submit an updated version of the New Environment Request, within the same Jira as the initial request, and change the status to In Review.
• The AWS Cost Review Group should then repeat the Approval Process, again within two weeks, including any necessary discussion.
  • Once re-approved, the AWS Cost Review Group should change status back to Active.
  • Before denying a renewal request, escalate the issue to a Community Council discussion / decision.  (This should require more oversight than denying an initial request, since it would interrupt existing work to shut down an environment.)  If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.  (Is there a "suspend" option in AWS?)

No Longer Needed

If an environment is no longer needed, or should be shut down on a certain upcoming date,

• The Dev Team should request that Kitfox shut down the environment on the appropriate date.

Monitoring Existing Environments

• Two weeks after each Flower Release, AWS Cost Review Group should review all Active environments to identify any that are a) not already under an extension review and b) have exceeded their latest updated Environment Request, i.e.
  • The End Date is past, or
  • The Total Cost is over the estimate by > 10%, or
  • The Monthly Cost for any recent month is over the estimate by > 10%.
• Ideally this will never happen, since the Dev Team should have submitted an updated request.  However if it does:
  • AWS Cost Review Group should request the dev team submit an updated request within two weeks.
  • In the unlikely event that the Dev Team is unresponsive after that time period, the AWS Cost Review Group should escalate the issue to Community Council.  If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.  (Is there a "suspend" option in AWS?)

Shutting Down an Environment

• When Kitfox shuts down an environment (or does the Dev Team do that?), Kitfox should also:
  • Change the Jira status to Shut Down.