DRAFT in progress by TC AWS Costs Subgroup.  To be reviewed by TC, CC.

Jira

• Project: AWS Costs (Key: COSTS)
• Issue Type: AWS Environment
• Status values:
  • Draft
  • Submitted
  • Approved
  • Denied
  • Active
  • In Review
  • Shut Down

Requesting a New Environment

• Anyone can create a Draft.
• A Dev Team Representative (up to the team – for example the Dev Lead, PO, Scrum Master or QA Lead) can change status from Draft to Submitted.
  • The Dev Team Representative should be the person who will be available to answer questions.
  • Should be submitted at least two weeks before it’s needed.  Ideally four.
  • See required information.
• AWS Cost Review Group can change status from Submitted to Approved or Denied
  • See approval process.
• Kitfox or dev team (?) can change status from Approved to Active
  • See activation process.

Required Information in an AWS Environment Request

As with the New Module Evaluation Process, the onus us on development teams to fill out a template with required information, hopefully fixing any issues & addressing concerns before the request is actually submitted.  The benefit then to a team is that then the actual approval should be as easy & quick as possible.   The required information should be sufficient, but no more than necessary, to make that (easy, quick) decision.

Basic Info (one-liners)

1. Dev Team.  Who will be using the environment?
2. Purpose.  Specific feature(s) being tested?
3. Modules.  Primary module(s) related to the purpose of the environment.
4. Expected Start Date
5. Expected End Date
6. Data Set / Size

Justification

1. Expand on the purpose as necessary.  Why is this new environment needed?
2. Are there any existing environments serving the same team and/or modules?  If so, explain why they are not sufficient for this need.
3. Cost estimates: a) monthly and b) total for the life of the environment.  
   1. Are there any past (or still existing) environments that were used in a similar manner, such that this environment would likely have similar monthly costs?
4. Impact / risks if the request is declined.

Cost Management Plan

1. Will the dev team be following the “pausing/stopping” guidelines?  Explain any planned divergence from those guidelines.
   • Refers to  “D4: Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.”
   • Specifically what will be the expected operating hours?  Or will it need to run 24/7?
2. What budgets, budget alerts, cost anomaly detection will be used?
   • Refers to “D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates” and “D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations”

Approval Process

• AWS Cost Review Group notified when Jira items moved to Submitted (how?)
• Within two weeks: either approve, deny, or ask questions on the Jira.
  • PO and Dev Lead expected to respond to any questions, meet to discuss synchronously if requested.
• Approval Checklist
  • All Basic Info fields submitted?
  • All Justification questions addressed?
  • All Cost Management Plan questions addressed?
  • All answers reasonable?
• Once approved, notify Treasurer (how?) to be aware of potential impact.
• If denying the request, explain in writing and notify Community Council.

Activation Process

• Dev Lead / PO changes status to Active before (or as soon as) the environment is actually used, i.e. incurring non-trivial costs.
• Within one week of changing to Active, dev team is expected to have Cost Management Plan implemented / active.

Dev Team Review / Extending an Environment

Dev Teams should regularly review (at least monthly, or ideally during each sprint planning) whether their currently Active environments are still needed.

Still Needed

If an environment is still needed, the Dev Team should review whether it is currently exceeding (or likely to soon exceed) any of the details submitted in the original request:

• Expected End Date
• Monthly Cost Estimate
• Total Cost Estimate

If so:

• The Dev Team Representative should submit an updated version of the AWS Environment Request, within the same Jira as the initial request, and change the status to In Review.
• The AWS Cost Review Group should then repeat the Approval Process, again within two weeks, including any necessary discussion.
  • Once re-approved, the AWS Cost Review Group should change status back to Active.
  • Before denying a renewal request, escalate the issue to a Community Council discussion / decision.  (This should require more oversight than denying an initial request, since it would interrupt existing work to shut down an environment.)  If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.  (Is there a "suspend" option in AWS?)

No Longer Needed

If an environment is no longer needed, or should be shut down on a certain upcoming date,

• The Dev Team should request that Kitfox shut down the environment on the appropriate date.

Monitoring Existing Environments

Two weeks after each Flower Release, the AWS Cost Review Group should review all Active environments to identify any that have exceeded their latest updated Environment Request, i.e.

• The End Date is past, or
• The Total Cost is over the estimate by > 10%, or
• The Monthly Cost for any recent month is over the estimate by > 10%.

Ideally this should never identify any environments, since the Dev Team would have submitted an updated request and changed the status to In Review.

However if it does:

• AWS Cost Review Group should request the dev team submit an updated request within two weeks.
• In the unlikely event that the Dev Team is unresponsive after that time period, the AWS Cost Review Group should escalate the issue to Community Council.  If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.  (Is there a "suspend" option in AWS?)

Shutting Down an Environment

When Kitfox shuts down an environment (or does the Dev Team do that?), Kitfox should also:

• Change the Jira status to Shut Down.