DRAFT in progress by TC AWS Costs Subgroup. To be reviewed by TC, CC.
| Table of Contents |
|---|
Jira Project for AWS Environments
- Project: AWS Costs (Key: COSTS)
- Issue Type: AWS Environment
- Status values:
- Draft
- Submitted
- Approved
- Denied
- Active
- In Review
- Shut Down
| Drawio | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
- Anyone can create a Draft ticket in the Jira project.
- A Dev Team Representative (up to the team – for example the Dev Lead, PO, Scrum Master or QA Lead) can change status from Draft to Submitted.
- The Dev Team Representative should be the person who will be available to answer questions.
- Should be submitted at least two weeks before it’s needed. Ideally four.
- See required information.
- AWS Cost Review Group can change status from Submitted to Approved or Denied
- See approval process.
- Kitfox or dev team (?) Dev team can change status from Approved to Active
- See activation process.
AWS Environment Request: Required Information in
...
the Jira ticket
As with the New Module Evaluation Process, the onus us on development teams to fill out a template with required information, hopefully fixing any issues & addressing concerns before the request is actually submitted. The benefit then to a team is that then the actual approval should be as easy & quick as possible. The required information should be sufficient, but no more than necessary, to make that (easy, quick) decision.
...
- Expand on the purpose as necessary. Why is this new environment needed?
- Are there any existing environments serving the same team and/or modules? If so, explain why they are not sufficient for this need.
- Cost estimates: a) monthly and b) total for the life of the environment.
- Are there any past (or still existing) environments that were used in a similar manner, such that this environment would likely have similar monthly costs?Reference Namespace configurations costs estimation as a guide.
- Impact / risks if the request is declined.
Cost Management Plan
- Has the dev team reviewed the Off-hour Environment Downscale policy? Will the dev team be following the “pausing/stopping” guidelines? Explain any planned divergence from those guidelines.
- Refers to “D4: Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.”
- Specifically what will be the expected operating hours? Or will it need to run 24/7?
- Note: If the dev team wants a different (same length) weekend shut down period, they should submit a RANCHER ticket, as this does not affect the budget.
Approval Process
AWS Cost Review Group is notified when Jira items moved to Submitted
...
The ACRG should then:
- Within two weeks: either approve, deny, or ask questions on the Jira.
- PO and Dev Lead expected to respond to any questions, meet to discuss synchronously if requested.
- Approval Checklist
- All Basic Info fields submitted?
- All Justification questions addressed?
- All Cost Management Plan questions addressed?
- All answers reasonable?
- Determine whether the purchase of reserved instances is warranted based on the length of time the environment is expected to exist. Reserved instances can be purchased for dedicated RDS and OpenSearch instances.
- Once approved, notify Treasurer (how?) to be aware of potential impact.
- If denying the request, explain in writing and notify Community Council.
Activation Process
- Kitfox creates the environment.
- Dev Lead / PO changes status to Active before (or as soon as) the environment is actually used, i.e. incurring non-trivial costs.
- Within one week of changing to Active, dev team is expected to have Cost Management Plan implemented / active.
...
- The Dev Team Representative should submit an updated version of the AWS Environment Request, within the same Jira as the initial request, and change the status to In Review.
- The AWS Cost Review Group should then repeat the Approval Process, again within two weeks, including any necessary discussion.
- Once re-approved, the AWS Cost Review Group should change status back to Active.
- Before denying a renewal request, escalate the issue to a Community Council discussion / decision. (This should require more oversight than denying an initial request, since it would interrupt existing work to shut down an environment.) If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment. (Is there a "suspend" option in AWS?)
No Longer Needed
If an environment is no longer needed, or not need longer needed permanently
- The Dev Team should request that Kitfox shut down the environment
If an environment should be shut down on a certain upcoming date,on
- The Dev Team should request that Kitfox shut down team designated member can stop the environment on the appropriate date.with the self-serve Jenkins job documented at Manage start-stop-project Jenkins job
Monitoring Existing Environments
Two weeks after each Flower Release, the AWS Cost Review Group should review all Active environments to identify any that have exceeded their latest updated Environment Request, i.e.
...
- AWS Cost Review Group should request the dev team submit an updated request within two weeks.
- In the unlikely event that the Dev Team is unresponsive after that time period, the AWS Cost Review Group should escalate the issue to Community Council. If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment. (Is there a "suspend" option in AWS?)
Environments that Predate this Process
When the ACRG first forms, it should ask the dev teams responsible for existing environments to submit an Environment Request for each environment by the next Flower Release.
Shutting Down an Environment
Kitfox can use its technical discretion on how to "shut down" an environment, such as just setting the number of tasks to zero, as long as it's no longer incurring non-trivial costs.
When Kitfox shuts down an environment (or does the Dev Team do that?), Kitfox should also:
- Change the Jira status to Shut Down.
...