DRAFT in progress by TC AWS Costs Subgroup. To be reviewed by TC, CC.
| Table of Contents |
|---|
Jira Project for AWS Environments
- Project: AWS Costs (Key: COSTS)
- Issue Type: AWS EnvironmentStatus values:Draft
- Submitted
- Approved
- Denied
- Active
- In Review
- Shut Down
| Drawio | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
- Anyone can create a Draft ticket in the Jira project.
- A Dev Team Representative (up to the team – for example the Dev Lead, PO, Scrum Master or QA Lead) can change status from Draft to Submitted.
- The Dev Team Representative should be the person who will be available to answer questions.
- Should be submitted at least two weeks before it’s needed. Ideally four.
- See required information.
- AWS Cost Review Group can change status from Submitted to Approved or Denied
- See approval process.
- Dev team can change status from Approved to Active
- See activation process.
AWS Environment Request: Required Information in
...
the Jira ticket
As with the New Module Evaluation Process, the onus us on development teams to fill out a template with required information, hopefully fixing any issues & addressing concerns before the request is actually submitted. The benefit then to a team is that then the actual approval should be as easy & quick as possible. The required information should be sufficient, but no more than necessary, to make that (easy, quick) decision.
...
- Expand on the purpose as necessary. Why is this new environment needed?
- Are there any existing environments serving the same team and/or modules? If so, explain why they are not sufficient for this need.
- Cost estimates: a) monthly and b) total for the life of the environment.
- Are there any past (or still existing) environments that were used in a similar manner, such that this environment would likely have similar monthly costs?
- Mark Veksler suggested: Maybe want to develop a calculator? Or an excel spreadsheet? Including all of the AWS components involved. Yogesh Kumar Peter Murray ? Make as simple as possible for development teamReference Namespace configurations costs estimation as a guide.
- Impact / risks if the request is declined.
Cost Management Plan
- Has the dev team reviewed the Off-hour Environment Downscale policy? Will the dev team be following the “pausing/stopping” guidelines? Explain any planned divergence from those guidelines.
- Refers to “D4: Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.”
- Specifically what will be the expected operating hours? Or will it need to run 24/7?
- Note: If the dev team wants a different (same length) weekend shut down period, they should submit a RANCHER ticket, as this does not affect the budget.
Approval Process
AWS Cost Review Group is notified when Jira items moved to Submitted
...
The ACRG should then:
- Within two weeks: either approve, deny, or ask questions on the Jira.
- PO and Dev Lead expected to respond to any questions, meet to discuss synchronously if requested.
- Approval Checklist
- All Basic Info fields submitted?
- All Justification questions addressed?
- All Cost Management Plan questions addressed?
- All answers reasonable?
- Determine whether the purchase of reserved instances is warranted based on the length of time the environment is expected to exist. Reserved instances can be purchased for dedicated RDS and OpenSearch instances.
- Once approved, notify Treasurer (how?) to be aware of potential impact.
- If denying the request, explain in writing and notify Community Council.
...
- The Dev Team Representative should submit an updated version of the AWS Environment Request, within the same Jira as the initial request, and change the status to In Review.
- The AWS Cost Review Group should then repeat the Approval Process, again within two weeks, including any necessary discussion.
- Once re-approved, the AWS Cost Review Group should change status back to Active.
- Before denying a renewal request, escalate the issue to a Community Council discussion / decision. (This should require more oversight than denying an initial request, since it would interrupt existing work to shut down an environment.) If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.
No Longer Needed
If an environment is no longer needed, or not need longer needed permanently
- The Dev Team should request that Kitfox shut down the environment
If an environment should be shut down on a certain upcoming date,on
- The Dev Team should request that Kitfox shut down team designated member can stop the environment on the appropriate date.with the self-serve Jenkins job documented at How to start and activate Rancher environment
Monitoring Existing Environments
Two weeks after each Flower Release, the AWS Cost Review Group should review all Active environments to identify any that have exceeded their latest updated Environment Request, i.e.
...
- AWS Cost Review Group should request the dev team submit an updated request within two weeks.
- In the unlikely event that the Dev Team is unresponsive after that time period, the AWS Cost Review Group should escalate the issue to Community Council. If ultimately necessary, the AWS Cost Review Group can then request that Kitfox shut down the environment.
Environments that Predate this Process
When the ACRG first forms, it should ask the dev teams responsible for existing environments to submit an Environment Request for each environment by the next Flower Release.
Shutting Down an Environment
...
- Change the Jira status to Shut Down.
Reporting on Active Environments
Need reports that includes the three budget points for each environment: end date, total cost, total monthly cost. May need to work with Kitfox to have a breakdown by team.
Mark Veksler you suggested this on 4/28 and that Yogesh Kumar might lead this. Would one of you please define more of the specifics here? A few questions I'm thinking of:
- Is this scheduled push (i.e. Kitfox generates the reports for the ACRG and dev teams every month) or on request?
- How can this be done to be as little work as possible for Kitfox? I.e. hopefully some initial setup and then just press a button (if that) to refresh the reports?