2023-04-28 Meeting Notes

Owned by Maccabee Levine
Last updated: Apr 28, 2023
3 min read

Attendees:


Agenda:

Reviewing Environments to Shut Down

D2. Define a process for reviewing existing tools and environments for candidates to be shut down (e.g. when a team leaves the project or the env is no longer needed)

  • Drafted AWS Environment Lifecycle sections on "Dev Team Review / Extending an Environment" and "Monitoring Existing Environments" from discussion last week.
  • Feedback?
    • CRV would work with Yogesh to request shutdown of the environment, and Kitfox would pull the plug.
    • "Shutdown"?  Kitfox can use discretion on setting number of tasks to zero, vs actual shut down.
    • Otherwise looks good.
  • Who should actually start up and shut down the environments – dev teams or Kitfox?  Should dev teams be admins?
    • Kitfox for now.
    • CRG should periodically review the permissions in AWS.
      • Mark Veksler draft guidelines on who should have permissions to what operations in AWS.  What will each team be allowed to do.
  • Need reports that includes the three budget points for each environment: end date, total cost, total monthly cost.  May need to work with Kitfox to have a breakdown by team.  
  • How would the dev team estimate costs.  Maybe want to develop a calculator?  Or an excel spreadsheet?  Including all of the AWS components involved.  Yogesh or Peter Murray ?  Make as simple as possible for development team.

AWS Cost Review Group

D3. Identify who is responsible for each part of these processes and what reporting requirements are needed

  • Re-drafted AWS Cost Review Group from discussion last week.
  • Feedback?
    • Purpose section: add bullet point to reflect budget setting (annual), and adjusting for each of the flower releases.
  • Otherwise ok.  Take a final look next week.

Off-Hours Guidelines

D4. Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.

  • Kitfox completed RANCHER-685 to study solutions.  Wrote up: Downscale modules instances during weekend(night) RANCHER-685
  • Is that doc the "guidelines / best practices"?  Or is there additional work to pick one of the two proposed solutions as the best practice?
    • Minutes last week say "This will take effect from 6PM EST (Friday) until 6:30 PM EST (Sunday) starting from 21st of April." but not in the guidelines?
    • Mark Veksler : document lists the different alternatives.  They chose the Jenkins option, and are now running those jobs.  We can monitor and ensure there are no spikes.
    • Kitfox themselves will set up the hours for each team.  If the dev team needs environments during different hours, they would have to contact Kitfox to spin it up.  Although eventually let the dev team do it themselves.
    • Mark Veksler will invite Yogesh to this meeting.  He will take this input, prioritize it relative to other Kitfox work, make sure it gets done.
  • Where the AWS Environment Request asks "Will the dev team be following (these guidelines) ... Explain any planned divergence from those guidelines"... does that make sense?
    • "Please be aware of the guideslines/practices around night/weekend."  but eventually the teams can do it themselves, so still ask the question.  
  • Mark Veksler will draft guidelines based on those hours.
  • In the future, would be good to have downtime every night during the week as well.

Budgets / Cost Anomaly Detection

D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates

D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations

  • Are there Jiras for these yet?  Sprints identified?
  • Mark Veksler Peter was setting up anomaly detection?  Set up the budgets in AWS Cost Explorer. 
  • At some point, will have to assign budget to each team.  How do we do that?  Maybe need a spike to do that.  Yogesh can create Jiras, and work with the team to prioritize those things.
  • AWS Cost Review Group should periodically review the actuals vs. budget.  And review the budget on a different cadence, maybe around each flower release.  And revising it annually (strategic).