Attendees:

Off-Hours Guidelines

D4. Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.

Since last week:

• Yogesh Kumar added them to the wiki: Off-hour Environment Downscale, with edits from last week
• Maccabee Levine updated the required information for a new environment to reference it.

Today:

• Good?
• Should release environments also have off-hours?  (From Kitfox sprint review: Peter Murray discussed bringing to PC and TC whether we bring the same weekend suspension to the hosted release environments.  Peter will look at what savings that would be.)

Budgets / Cost Anomaly Detection

D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates

D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations

Last time:

• How would teams estimate costs?  AWS Calculator lets you put together a set of AWS services, create a link to share estimate.  But you have to know what you are asking for.  X size databases, open search, etc.
  • All teams under folio-dev: Projects(Namespaces).  Hard for teams to do the calculation / effort.
  • Consensus: calculator may be too complicated, better for creating an entirely new cluster.  Never mind.
  • Standard enviorment is composed of X copmonents.  Can estimate that cost.  If they need to load 100 million records into the DB, that will cost more.  
  • Yogesh Kumar ask Kitfox to determine standard environment cost, i.e. Bama.  Maybe a price list, where the standard environment uses X resources, has X monthly cost.  Team may need to increase size of DB, so higher proportion of cost. 
  • Or simplify.  Maybe just two "recipes", and what the monthly cost was for those recipes.  "Standard" and "Premium".  Or "Dev" vs "Testing".  Start from there.
• ACRG periodically review actuals vs. budget?  Annual and each flower release?
  • For annual budget, need to look at roadmap, features planning to implement, how many dev teams.  Based on prior year's actuals, estimate next year's budget.  Given inflation and AWS pricing changes, maybe a 10% increase if all else stable. 
    • Repeat the eval after each flower release.
    • Let CC know updated estimates for total costs.
  • Could break into flower release, and budget per team.  But maybe not needed per team at that stage.
• Anomaly detection and cost budget alerts?
  • Discuss next week.

Today:

• Review new section on budget page, "Review by AWS Cost Review Group".  Accurate?
• Review draft environment "price list" / "recipes" from Yogesh Kumar if ready.
• What work has to be done to the budgets?
• to the budget alerts?
• to the anomaly detection?
• to the rightsizing recommendations?

Reviewing Environments to Shut Down

D2. Define a process for reviewing existing tools and environments for candidates to be shut down (e.g. when a team leaves the project or the env is no longer needed)

Last time / Prior:

• Mark Veksler draft guidelines on who should have permissions to what operations in AWS.  What will each team be allowed to do.  Link from ACRG doc.
• Didn't get to this.

Today:

Future Topics

• AWS Environments that are not for dev teams.  Do we apply similar processes?  (If so how?)
• Off-hours shutdown during weekday evenings
• Environment requests for the existing environments when the new procedures are approved