2023-05-05 Meeting Notes

Owned by Maccabee Levine
Last updated: May 05, 2023

Attendees:

Off-Hours Guidelines

D4. Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.

Last time:

Today:

  • Reviewed document.
  • Permanent different hours (i.e. Australia)?  Adjusted.
  • Self-service tool should only allow startup if the team themselves shut it down?  Yogesh Kumar maybe add controls?  Peter Murray maybe add notifications on a channel, so if we recognize a team is doing it repeatedly, we can inquire.  Yogesh Kumar will consider this logging as well.
  • Email?  Jira instead.  Yogesh Kumar will change.
  • Yogesh Kumar put on the wiki

Budgets / Cost Anomaly Detection

D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates

D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations

Last time:

  • Are there Jiras for these yet?  Sprints identified?
  • Mark Veksler Peter was setting up anomaly detection?  Set up the budgets in AWS Cost Explorer. 
  • At some point, will have to assign budget to each team.  How do we do that?  Maybe need a spike to do that.  Yogesh can create Jiras, and work with the team to prioritize those things.
  • AWS Cost Review Group should periodically review the actuals vs. budget.  And review the budget on a different cadence, maybe around each flower release.  And revising it annually (strategic). 
    • Maccabee Levine asked: Are you suggesting the group just estimate from the current active environments (which should be doable -- each one will have a monthly budget, and end date, listed in the environment request)?  But how would they take into account any new requests that might crop up later in the year?  I imagine there would be a lot of guess work going on, but maybe you have a metric in mind?
    • Yogesh Kumar noted: We could use the historical new requests data to determine what's needed. Would it work to chat about it on Friday? I'd appreciate more context so I can better understand the request.
  • How would the dev team estimate costs.  Maybe want to develop a calculator?  Or an excel spreadsheet?  Including all of the AWS components involved.  Yogesh Kumar or Peter Murray ?  Make as simple as possible for development team.
  • Mark Veksler or Yogesh Kumar Need reports that includes the three budget points for each environment: end date, total cost, total monthly cost.  May need to work with Kitfox to have a breakdown by team.

    • Mark Veksler you suggested this on 4/28 and that Yogesh Kumar might lead this.  Would one of you please define more of the specifics here?  A few questions I'm thinking of:

      • Is this scheduled push (i.e. Kitfox generates the reports for the ACRG and dev teams every month) or on request?
      • How can this be done to be as little work as possible for Kitfox?  I.e. hopefully some initial setup and then just press a button (if that) to refresh the reports?
      • Who is it for?

Today:

  • How would teams estimate costs?  AWS Calculator lets you put together a set of AWS services, create a link to share estimate.  But you have to know what you are asking for.  X size databases, open search, etc.
    • All teams under folio-dev: Projects(Namespaces).  Hard for teams to do the calculation / effort.
    • Consensus: calculator may be too complicated, better for creating an entirely new cluster.  Never mind.
    • Standard enviorment is composed of X copmonents.  Can estimate that cost.  If they need to load 100 million records into the DB, that will cost more.  
    • Yogesh Kumar ask Kitfox to determine standard environment cost, i.e. Bama.  Maybe a price list, where the standard environment uses X resources, has X monthly cost.  Team may need to increase size of DB, so higher proportion of cost. 
    • Or simplify.  Maybe just two "recipes", and what the monthly cost was for those recipes.  "Standard" and "Premium".  Or "Dev" vs "Testing".  Start from there.
  • ACRG periodically review actuals vs. budget?  Annual and each flower release?
    • For annual budget, need to look at roadmap, features planning to implement, how many dev teams.  Based on prior year's actuals, estimate next year's budget.  Given inflation and AWS pricing changes, maybe a 10% increase if all else stable. 
      • Repeat the eval after each flower release.
      • Let CC know updated estimates for total costs.
    • Could break into flower release, and budget per team.  But maybe not needed per team at that stage.
  • Budget reporting?
    • Team-specific budgeting may not be necessary initially.  Consensus: to be determined in the future.  If asked by the financial people, then figure out what reports need to be created.
    • Budget alerts (monthly) do matter.
  • Anomaly detection and cost budget alerts?
    • Discuss next week.

Reviewing Environments to Shut Down

D2. Define a process for reviewing existing tools and environments for candidates to be shut down (e.g. when a team leaves the project or the env is no longer needed)

Last time:

  • Mark Veksler draft guidelines on who should have permissions to what operations in AWS.  What will each team be allowed to do.  Link from ACRG doc.

Today:

  • Didn't get to this.

Future Topics

  • Off-hours shutdown during weekday evenings
  • Environment requests for the existing environments when the new procedures are approved