2023-04-14 Meeting Notes

Owned by Maccabee Levine
Last updated: Apr 14, 2023
3 min read

Attendees:


Agenda:

Peter Murray Review where things are now

  • Described AWS Budgets, Budget Alerts, Cost Anomaly Detection, and Rightsizing Recommendations
  • Budgets / Alerts
    • "Non-covered" costs are things that you can't make a plan to reduce costs
    • Long-term contracts are for one-year duration, whether you use it or not.  Break-even depends on usage, Open Search is around 70%.
    • Can set overall budgets and for individual components.
  • Cost Anomaly Detection
  • Rightsizing Recommendations
    • No alerts – you have to look for the recommendations.
  • Peter has been paying attention to the Slack channel, adjusting as needed.
  • Kubecost.   We have a free license thanks to Peter.  Look at Kubernetes costs, how much each team is spending.
    • Hard to split costs between teams for shared resources.  Simplest is to do an even divide.  Better to split on usage.
  • what kit fox accomplished in the prior sprint: - reduced the number of instances (from 16 -> 5) - switched to mem-optimized instance types - reduced # of availability zones from 4 to 2
  • Is there a list of hosted environments?  Mark Veksler will double-check with Kitfox.  Rancher environments, plus snapshot, jenkins, etc.

Mark Veksler Discuss plans for D4, D5, D6

D4. Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.

D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates

D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations

  • Mark Veksler Third party tools.  We have to pay for them.  They may have options not to charge OSS, but not sure.  Aptia "Cloud Ability"? allows you to set up additional alerts and reports.
    • Ideally move to unit economics, per tenant or per team.
  • Mark Veksler There are ways to shut down idle environments.  Or rely on serverless option for DB to scale up/down based on usage.
  • Mark has asked the team to add these three items to their backlog.  Mark Veksler Which sprint?  
  • Teams will start shutting down the environments on evenings and weekends.  Implementing those Jenkins jobs.
    • First technical aspect.  Then finding out what are the least used hours.
    • Also allowing the environments to be spun up on demand if weekend use etc.
  • Consensus that we should wait for them to establish these practices, and then document them.

Maccabee Levine Discuss draft of D1

D1. Define a process for requesting, reviewing, approving, and documenting new tools and FOLIO environments as well as the addition of new AWS services to the project

  • Draft: https://docs.google.com/document/d/1Q2m39Vw7L3eNDhEHDX1nGQzRtDogOd-UkrfTfmLdHoc/edit#
  • Periodic review?  Discuss for D2.
  • Ok for anyone to create the draft.  A team's representative (Dev Lead, PO or Scrum Master, QA Lead) can move to Submitted.
    • The team representative should also be the one available to answer questions.
  • Information required.  Ok as-is.
    • End date: just use a date.  If they want it for a flower release, they can look at what that date would be.
    • Ask whether it needs run continuously, or what idle hours will be.
  • Should budgets, budget alerts be set up on an individual environment basis?
    • AWS lets you tag each resource with a name/value pair.  Could create a budget (and alert) for that tag.  See if budget intrinsic cost is an issue.
  • Agree to run by CC after we spec out process.
  • Activation process ok.
  • AWS Cost Review Group
    • RACI chart - responsibility assignment matrix.  different groups, who is responsible (doing the work), accountable (for others' work), consulted, informed.
    • Maybe this four group for 6m, with council ok.
  • Maccabee Levine revise and put on wiki