2023-04-07 Meeting Notes

Owned by Maccabee Levine
Apr 07, 2023

Attendees:

Review of original deliverables (D1 - D6) – do they still make sense?

  • All still make sense
  • Discuss how and who might accomplish them:

D1. Define a process for requesting, reviewing, approving, and documenting new tools and FOLIO environments as well as the addition of new AWS services to the project

  • Requests originating?  Typically dev teams.  Driven by product owner, or team lead.  I.e. product team may request addl env for UAT.  Dennis Bridges working on a new feature for acquisitions, wants to do month-long UAT.  need to know start & end time.  What data set, to size environment appropriately.  
    • Bug fix team may request additional environment to test feature.  Performance testing bulk edit.  Remember to shut down when done.
    • Ideally requests come in the form of Jiras, to track.  
    • Hold teams responsible for managing those environments and the costs.
    • Probably a new Jira project for managing costs.  New environment would be one type.  Buying a new tool, subscribing to a service.
  • Approval / review should be expedient.
    • If we have regular check-ins every 2-4 weeks.  Teams should know well in advance when they will need a new env.  PO thinking about UAT, not done the day before.  Submit the request ahead of time.  Give them a sprint or two lead time.
    • Can deal with urgent requests for production issues in Jira.  Either reconvene or communicate async.
    • Should we develop a checklist for approve/deny?  Mark: should have some guidelines but not go overboard.  If it’s a serious production issue (maybe use same definitions as support group, P1), then lean to approve.
    • Ask team to estimate costs.
    • Peter: keep as lightweight as possible.  Helped project move fast, don’t want to clamp down.
    • Mark: if env needs are huge, still approve but raise to treasurer that we’re going on track?
  • Maccabee Levine will draft from above

D2. Define a process for reviewing existing tools and environments for candidates to be shut down (e.g. when a team leaves the project or the env is no longer needed)

  • TBD

D3. Identify who is responsible for each part of these processes and what reporting requirements are needed

  • TBD

D4. Define guidelines/best practices around pausing/stopping environments when they're not in use - e.g. off-hours/weekends/etc.

  • Mark: Kitfox is developing jenkins jobs to shutdown and restart.  With start and end parameters.  Start with simple timers, see how far it gets us.
    • For EBSCO customer environments, even shutting down idle connections.
    • EBSCO paying for some tools to monitor costs, allocate budget per team and let them manage it, just monitor that they stay within accepted parameters.

D5. Create AWS Budgets and AWS Budget Alerts for daily and monthly spend rates

& 

D6. Explore AWS Cost Anomaly Detection and Rightsizing Recommendations

  • Kitfox spikes?
  • Peter: some work has been on budget alerts.  Reserve instance coverage.  Savings by purchasing in advance.  Avoid runaway costs eating into budget.
  • Could use expertise to do what we can with alerts & budgets.
  • AWS provides these tools.  Don’t think either are configured for the FOLIO project.  Cost anomaly might be.  Recommendations: AWS makes recommendations  Making sure thos ethings are configured, monitoring them.  Some group taking into account what those alerts / recommendations are teling sin.
  • Mark: forwarded a spreadsheet from Kitfox of a cost breakdown.  They are reporting in each spring review.  Taking cost optimization steps.
  • Mark: Kitfox already doing 4, 5, 6.  Just need more transparency * visibility.
  • Mark Veksler can write up for 4, 5,6 what wer’e doing and how we’re being transparent about.  He will ask them to create user stories to address those.
    • This subgroup becoming stakeholders to those presentations, comment on it from cost management perspective.
  • Maccabee proposed us recommenditng a standing cost review subgroup with CC and TC representation, or at least some communication.  To be the stakeholders.
  • Peter Murray  will inventory the state of things as they are now.  The extent to which he took it over the years.  As basis for deciding what & who will carry forward.