EPAM-EBSCO Team Leads' Forum

Permanent Forum Members:


Meeting Minutes

Meeting December 10


  1. Tjet exploring remote debugging on Rancher; building local UI; starting experimenting with Telepresense
  2. Citation 16GB is not enough
  3. Volaris problems building and running UI modules; trying to figure out problems with running DCB module
  4. Folijet no progress, using deploying to Rancher
  5. Firebird. no progress
  6. Corsair 16 GB limitation as well
  7. Spitfire no need for tools
  8. Vega no progress


Meeting December 3

Attendees: PK, Sasha K, Gurleen, Matt, Ostap, Serhii, Boburbek Kadirkhodjaevlava, Kate, Yogesh, Pavlo, Craig McNally, Martin, Yogesh

Topics: Local development on Eureka platform. Teams' readiness to develop for Eureka and on Eureka

Quick notes

Several Teams have tried the Golang tool and there are the results: 

  1. Tjet (the tool author) can run the Acquisition app locally
  2. Citation tried and got some container errors. PK will troubleshoot with Boburbek
  3. Volaris was able to run some internal modules, but not the DCB (also container errors). Gurleen or her deputee will work with Boburbek
  4. Folijet tried to deploy the Import app locally, but seemingly ran into memory issues. Kate is going to report more results shortly
  5. Firebird. Slava intends to try the Bash tool in the next sprint
  6. Corsair is using Rancher to deal with Eureka issues for now, but going to try the Go tool in the next sprint
  7. Spitfire. Pavlo stated that Spitfire is mostly focusing on developing microservices independent on other modules. When they need to test integration they are doing well with Rancher, so they do not need local Eureka env. 
  8. Vega is still struggling with the Ramsons and Ramsons+ scope. Too soon to even dream about Eureka (smile)


In general, all team leads understand that in Sunflower we will be developing features exclusively on the Eureka platform, and we will touch Okapi only support production (which I'm afraid we're going to do quite a lot judging by the frenzy in which we're wrapping up Ramsons). 

Please remember that the team leads are responsible for creating the most favorable development conditions for your teams. If your team chooses to continue using Rancher as Spitfire does - I respect that, however please do not complain that developing and troubleshooting on Eureka is slow and painful.  



Meeting November 26

Attendees: PK, Sasha K, Sasha H, Gurleen, Matt, Ostap, Serhii, Boburbek Kadirkhodjaevlava, Kate, Yogesh, Olamide, Pavlo, Oleksii Petrenko, Oleksii Kuzminov, Craig McNally, Steve Ellis

Topics: Local development on Eureka platform and tooling for that. In the meeting, we discussed the Go tool developed by Boburbek Kadirkhodjaev and Serhii Nosko

Quick notes

We saw a very impressive presentation of the Go tool. The tool appears to be ready for prime time, but we are all anxious to hear from the users of the tool about their experience working with the tool. The Eureka team is going to record their tool demo later. Also, I scheduled another leads' meeting next week, 12/3, and I invite Eureka to present their solution then. 


Another very important comment: Teams, please do not dump issues in Eureka's lap automatically when something works on Okapi but fails on Eureka. Please do some troubleshooting first. Please remember we're in it together.  This is a link to the Eureka troubleshooting guide: https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/628916278/How+to+triage+permissions+or+403+401+problems+on+Eureka+envs

Comments: 

  1. The ECS module is not loading automatically, so it needs to loaded manually (should not be a big deal, I'm hearing)
  2. From oleksandr_haimanov: can we figure out how to not load some standard components locally, like Kafka and postgres, to reclaim some memory space? See this tool https://www.telepresence.io/#gsc.tab=0

Team's readiness to start using the Go tool. 

Teams ready to get started on the Go tool: Volaris, Thunderjet, Citation, Corsair

Teams ready to get started on the Bash Bootstrap tool: Firebird

Teams not ready to get started on Eureka work: Folijet, Vega

Teams who do not think any tool is necessary for local development on Eureka: Spitfire (Pavlo Smahin: I'm looking forward to hearing about their way of developing applications on the Eureka platform


Links

Meeting Recording: 

https://ebscoind-my.sharepoint.com/personal/lbraginsky_corp_epnet_com/_layouts/15/stream.aspx?id=%2Fpersonal%2Flbraginsky%5Fcorp%5Fepnet%5Fcom%2FDocuments%2FRecordings%2FFolio%20Team%20Leads%27%20Forum%2D20241126%5F074721%2DMeeting%20Recording%2Emp4&referrer=StreamWebApp%2EWeb&referrerScenario=AddressBarCopied%2Eview%2E5b30b39c%2D211d%2D46dc%2Dad88%2D929342d0d282


Serhii/Boburbek presentation

https://epam-my.sharepoint.com/:p:/p/serhii_nosko/EYJ7NLEYsaNJkKHopAVnihUBAIn9kF46wYYebFlsKqH5Iw?e=us3gR1


Eureka's guide to troubleshooting issues

https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/628916278/How+to+triage+permissions+or+403+401+problems+on+Eureka+envs




Meeting October 29

Attendees: PK, Sasha K, Sasha H, Gurleen, Matt, Serhii, Boburbek Kadirkhodjaevlava, Kate, Yogesh

Topic: 1) Discuss Github Copilot tool to improve teams' productivity. Use cases considered included generating unit and Karate tests, code refactoring, unfamiliar code understanding (Python). Hoping to see a live demo next time

2) Discuss LoC support model. How do we allow developers to download LoC logs? How do we mask sensitive data pieces in the application logs? 


Meeting October 8

Attendees: Ostap, PK, Sasha K, Gurleen, Matt, Pavlo, Ostap, Serhii, Boburbek Kadirkhodjaev

Topic: Oleksii spoke about module resource management

Presenter: Oleksii Petrenko


Meeting September 24

Attendees: Ostap, PK, Sasha K, Gurleen, Matt, Pavlo, Ostap, Serhii, Boburbek Kadirkhodjaev

Topic: Serhii and Boburbek present the tool developed by TJet to set up a Eureka environment locally along with remote troubleshooting

PresenterSerhii Nosko, Boburbek

Video Link

Presentation




Meeting August 20, 2024

Attendees: Ostap, PK, Sasha K, Gurleen, Yogesh, Slava, Matt, Pavlo, Ostap, Serhii, Denis Kovtun

Topic: Denis Kovtun's talk about production support, issues in Rally, RRT

Presenter: Denis K

Video Link

Other materials

RRT Info

RRT Best practices

RRT checklist for troubleshooting

RRT troubleshooting a P1


Yogesh's topic Change Logs

Nightly Karate and Cypress run Change logs

Quick Note: 

Please think about how to reduce the number of bugs that end up in Prod (or Bugfest). Improving logs was one suggestions. Resurrecting the RCA program to improve the code area where the issues occur was another. 

Also, please monitor the daily Karate and Cypress runs on the QualityGatesCI Pipeline and address issues immediately. The Change log address is above. 


Meeting August 6, 2024

Attendees: Ostap, Matt, Yogesh, PK, Sasha K, Gurleen, Kate, Sasha H, Martin, Mikita

Topic: Gitlab branching strategy

Presenter: Mikita Siadykh

Presentation link

Video Link

Quick Note: This is coming hopefully this Fall. Looking forward! 



Meeting July 23, 2024

Attendees: Ostap, Matt, Yogesh, PK, Sasha K, Serhii, Gurleen, Kate, Slava, Pavlo, Oleksii Petrenko and Tetiana Gusar

Topic: Reporting Portal and e2e test maintenance by FE engineers

Presenter: Oleksii, Tetiana

Presentation link

Video Link

Quick Note

Reporting Portal link: https://report-portal.ci.folio.org/ui/. Please use your Github creds to log in

Leads - since now we have our Quality Gates pipeline we need to carefully monitor the e2e failure (as well as Karate failures). Please use the reporting portal to see the failures.  Please make sure your FE engineers know what to do. 

Some action items from the meeting (as per chatgpt (smile))


  • Tatiana Gusar:

    • Provide additional training for FE engineers on Cypress test maintenance
    • Ensure Scrum Masters are involved in the troubleshooting process.
    • Communicate with John C and Zach about monitoring Stripes e2e failures
  • All Team Members:

    • Actively participate in the troubleshooting process.
    • Utilize the report portal for tracking and resolving test failures.
  • Lee Braginsky/Yogesh Kumar/Oleksii:

    • Monitor the engagement and progress of the team in reducing bugs.
    • Ensure that all dependencies are addressed and team members are informed.




Meeting July 9, 2024

Attendees: Ostap, Matt, Yogesh, PK, Roman, Sasha K, Serhii, Gurleen, Kate, Slava, 

Topic: general discussion on QualityGates, Karate failures and the priority of fixing issues 




Meeting June 25, 2024

Attendees: Ostap, Matt, Yogesh, PK, Roman, Sasha K, Serhii, Gurleen, Kate, Slava, 

Topic

  1. Discussed various topics including setting up developing envs using BusyBee vs. Vagrant. 
  2. We are ready to include Gatling reports into QualityGateCI pipeline

PresenterSerhii Nosko

Presentation link

Video Link

Quick Note

We are going to include Gatling report into QualityGatesCI pipeline. Yogesh is going to create ticket for Kitfox. Please continue identifying Karate tests to be included into Gatling report

BusyBee seems be a good reliable way to quickly create Eureka local development environment. 



Meeting June 11, 2024

Attendees: Ostap, Matt, Yogesh, PK, Roman, Sasha K, Serhii, Gurleen, Sasha H, Slava, 

Topic

Discussed various topics including setting up developing envs using BusyBee vs. Vagrant, Gatling reports and FE engineers doing Cypress maintenance 

Presenter: all

Quick Note

Serhii Nosko has a script to quickly and reliably set up development env using BusyBee


The Quality Gate CI Pipeline is ready. https://jenkins-aws.indexdata.com/job/folioRancher/job/folioQualityGates/job/folioCiQualityGates/

Next Time: Continue the quality topic


Meeting May 28, 2024

Attendees: Ostap, Matt, Yogesh, PK, Roman, Sasha K, Serhii, Gurleen

Topic

  1. Gurleen presented her "quick wins" with Gatling report. Thank you Gurleen and Team Volaris 
  2. Developers' maintaining e2e tests. Tjet is doing it (I think Fjet as well, but Kate wasn't in the meeting to confirm). Leads - please work with your FE developers and have them work with Ostap to come up to speed on e2e maintenance. 
 

Presenter: all

Quick Note: We are hearing that often POs prioritize tickets related to "quality work", such as Karate tests, Gatling performance report, or e2e maintenance lower than functional features. Dear Captains, this is where you need to negotiate with your POs and explain that this type of work makes the business features more robust. I mean, at this point, nobody questions the need to implement unit tests or integration tests, right? Gatling and e2e maintenance should become as natural as unit-testing...

BTW, the Quality Gate CI Pipeline is ready. https://jenkins-aws.indexdata.com/job/folioRancher/job/folioQualityGates/job/folioCiQualityGates/

Next Time: Continue the quality topic



Meeting May 14, 2024

Attendees: Ostap, Kate, Yogesh, PK, Martin, Sasha K, Serhii, Sasha H

Topic

  1. Discuss progress with Karate/Gatling tests and reports
  2. Developers' maintaining e2e tests
 

Presenter: all

Quick Note: Sasha K and Serhii stated that teams Vega and Thunderjet finally have stably green Gatling reports. Thank you! Who's next please? 

Still not many teams have their FE developers maintain e2e tests. Folijet seems to be the only exception where FE developers are up to speed on Cypress. Success stories on the subject are expected next time. 

Next Time: TBD


Meeting April 30, 2024

Attendees: Ostap, Kate, Yogesh Matt, PK, Martin, Sasha K, Gurleen, Sasha H

Topic

  1. Discuss progress with Karate/Gatling tests and reports
  2. Discuss how teams are reacting to newly found bugs
 

Presenter: all

Quick Note: Sasha/Gurleen mentioned that Gatling is not running consistently, we need to look into it deeper. More - next time

WRT bugs, I'm hearing that in a development cycle, while newly found P1s are usually addressed immediately, the P2s are often placed on the backlog and addressed later (sometimes MUCH later - per PO's prioritization). Which means that we often complete stories with P2s. I personally don't like it, as I think this leads to more bugs before BF and during BF. Let's discuss further

A word of wisdom from Martin:

For every bug we have to ask 5 "why's".  Why did the bug happen? Let's say it's coding (i.e, not due to requirements), why coding? If we didn't know the impact that the new implementation causing on other parts of the code, why didn't we know? Because we didn't have an automated test. Why didn't we have an automated test for other parts of the code?  Because we didn't have time to write tests. Why didn't we have time to write tests? Because we don't have skills or because the code is not easily testable, or because we don't have time to skip features... Keep asking why will eventually get to the root cause, not just a superficial cause.  Every developer should be asking this question and help the team improve.  In this case if we don't have enough skills, work with management to get the skills. If the code is not testable, have a story and work with the architect to redesign the code. If no time, work with PO next time on priorities. 

Next time: 1) Summarize our progress with Gatling. 2) Continue discussion of how to reduce the number of P1/P2 defects in newly developed features




Meeting April 16, 2024

Attendees: Ostap, Kate, Matt, PK, Martin, Olamide, Sasha K, Slava, Gurleen, Sasha H

Topic: Continue with Karate tests. How do we address Karate failures immediately (as opposed to schedule bug fixes)

 

Presenter: all

Quick Note: According to the Leads, teams almost always look into Karate failures pretty much immediately and address them right away if the fix is expected to be a quick one. However, if the fix is not expected to be quick (maybe a day or two) then it's up to POs to prioritize and schedule the fix. We want to be more assertive in addressing Karate failures. Lee is going to say a few words on this subject at Mark V's FOLIO status meeting on Thursday



Meeting April 2, 2024

Attendees: Ostap Voitsekhovskyistap, Kate, Matt, PK, Yogesh, Martin, Olamide, Sasha K, Gurleen, Serhii, 

Topic: Continue with Karate tests. Using Karate tests for performance measuring. 

PresenterOlamide Kolawole

Quick Note from Olamide: let's get started! 


































We need each team lead to implement a scenario or more in one maven module within each team lead's ownership. You can reuse existing scenarios or create new one. Let's target next week tuesday(April 9th) to have this done so that we can have further conversations regarding its execution environment, test strategy, report collation, inclusion in continuous integration as quality gate. This first step's purpose is to gain a shared understanding. If you need any help or support please don't hesitate to reach out to me.

Link to PR

Link to recording

Link to official Karate docs


Meeting March 19, 2024

Attendees: Ostap Voitsekhovskyi, Kate, Matt, PK, Yogesh, Martin, Olamide, Sasha K, Gurleen, Serhii, 

Topic: Continue with Karate tests. Using Karate tests for performance measuring

PresenterOlamide Kolawole

Quick Note:

Thank you for your attention in todays meeting! Regarding Karate tests, we came to the following conclusions. This will be sort of be a list of action items that we will work towards to make karate tests better
  • Attain 100% pass rate. Teams will work to resolve flaky tests. Continuously erring tests can be excluded. We must be prepared to compromise.
  • After reaching a stable "100%" pass rate, teams have committed to hop on test failures for immediate resolution. This can have an impact on sprint commitments.
  • After reaching a stable "100%" pass rate for a flower release, a branch will be cut in the karate tests repo to represent the test suite that verifies the flower release.
I will continue to update this list as we work the items in our meetings. The immediate requests from teams regarding Karate tests is stabilization. I will also be preparing guidelines for Karate performance tests for your review. Thanks!


Link to recording

Link to presentation - none


Meeting March 5, 2024

Attendees: Ostap, Yogesh, Sasha K, Gurleen, Serhii, PK, Pavlo, Kate, Matt, Lee, Sasha H, Olamide, Slava 

Topic: Excellent presentation by Olamide on Karate testing strategies. This topic will continue next time as many participants had questions

PresenterOlamide Kolawole

Link to recording:

Link to presentation:

Next time: we will continue with the Karate testing discussion


Meeting February 20, 2024

Attendees: Ostap, Yogesh, Sasha K, Gurleen, Serhii, PK, Pavlo, Kate, Matt, Lee 

Topic: Yogesh Kumar is presenting the Shift-left program and its results so far

Quick notes: Yogesh reminded us why shift-left is important for early feedback on Quality. 

Presenter: Yogesh emphasized the importance of shift-left for early quality feedback. He also provided an update on our shift-left journey.

Link to recording

Link to Presentation



Meeting January 19, 2024

Attendees: Ostap, Yogesh, Sasha H, Sasha K, Gurleen, Serhii, PK, Matt, Lee 

Topic: Ostap Voitsekhovskyi is presenting automated testing and Team Mriya 

Quick notes: Ostap went through the "theory" of quality automation and why it's so important to us, as well as the technique of creating Cypress tests. I think we should continue this discussion next time and proceed to leads' questions. 

Presenter: Ostap Voitsekhovskyi 

Link to recording

Link to Presentation




Meeting December 19, 2023

Attendees Sasha K, Serhii, Kate, Pavlo, Matt, Gurleen, Lee, Sasha H (now part of the forum), Yogesh, Oleksii P

Topic: Sasha H presentation on how to use the Unified Pipeline. 

Quick notes: The Unified Pipeline is a set of jenkins jobs to create rancher environments for teams. Can the leads/team members use it so the teams take care of their immediate devops needs and free up team Kitfox to focus on strategic automation initiatives? 

Team leads are also encouraged to come up with additional requests for Kitfox. 

Presenter: oleksandr_haimanov 

Link to recoding

Link to Presentation (documentation)



Meeting December 5, 2023

Attendees Sasha, Serhii, Kate, PK, Matt, Slava, Lee

Topic: Linked Data and Bibframe

Quick notes: Team Citation is working on adoption of the concept of Linked Data Bibframe for FOLIO. 

Presenter: Punnoose Kutty Jacob Pullolickal  

Link to recoding

Link to Presentation

Next time: We will continue our conversation about LinkedData and FOLIO. Please prepare questions for Punnoose Kutty Jacob Pullolickal  


Meeting November 7, 2023

Attendees Sasha K, Serhii, PK, Matt, Gurleen, Slava, Lee

Topic: DCB (Direct Consortia Borrower)

Quick notes: The DCB is K int's product, and the main challenge is to integrate this product with FOLIO. Team Volaris is releasing the initial version of DCB-FOLIO integration today! (11/7/23)

Presenter: Gurleen Kaur1 

Link to recoding

Link to Presentation

Next time: LinkedData and FOLIO presented by Punnoose Kutty Jacob Pullolickal 


Meeting October 24, 2023

Attendees + Guests: Mark V, Maccabee Levin, Peter Murray, Sasha K, Kate, Serhii,  PK, Matt, Gurleen, Pavlo, Slava, Lee

Topic: Controlling AWS hosting costs (subgroup of Tech Council)

Presenter: Maccabee Levine 

Link to recording

Quick note: Starting from Q-release, new development environment requests for community projects have to go through the approval process by the Tech Council. This is done so that we can project the costs of AWS hosting. All such requests must be handled through Jira tickets assigned to Jira project COSTS

useful links: 

TC AWS hosting costs Subgroup: https://folio-org.atlassian.net/wiki/display/TC/Controlling+AWS+Hosting+Costs+subgroup

Process of creating Jira tickets for requesting an justifying new environments: https://folio-org.atlassian.net/wiki/display/TC/AWS+Environment+Lifecycle




Meeting October 10, 2023

Attendees: Matt, PK, Kate, Slava, Gurleen, Sasha K, Serhii, Lee

Topic: Matt's presentation on FQM and ListApp

Presenter: Matt Weaver 

Link to session recording

Link to FQM slide deck

Next topics (not in priority order). 

  • More on shift-left
  • DCB overview and challenges (Gurleen: 7th Nov)
  • COMPLETED: FQM overview and challenges, and the process of TCR
  • LinkedData overview and challenges
  • Onboarding external teams
  • Review the RCA program



Meeting September 26, 2023

Attendees: Matt, PK, Kate, Gurleen, Sasha K, Serhii, Lee

Topic: Leads Forum Format pseudo retrospective: how do we make the forum more effective 

Presenter: all


Overall Impression:

The forum has been useful!


Thoughts:

PK

  • Like the idea of sharing best practices, both technical and process related


Sasha K

  • Leave more space for interactions – i.e. never enough room for discussions after presentations.


Serhii

  • Presentation prep time is well-spent. You have the opportunity to rethink the previous decisions when you’re putting things on paper


Matt

  • Tech topics useful, learning a lot
  • Consortia support information is very valuable for FQM


Kate

  • Definitely keep it technical
  • Challenges: RRT support; Rapid response team is not that “rapid”.
  • RCA – how helpful is it?
    • Tech debt feature in each team. Maybe make it more specifics?

Gurleen

  • Gained useful knowledge from this forum.
  • DCB has been challenging


Decisions:

Continue with presentation, but have more time for discussion after presentations: Going forward we’ll start at 7:45 AM EST


Next topics:

  • More on shift-left
  • DCB overview and challenges
  • FQM overview and challenges, and the process of TCR
  • LinkedData overview and challenges
  • Onboarding external teams
  • Review the RCA program

Please suggest more




Meeting September 5, 2023

Attendees: Matt, PK, Slava, Kate, Sasha, Serhii, Lee

Topic: ECS (Extended Consortium Support) continued. Applause and big thanks to Serhii_Nosko !

Presenter: Serhii_Nosko

Link to session recording (new)

Link to ECS doc on Wiki (same as previously)

Link to Serhii's presentation (same as previously)

Next time: Let's discuss the format of our forum: what's working, what's not working, how to improve the usefulness, etc 

Potential next topics:

1) How's shift-left working (all)

2) FQM and List App (Matt Weaver )

3) DCB and Reshare (Gurleen Kaur1 )

4) Linked data and Bibframe (Punnoose Kutty Jacob Pullolickal )

5) please propose other topics


Meeting August 22, 2023

Attendees: Gurleen, Matt, PK, Slava, Kate, Sasha, Pavlo, Serhii, Lee

Topic: ECS (Extended Consortium Support)

Presenter: Serhii_Nosko

Link to session recording

Link to ECS doc on Wiki

Link to Serhii's presentation

Next time: Continue with the Consortium functionality


Meeting August 8, 2023

Attendees: Gurleen, Matt, PK, Slava, Kate, Sasha, Serhii, Lee

Topic: Shift-left Testing diagram within a sprint

Presenter: Alexander Kurash 

Presentation attached

Next time: Discuss Consortium functionality


Meeting July 28, 2023

Attendees: Gurleen, Matt, Pavlo, Sasha, Serhii, Lee

Topic: "What Went Wrong...". This is a discussion of development efforts that should have gone better, and what conclusions can be drawn from those efforts

Presenter:  Serhii_Nosko 

Presentation attached

Next time: We will continue discussing Serhii's prez, and we'll focus on TESTING, especially automated testing, especially automated testing of asynch architectures


Meeting July 11, 2023

Attendees: Gurleen, Matt, PK, Pavlo, Kate, Sasha, Serhii, Lee

Topic: "What Went Wrong...". This is a discussion of development efforts that should have gone better, and what conclusions can be drawn from those efforts

Presenter:  Kate Senchenko

Presentation attached