EPAM-EBSCO Team Leads' Forum
Permanent Forum Members:
Name | Team |
|---|---|
@Siarhei Charniak | Firebird |
@Pavel Filippov | Volaris |
@Kateryna Senchenko | Folijet |
@Alexander Kurash | Vega |
@Serhii_Nosko | Thunderjet |
@Pavlo Smahin | Spitfire |
@Pk Jacob Pullolickal | Citation |
@Matt Weaver | Corsair |
@Martin Tran | PTF |
@oleksandr_haimanov | Kitfox |
@Ostap Voitsekhovskyi | Mriya (AQA) |
@Yogesh Kumar | QA Manager |
@Roman_Fedynyshyn | PTF |
@Oleksii Kuzminov | Eureka |
@Craig McNally | Eureka technical PO |
@Varun Javalkar | Hosting Hub |
Meeting Minutes
Meeting Mar 3, 2026
Attendees: Serhii N, Oleksii K, Pavlo S, Matt, PK, Varun, Sasha H, Ostap, Sasha K, Pavel F, Oleksii K
Special Guest @Oleksii Petrenko
Topic: How do we ensure that Trillium delivers higher-quality software than our previous releases?
Oleksii’s proposal is to introduce team-specific, long-living branches where code would be tested before being merged into master. This would serve as the first step toward the broader GitLab strategy built around three branch levels: Development → Staging → Master.
However, we have already acknowledged that keeping team branches synchronized could become a significant challenge. Promotions from team branches to master could be a challenge as well. In addition, Sasha H. pointed out that creating all the necessary DevOps artifacts for Trillium may also present a substantial effort and risk.
Define “better quality”:
Escape defects: fewer Sev1/Sev2 bugs in first 2–4 weeks after release
No regressions.
Change failure rate: fewer rollbacks / hotfixes
Lead time for fixes: faster time from bug found → fix in prod
Release readiness: fewer “known issues” at GA
To me, the MAIN QUESTION IS: how do we continue with trunk-based development for now, but keep master always green?
Meeting Feb 17, 2026
Attendees: Serhii N, Oleksii K, Pavlo S, Yogesh, Matt, PK, Varun, Sasha H, Ostap, Sasha K
Topic: Use of AI tool for development productivity increase
Notes: Team Eureka is working on a library of standardized prompts for AI across multiple tools. Presently, they have developed prompts for generating unit-test code and generating documentation. Here is the link https://github.com/folio-org/folio-eureka-ai-dev
@Serhii Nosko 's team will contribute prompts for writing Karate test.
@Pk Jacob Pullolickal spoke about his experience of working with GPT 5.3-codex tool. He states that it’s more intelligent than copilot in interpreting the English-language requirements and generating comprehensive code. EBSCO presently has enterprise license for OpenAI. Please reach out to Laxmikant Deshpande via Teams.
Meeting Feb 3, 2026
Attendees: Serhii N, Siarhei C, Yogesh, Matt, PK, Varun, Sasha H, Ostap, Pavel F
Topic: what can we do to reduce the number of escaped defects.
Notes:
Serhii N
Environment Parity: Align test environments (Bugfest) with production. Why: Currently, we test different flows (e.g., Login Dropdown) than what clients use (Single Tenant UX). We must test the actual configuration users face to catch real-world bugs.
Feature Flags: Adopt an open-source tool for managing feature toggles. Why: This allows us to test more configuration combinations. Feature flags count is growing and support of new feature flags would be harder in future
Centralized Configuration: Move configuration from Confluence pages to a dedicated tool/service.
Why: Confluence is static and error-prone. A tool provides a "single source of truth," allows specific overrides for large Consortia customers, and lets the PTF team push performance tuning directly to the centrilized system.
Siarhei C
Features Estimates: think about possible issues based on prior experiences, such as performance, configuration, ECS handling
Yogesh
Larger test environments for Sprint testing with more tenants. More realistic environment
User migration testing (even with Eureka to Eureka)
Performance testing during sprint testing. Load testing rather than speed testing
Performance Gatling reports at unit and integration level. Individual speed testing
Matt
Identify more edge cases upfront
More realistic test environment
PK
Are we overusing Karate tests and spend a lot of time where other techniques could help? For interactions happening within one module, make use of Mock MVC and Spring test containers for integration tests. Use Karate for inter-module interactions. Follow the Quality Pyramid.
Liquibase data migration testing
Ostap
Use multiple data sets for testing that embrace customer-specific data peculiarities.
Oleksandr H
Stricter adherence to feature and code freeze deadlines, and to the timeline in general
We ran out of time before Varun and Pavel could speak
Meeting Jan 20, 2026
Attendees: Slava K, Serhii N, Sasha K, Siarhei C, Pavlo S, Matt, PK, Varun, Kate S, Martin, Sasha H
Topic: New Implementation of Gatling Performance reports in Java (the previous version that we abandoned earlier was in Scala)
Next time, we’re going to talk about Gatling execution pipeline.
Meeting Dec 9, 2025
Attendees: Serhii N, Sasha K, Siarhei C, Pavlo S, Pavel F, Matt, PK, Yogesh, Boburbek, Varun
Topics: Link Karate tests/scenarios and their results to TestRail test cases. Team Thunderjet created a tool to automatically link the Karate scenarios to TestRail cases. More automation is always good. Potentially replace some e2e tests with Karate tests, which are easier to build and faster to run.
Meeting Nov 25, 2025
Attendees: Matt, Serhii N, Sasha K, Sasha H, Pavel F, Pavlo S, Ostap, Yogesh, Oleksii K, Boburbek, Varun, Ostap
Topics:
Introduce new member @Varun Javalkar , Team Lead for Hosting Hub (Operational Portal)
@Serhii Nosko and @Boburbek Kadirkhodjaev introduced v2.0 of Thunderjet’s tool Eureka CLI. Most important new features in 2.0
Architectural Overhaul: Transitioned from monolithic scripts to a modular Service-Oriented Architecture using Dependency Injection.
Improved Performance: Introduced Native Sidecars (GraalVM/Quarkus) via the combined-native profile, reducing sidecar memory usage by ~90% and startup times to milliseconds.
Solid Stability: Added 720+ AI-generated unit tests and replaced panic-based failures with robust, centralized error handling.
ECS & Consortium Support: Out-of-the-box support for Enterprise Consortium Stack deployments with dedicated ecs and ecs-single profiles.
Modular Profiles: New targeted configuration profiles (search, export, import, edge) to easily layer functionality onto the base platform.
Enhanced Tooling: Upgraded to Go 1.25.3 and AWS SDK v2, featuring automated security scanning (Trivy) and semantic release automation.
@Varun Javalkar demoed the Log Download feature of Operational Portal. Please note that we want all people involved in RRT work to be able to access Log Download. Also please note that Varun’s team continues to improve the feature by introducing log stream aggregation and search. At this time, most leads have confirmed their ability to access the Portal. Sasha K and Sasha H are unable - they will work with Varun and we’ll go from there.
More documentation on the Eureka CLI tool: https://github.com/folio-org/eureka-setup/tree/master/eureka-cli#readme
Meeting Sep 30, 2025
Attendees: PK, Matt, Serhii N, Sasha H, Pavel F, Siarhei C, Ostap, Yogesh
Presentation by PK Jacob re AI MCP Server. Can FOLIO use a NLM input?
Another link from Sasha H: https://ref.tools/
Here’s a simple architecture sketch and flow showing an MCP server between FOLIO and an AI assistant.
┌──────────────────────────────┐
│ AI Assistant │
│ (Chat UI / LLM client app) │
└──────────────┬───────────────┘
│ (natural language)
▼
┌───────────────┐
│ LLM (any) │
│ (Claude/GPT/…)│
└───────┬───────┘
│ (MCP client)
▼
┌──────────────────────┐
│ MCP Server │ ⇦ “FOLIO AI” module
│ (tool registry + │
│ auth + schemas) │
└─┬───────────┬────────┘
│ │ (tool calls: structured JSON)
┌──────▼──────┐ ┌──▼───────────┐
│ FOLIO Tools │ │ FOLIO Tools │ …additional tools
│ Read APIs │ │ Write APIs │
│ (search, │ │ (create, │
│ get by id) │ │ update, del) │
└─────┬───────┘ └──────┬────────┘
│ (HTTP/RPC) │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ FOLIO Module │ │ FOLIO Module │ …(Inventory, Users, Orders, etc.)
│ Inventory │ │ Users │
└──────────────┘ └──────────────┘
[Shared services: RBAC/permissions • Audit log • Rate limits • Secrets]
How it works (1–2–3)
User asks in plain English: “Update user 123’s email to …”
LLM plans the steps and calls MCP tools (e.g.,
folio.users.updateEmail) via the MCP client.MCP server validates auth/permissions, executes against FOLIO APIs, returns structured results; LLM replies to the user.
What lives in the MCP server (“FOLIO AI”)
Tool registry: declarative list of allowed operations with JSON schemas and examples.
Policy guardrails: RBAC check, scope rules (read/update), dry-run mode, approvals for sensitive ops.
Audit & observability: full traces of prompts, tool calls, inputs/outputs.
Provider adapter: pick/swap LLMs without changing FOLIO tools.
Meeting Sep 16, 2025
Attendees: PK, Matt, Serhii N, Sasha K, Sasha H, Yogesh, Ostap V
Topic: Use Copilot to generate Karate tests and conduct self-code review.
Presentation by @Serhii Nosko
Meeting August 19, 2025
Attendees: PK, Matt, Kate, Serhii N, Pavel F, Yogesh, Slava, Sasha K, Boburbek, Eldiar, Oleksii K, Martin T
Topics:
Create environments, local and Rancher, to run multiple consortia in one cluster
Presentation by @Serhii Nosko supported by @Boburbek Kadirkhodjaev and @Eldiiar Duishenaliev .
Notes (from Copilot):
Generated by AI. Be sure to check for accuracy.
Meeting notes:
Implementation of Multi-Consortia Support: Serhii presented the new feature enabling support for multiple consortia within a single cluster, explaining its implementation, verification, and adoption by Kitfolk, with Lee and Eldiiar participating in the discussion and demonstration.
Feature Overview and Purpose: Serhii explained that the new release introduces support for adding a second consortium, allowing teams to deploy multiple consortia within the same Rancher cluster to optimize resource usage and reduce costs, particularly for teams like FSC with large and small consortia.
Declarative Configuration and UI Bundles: Serhii described the declarative approach for configuring the number of consortia and tenants via a YAML file, and detailed how different UI bundles are mapped to different ports to support multiple consortia.
Database Schema and Tenant Mapping: Serhii demonstrated how each tenant is mapped to a specific consortium in the database, with separate schemas for each consortium, and explained the storage of consortium and central tenant IDs for each tenant.
Elasticsearch Indexes per Consortium: Serhii outlined the creation of separate Elasticsearch indexes for each central tenant, ensuring that data for each consortium is managed independently, and showed how the correct central tenant ID must be passed when enabling search functionality.
User and Shared Instance Management: Serhii demonstrated that users and shared instances can only be shared within the same consortium, not across consortia, and showed how settings and roles are isolated per consortium, with Lee confirming the expected behavior through questions.
Technical Design and Internal Workflows: Serhii provided a detailed explanation of the internal technical design, including API usage, database schema decisions, and the rationale for storing consortium and central tenant IDs, with Pk Jacob asking clarifying questions about data storage and dummy records.
User Tenant API and Data Flow: Serhii described the use of the user tenant API, which provides central tenant and consortium IDs for each tenant by extracting the tenant ID from headers and querying the appropriate database schema, ensuring correct identification for each tenant.
Consortium ID Parameterization: Serhii explained that the system was designed from the beginning to accept a consortium ID parameter, which allowed for the addition of multi-consortia support without major refactoring, as most endpoints already handled this parameter.
Dummy Records in Database: In response to Pk Jacob's question, Serhii clarified that dummy records are used in the mod users table to store central tenant and consortium IDs, facilitating integration and minimizing dependencies between modules, and supporting features like password recovery.
Central Tenant and Member Tenant Associations: Serhii detailed that the central tenant stores associations for all users, which is necessary for certain functionalities, while member tenants use dummy records to indicate their central tenant and consortium associations.
Rancher Integration and Documentation Updates: Serhii and Eldiiar discussed the integration of multi-consortia support into Rancher environments, the implementation of related jobs, and the updating of documentation, with Lee requesting documentation links and Eldiiar confirming the work.
Rancher Environment Support: Serhii explained that the new functionality is now available in Rancher environments, allowing teams to add a second consortium either during namespace creation or to existing Rancher setups, with jobs provided to facilitate this process.
Documentation and Slack Integration: Serhii mentioned that documentation has been updated to reflect the new features, and that agent bots in Slack can be used to guide users through the process of adding consortia in Rancher.
Collaboration and Implementation: Serhii credited Eldiiar for implementing the Rancher integration and updating documentation, and Eldiiar confirmed that there was nothing further to add regarding the Rancher work.
Demo and Adoption by Teams: Serhii noted that a demonstration was conducted for the FSC team, and that other teams are beginning to set up their environments using the new multi-consortia support.
Tooling Support and Acknowledgments: Serhii acknowledged Bobur for maintaining the Erika setup tool, which supports local multi-consortia deployments, and highlighted the tool's ability to run multiple consortia from a single command, with Lee and Eldiiar commenting on Bobur's role.
Erika Tool Maintenance: Serhii thanked Bobur for supporting and updating the Erika setup tool to accommodate changes in contracts and enable local deployment of multiple consortia, making it easier for teams to test and develop locally.
Follow-up tasks:
Documentation Sharing: Paste the updated Rancher documentation links into the Teams chat window for team access. (Serhii)
Documentation links:
https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/865730596/How+to+recreate+Eureka+Rancher+environment
https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/699498501/Create+Eureka+environment
https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/508690454/Namespace+useful+info+tools
https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1123254306/Add+second+consortium
Automated performance testing during feature development
Martin raised the issue that performance degradations are often discovered late in the cycle by his team, creating significant disruption and urgent rework for development teams. To address this, we are setting a goal for FY2026 to detect performance degradation early in the development cycle. Achieving this will reduce downstream bottlenecks, prevent last-minute fire drills, and improve both product quality and delivery efficiency.
Previously, we discussed the use of the Gatling framework for performance testing, and some teams created a number of tests. Kitfox, for example, has an established Gatling pipeline:
👉 Kitfox Gatling Pipeline
Unfortunately, this practice has since been abandoned by all teams. Reviving, standardizing, and integrating such performance testing into the active development workflow will be essential to meeting our FY2026 goal.
@Viachaslau Khandramai volunteered to look into it.
Meeting July 22, 2025
Attendees: PK, Matt, Kate, Serhii N, Pavel F, Yogesh, Slava, Ostap, Sasha K
Topics:
@Ostap Voitsekhovskyi discussed Cypress test monitoring instrumentation. Leads - please make sure your team’s test, at least Smoke and Critical Path ones, run smoothly in nightly runs. Also please make sure your FE developers know how to maintain the tests and know how to use the instrumentation that AQA team have created.
Links to AQA documentation:
https://folio-org.atlassian.net/wiki/spaces/DQA/pages/722108508/Environments
https://folio-org.atlassian.net/wiki/spaces/DQA/pages/2660627/AQA+Setup+TAF+locally
https://folio-org.atlassian.net/wiki/spaces/DQA/pages/2666540/Filtering+tests+by+tags
Link to Ostap’s presentation:
Folio Team Leads' Forum-20250722_074808-Meeting Recording.mp4
Meeting June 10, 2025
Attendees: PK, Matt, Kate, Serhii N, Pavel F, Yogesh, Slava
Topics:
@Viachaslau Khandramai ‘s report on how Firebird integrates their modules, notably, Bulk Ops (and soon Data Export), with FQM. Slava notes the significant performance improvement when creating multiple streams when returning the datasets. He recommends, up to 10 streams (or threads) and up to 10,000 records in each. Slava’s presentation is here: https://open-libr-foundation.slack.com/files/U03R4UMKG0M/F090L0AGEJX/fqm_integrating.pptx
Presentation recording is here: Recap: Folio Team Leads' Forum Tuesday, June 10
We discussed as a team the importance of maintaining e2e test cases. All leads should ensure their teams are aware that these tests should not break due to changes in screens or logic. Responsibility for keeping tests up to date lies with the individual teams—not with the AQA team.
To check how your team’s test cases are running, please refer to the Allure report:
https://jenkins.ci.folio.org/job/folioScheduledTesting/job/folioQualityGates/1690/allure/You should also monitor #rancher_tests_notifications on Slack for updates.
Interesting point was brought up by @Matt Weaver who mentioned that the Gen AI agent built into IntelliJ can potentially significantly reduce the time developer spends creating unit and integration test cases, while increasing their value and quality. We’re looking forward to more info from you Matt!
Meeting May 27, 2025
Attendees: PK, Sasha K, Matt, Kate, Serhii N, Pavel F, Oleksii K, Boburbek, Yogesh
Topic: Evolution of the Eureka-cli Go tool for local troubleshooting
Serhii N presented the next incarnation of the Eureka-cli tool. Its key features:
1. Ability to deploy child applications for search, export and edge modules functionality
Add Elastic search, Kibana to list of supported containers
Add
getEdgeApiKeycommand to the tool functionalityAdd ability to use custom sidecars built locally
Add --Required option to
deployApplicationcommand to have ability to deploy only required containers to save memoryImprove Readme documentation, add different stabilizations and improvements to the tool source code
Documentation: https://github.com/folio-org/eureka-setup/tree/master/eureka-cli
Meeting April 1, April 15, 2025
Attendees: PK, Sasha H, Sasha K, Siarhei Ch, Matt, Kate, Serhii N, Craig,
Topics: pre-defined Dataset
LoC has allowed us to use their 1,000,000 sample data for testing. @Pk Jacob PullolickalK Jacob created several data sets out of the LoC sample data with different sizes. Please see this link: Test data - Sample MARC records for setting up test environments
The data can be loaded into individual envs using Data Import mechanism
Recording of PK's short presentation is here: https://ebscoind-my.sharepoint.com/personal/ykumar_corp_epnet_com/_layouts/15/stream.aspx?id=%2Fpersonal%2Fykumar%5Fcorp%5Fepnet%5Fcom%2FDocuments%2FRecordings%2FFolio%20Team%20Leads%27%20Forum%2D20240319%5F074732%2DMeeting%20Recording%2Emp4&referrer=StreamWebApp%2EWeb&referrerScenario=AddressBarCopied%2Eview%2Ef370ab13%2D24a9%2D42d6%2Da3ff%2D9ac6569c5640
Questions from @Serhii Nosko:
1. Contract with UI, is it defined in some way?
2. Did you face some issues with parallel processing, if tests running in 10 parallel threads - its not only functional testing, but also some kind of performance and stress testing?
3. Currently we have 2k tests, the plan is to have 5k tests? The plan is to automate all test cases for Bugfest?
4. If we have a new feature, when for your practice automated tests are written for it, in the next release?
5. From your practice, do we have some rounding failures, is it connected with infrastructure?
@Pk Jacob Pullolickal - this is for you
Meeting March 11, 2025
Attendees: PK, Sasha H, Sasha K, Gurleen, Matt, Yogesh, Pavlo, Slava, Kate, Serhii N, Craig, Ostap, @Oleksii Petrenko, @Eldiiar Duishenaliev
Topics: Rancher self-service, Kitfox prepared documentation
With the help of the documentation below, Development Teams should be able to take care of their own ranchers envs.
https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/840138784/Rancher+FAQ
Main points:
Kitfox Team will be reduced in size, so the development teams will need to practice "self-help" more. Kitfox will continue to improve documentation. Use the Chatbot (https://copilotstudio.microsoft.com/environments/Default-b41b72d0-4e9f-4c26-8a69-f949f367c91d/bots/cr4d9_folio/canvas?__version__=2&enableFileAttachment=true), but remember it's a POC
Kitfox is not going away so it will still provide support when needed, but, please try to minimize the need for such support
Shared envs, like etest, will continue to be managed by Kitfox
Teams should avoid deploying untested or partially tested code to shared envs. @Pavlo Smahin - if Spitfire needs an edev with a large dataset, please talk to @Matt Weaver of Corsair, as Matt's team has a rancher with large dataset
Next time - please come with your ideas how Kitfox could make the feature teams more efficient in resolving their devops issues
Meeting February 18, 2025
Attendees: PK, Sasha K, Gurleen, Matt, Yogesh, Pavlo, Slava, Kate, Serhii N
Topics:
Upgrading Java 17 to 21
Slated for SF, FOLIO-4182 Migration to Java 21 and Spring Boot 3.4
Could gain performance because of the new Virtual Threads. Also fewer containers (cost optimization)
It is possible to migrate some modules to 21 and keep others on 17
Eureka Guide
Please continue reviewing and providing comments: Eureka Developer Guide
Feature Toggles
Several teams are either doing them or already have them, namely, Tjet, Volaris, Spitfire, Vega. Citation is a special case as their FT is to run LDE outside of FOLIO and hence cannot use FOLIO tools
It appears that we do not have a single way of implementing FTs, i.e. each team is doing it their own way. We should regroup on that
It appears that the Eureka platform may have a mechanism to implement FT through its Env Vars, which would be the unified way of implementing FTs
Sensitive info
@Viachaslau Khandramai is running the show. He outlined what we mean by "sensitive". Please remember that it's only the PII, but also information containing account and invoice numbers, dollar amounts and things of this nature. Scott Macdonald: 'if it feels sensitive, treat it as such'
POs are scheduling stories for all teams in SF
Meeting January 21, 2025
Eureka Adoption
Tjet
Developing on Eureka (except Karate tests)
Use Telepresence
Karate tests still use Okapi
Single person can debug – deal with it by intrateam communication
Vega
Continue on Okapi and deploy to Eureka environment
Lots of business func that works for both
Eureka is much slower than Okapi
Citation
Develop on Eureka
Use Telepresence
Keep 1 Okapi env for Karate tests
Volaris
Develop on Eureka, use Telepresence
Keep 1 Okapi environment for Karate, DCB and Inreach
Inreach is NOT moving to Eureka???
Corsair
Still mostly use Okapi but have Eureka env with Telepresence
Primary env is still Okapi, but planning to replace to Eureka in about a week
Firebird
Development work on Eureka
Telepresence
1 Okapi env for Karate test
Folijet and Spitfire were unrepresented
Meeting January 7, 2025
Happy New Year all!
Attendees: PK, Sasha K, Gurleen, Matt, Sasha H, Ostap, @Boburbek Kadirkhodjaevlava, Yogesh, Pavlo, Craig McNally, Martin
Topics: 1) Local and remote debugging on Eureka platform 2) Sensitive info in application logs
Quick notes:
Eureka debugging
Setting up the local development env via the Go tool only works for those with at least 32 GB of RAM.
Remote debugging appears to be a viable option as well. Several notes here:
At this point only 1 developer can hit breakpoints in Rancher (all others will be blocked when a breakpoint is hit)
Sasha H said that there could be an option to use separate namespace for each tenant and hence enable multiple developers to debug. This needs to be explored further
PK suggested that if there was a way to quickly spin up a rancher env per developer and tear it down once the debugging is done, this could be another option. Cost $$ is a factor here, but labor and efficiency maybe worth more...
Lee B contacted Embassador Labs to understand the pricing model of their multi-user Telepresence option. I'll keep you posted https://www.getambassador.io/telepresence-pricing
Sensitive info in Logs
Reiterated to the leads that the sensitive info (not only the PII, but also things like dollar amounts, account numbers, etc) should NOT be at the INFO level in logs. Preferably it should not be there at all, but if we find it necessary to troubleshoot production issues, put it at the DEBUG level.
Teams TJet (mod-invoice, mod-orders) and Volaris (mod-audit, mod-email) are creating stories to remove the sensitive info.
Meeting December 10
Tjet exploring remote debugging on Rancher; building local UI; starting experimenting with Telepresence
Citation 16GB is not enough
Volaris problems building and running UI modules; trying to figure out problems with running DCB module
Folijet no progress, using deploying to Rancher
Firebird. no progress