Spike - Identify opportunities to split mega applications into smaller pieces

Spike Overview

JIRA ID: EUREKA-305

Objective:
We have several large applications, often referred to as "mega applications." Our goal is to break these down into smaller, more granular applications to enable more frequent releases and deploy only the necessary components to the cluster for customer needs. The first step is to achieve this at the module level without refactoring the code or modifying dependencies. Next, we will align this approach with subject matter experts, who will independently define applications from a FOLIO perspective. We will then collaborate to find a balanced solution between both approaches.

Scope:

• Identify suitable candidates for breaking modules out of the mega applications into smaller, standalone applications.

• Focus on modules that many others depend on but that have fewer dependencies themselves, such as inventory or potentially finance.

• Consider team responsibilities when splitting applications, especially in cases where applications span multiple team boundaries. For example, mod-inventory and ui-inventory are owned by Folijiet, while mod-inventory-storage is managed by Spitfire.

The application descriptor we are analyzing and attempting to split is:

app-platform-complete - GitHub link
It includes 53 backend modules and 54 UI modules.

Representing dependencies using a graph approach:

To illustrate the dependencies between modules, I decided to use a graph representation, where each module is a node, and the dependencies are represented as edges between them.

In the screenshot below, you can see all backend modules in the system, along with their dependencies, arranged in a hierarchical layout.

Open

Evaluation

To group or aggregate nodes into larger nodes (or applications) based on dependencies—where nodes within a group only have dependencies on each other (i.e., they are internally connected but not externally)—the following steps can be followed:

Steps to Aggregate Nodes Based on Dependencies:

1. Identify Subgraphs:

   • Connected Components: Begin by identifying connected components (subgraphs) within the graph. These represent groups of nodes that are all directly or indirectly connected. If dependencies are confined within a connected component, the nodes in that group can be aggregated.

2. Dependency Analysis:

   • For each node, evaluate its dependencies. If all dependencies are confined to a specific subset of nodes, that subset can potentially be grouped into a larger node.

3. Cycle Detection:

   • Strongly Connected Components (SCC): In directed graphs, look for SCCs, where each node is reachable from every other node in the group. Nodes within the same SCC can be grouped together.

4. Community Detection Algorithms:

   • Utilize clustering algorithms, such as modularity-based or hierarchical clustering, to group nodes that are more interconnected internally than with nodes outside the group. This method allows for more flexible groupings beyond SCCs or connected components.

5. Check External Dependencies:

   • After identifying potential groups, ensure no nodes within the group have dependencies outside of it. If external dependencies exist, the group may need further refinement.

6. Merge Nodes:

   • Once valid groups are identified, where dependencies are self-contained, aggregate them into larger nodes. This process reduces the graph by replacing smaller nodes with single, larger nodes.

The tools used to build the graph and visualize dependencies are outlined below.

TBD link on GitHub after clearing the code

Implementation

The following modules have no dependencies and can be easily divided into separate applications:

Candidates for Splitting with Minimal Changes:

The following modules can be divided into smaller applications if certain dependencies are resolved or eliminated.

The primary candidate is the Notification application, which consists of five modules.

Please refer to the screenshot below:

Open

Catalog of applications with modules, structured according to business operations and dependencies.

Below is a preliminary catalog of modules grouped by backend and UI within the application, based on the technical analysis. This catalog highlights applications that can be separated without requiring code changes.

A dependency graph of applications divided by business operations.

The diagram below shows a graph representing all dependencies between the applications listed in the catalog. The graph includes applications without circular dependencies, which can be split independently; however, some applications have circular dependencies that must be resolved before further separation into smaller components.

The teams assigned to manage the maintenance of the newly split applications.

Decision record

Based on our discussion, we have agreed not to add additional functionality in EUREKA to manage circular dependencies. Instead, we’ll ask the team to resolve these dependencies first, allowing us to split them into smaller applications further once ready. Simultaneously, we have agreed to split approximately seven smaller applications during the Sunflower release and assign the appropriate team for future maintenance. Additionally, we identified one candidate application, app-organization, which can be split if the Sunflower team can do so during the release.

Conclusion

To continue applying the algorithm, we can attempt to group certain modules within the applications and create stories for the team, outlining the required changes and how to implement them. However, this process must still be aligned with subject matter experts, who will also work on splitting modules into applications, and then negotiate an approach that strikes a balance.

Spike Status: In Progress