SPIKE: Define changes needed to allow R/W split of the DB for mod-source-record-manager

Owned by Kateryna Senchenko
Last updated: May 30, 2023
3 min read

Spike Overview

Jira link: MODSOURMAN-966 - Getting issue details... STATUS

Spike Status: COMPLETED

Objective: To enable the R/W split of the DB for mod-source-record-manager to have improvements in performance

Background

RMB-348 (RMB v35.0.0) introduced the ability to direct the read and write calls to the respective database node, the reader or writer (if the database is configured in such a way). The separation of read and write queries brings performance improvements and scalability as the Write node is freed up to do more writing while the read node is busy making retrievals. Read-only code in RMB has been updated to communicate to the Read node directly. To take advantage of the read/write split, mod-source-record-manager replaced the read-only code that calls the existing RMB read methods (which could handle both read and writes) with one of the new RMB read-only methods that exclusively communicates to the read node (MODSOURMAN-883). 

Problem Statement

Update of the JobExecution to assign the JobProfile (write query) and the request to process a record (reads assigned JobProfile for JobExecution) come separately (sequence of requests is described in data-import-workflow). In case Read instance is not synchronizing fast enough, some single record operations and importing via API could fail. See details in MODSOURMAN-909 and MODSOURMAN-906. To prevent such errors R/W split for mod-source-record-manager is disabled for Nolana (R3 2022) and Orchid (R1 2023) releases.

The problem was not observed for imports initiated via mod-data-import (importing a file) - in this case records come for processing in chunks via Kafka.

Scope

In Scope

Review of changes required to enable R/W split for mod-source-record-manager.

Out of Scope

Investigation of R/W split impact on other DI modules, measuring of performance gains by enabling the R/W split for mod-source-record-manager.

Research Questions

  1. What changes are required in mod-source-record-manager to enable R/W split?
  2. Are there any other places potentially causing problems similar problems? 

Deliverables

Implemented changes required to enable R/W split for mod-source-record-manager.

Option 1 - Selected

Use regular RMB read method (which can handle both read and writes) for request to get JobExecution by id. This way the request will go to the write version of the DB, which has the necessary information.

Pros

  • Simple and quick to implement
  • Easy to back-port to Nolana and Orchid
  • Low risk of regression
  • No reason to expect significant impact on performance 

Cons

  • The read-only request will go to the Write Instance of the DB, which breaks the concept of R/W split. It is applicable to one request though, other read-only requests will go to the Read instance.

Option 2

Implement a retry mechanism to query again in case JobProfile is not set for the JobExecution.

Pros

  • Not hard to implement
  • All read-only requests would go to the Read instance of the DB
  • Low risk of regression

Cons

  • Additional unnecessary calls to the DB

Option 3

Redesign mechanism of setting necessary information for processing (assigning JobProfile) and triggering the processing of the record. For single record operations (quickMarc and ISRI) the API can be adjusted to accept the DTO that has the record to process and the profile to process it with.

Existing sequence might stay as is for the regular DI flow initiated through mod-data-import (importing a file).

Pros

  • All read-only requests would go to the Read instance of the DB
  • Lower number of HTTP requests would be required for single record operations

Cons

  • Significant API changes that will require additional changes in mod-quickmarc, mod-copycat, possibly other places
  • Excessive testing will be required to prevent any regression
  • Overall flow for importing via API will be changed

Conclusion

The most easy and quick solution is Option 1. Functional testing is still required to make sure there is no other places, where errors might occur when R/W is enabled. Performance testing is required to measure the impact of R/W split