Table of Contents

...

Pic. 1 Example "Kafka UI topics & partitions"

Adjust Kafka messages retention

Before starting of indexation find log.retention.minutes (if log.retention.ms is null) property and set it to 24 hours (1440 minutes). Could be set on broker level.

If decided to do this on a topic level only - log.retention.ms should be changed because it has higher precedence over ..minutes and it's usually already set to some value.

Image Added

Scale-up OpenSearch

As indexation is a heavy process with high CPU and memory resource consumption, it is required (and strongly recommended) to scale up the shared OpenSearch AWS service.

r6g.xlarge → r6g.2xlarge

Tune mod-search config

KAFKA_EVENTS_CONCURRENCY (default - 2) with higher value could increase instances reindex.

KAFKA_CONTRIBUTORS_CONCURRENCY (default - 2) with higher value could increase instances reindex.

KAFKA_SUBJECTS_CONCURRENCY(default - 2) with higher value could increase instances reindex.

No sense to make this higher than topic partition number because consumers will be created max 1 for partition.

So if we have 50 partitions and 4 mod-search instances - we may set KAFKA_SUBJECTS_CONCURRENCY to 13 so 4*13 = 52 and 12-13 consumers will be created for each app instance.

Considering that there should always be more subjects/contributors than instances - only subjects/contributors should be tuned. If there's an observation that subjects/contributors are read from topic faster than published - then we may want to also tune instances topic.

Scale-up backend modules

For better performance, please scale up backend modules.

...

After completion of all pre-required steps, trigger index with POST Postman request.

URI: /search/index/inventory/reindex

Headers: X-Okapi-Tenant & X-Okapi-Token

Body: 

{
  "recreateIndex": true,
  "resourceName": "instance"
}

...

After the indexation process is finished, do not forget to scale down the shared OpenSearch AWS service

r6g.2xlarge → r6g.xlarge (If no other conditions/constraints)

Adjust Kafka messages retention back

Return previous value to log.retention property (usually 8 hours).

Tune mod-search config back

Return modified env variables to default values