reindex.md

1. Use Cases and Scenarios

1.1 Full Reindex

Purpose: Pulls all data from mod-inventory-storage to mod-search postgres database, recreates elasticsearch indices and fills them (instances, holdings, items, contributors, subjects, classifications, call-numbers).
When to use:
- After MARC to FOLIO mapping changes.
- When index corruption is suspected.
- After data migrations
- After mod-marc-migrations usage with publishEvents=false parameter value.

1.2 Upload Reindex

Purpose: Re-uploads records to the search index without merging from the source database.
When to use:
- When only the search index needs to be refreshed (e.g., after OpenSearch upgrade or index deletion).
- For faster recovery when source data is unchanged.

1.3 Reindexing Failed Ranges

Purpose: Retries only the failed ranges from a previous reindex operation.
When to use:
- After a partial failure due to infrastructure or data issues.
- To minimize downtime and resource usage by not repeating successful work.

2. API Usage Guide

2.1 Start Full Reindex

Endpoint:
POST /search/index/instance-records/reindex/full

Description:
Initiates a full reindex for all records.

Full reindex API docs

Request Body:
No parameters required. Optional body example:

{
  "numberOfShards": 1,
  "numberOfReplicas": 1,
  "refreshInterval": 1
}

Please note that “numberOfShards” parameter is static and cannot be changed at runtime, only during index recreation. Other parameters are dynamic and could be changed at runtime. Update Index Settings could be used to revert index dynamic settings to their original state.

Response:
Returns a task ID for monitoring.

2.2 Start Upload Reindex

Endpoint:
POST /search/index/instance-records/reindex/upload

Description:
Starts an upload-only reindex, skipping the merge phase.

This API will recreate indexes for listed entity types and fill them with data from postgres. This api doesn’t interact with mod-inventory-storage, data is being taken from postgres, then sent and consumed by mod-search via Kafka.

Upload reindex API

Request Body:
Required only example:

{
  "entityTypes": [
    "instance",
    "subject",
    "contributor",
    "classification",
    "call-number"
  ]
}

Possible entity types are listed above.

Required and optional example:

{
  "entityTypes": [
    "instance",
    "subject",
    "contributor",
    "classification",
    "call-number"
  ],
  "indexSettings": {
    "numberOfShards": 1,
    "numberOfReplicas": 1,
    "refreshInterval": 1
  }
}

2.3 Retry Failed Ranges

Endpoint:
POST /search/index/instance-records/reindex/merge/failed

Description:
Retries only the failed ranges from the last reindex attempt.

Request Body:
No parameters required.

2.4 Monitor Reindex Status

Endpoint:
GET /search/index/instance-records/reindex/status

Description:
Returns the status of the current or last reindex operation, including per-entity progress and errors.

3. Known Limitations

No Multi-Tenant Parallel Reindex:
Running reindex for multiple tenants in parallel is not supported and may result in deadlocks or failures.
Resource Constraints:
Large-scale reindexing is resource-intensive and may require scaling up mod-inventory-storage, mod-search, Postgres database and OpenSearch nodes.
Partial Failures:
Some entity types may fail while others succeed; always check the status endpoint for details.

4. Performance-Based Configurations

4.1 Database Sizing

For 10M records, use at least db.r6g.4xlarge or db.r6g.8xlarge for optimal performance.
For smaller datasets, db.r6g.2xlarge or db.r6g.xlarge may suffice.

4.2 OpenSearch Sizing

Use at least 4 data nodes of r6g.2xlarge.search or r6g.4xlarge.search.
Ensure EBS volumes have sufficient IOPS (3000+) and throughput (250 MiB/s+).

4.3 mod-search and mod-inventory-storage

Set task count to 4 or higher for parallelism.
Memory:
- Mem Hard Limit: 2592–25923 MB (depending on instance size)
- Xmx: -XX:MaxRAMPercentage=85.0
CPU: Allocate at least 2 vCPUs per task.

4.4 mod-search

FOLIO_REINDEX_MERGE_RANGE_PUBLISHER_CORE_POOL_SIZE=6: This environment variable in mod-search controls number of threads used to invoke mod-inventory-storage to publish inventory data. mod-search calls mod-inventory-storage via HTTP which causes mod-inventory-storage to publish inventory data into Kafka. Kafka messages are consumed by mod-search. It is important to note that only one instance of mod-search will make calls to mod-inventory-storage.
KAFKA_REINDEX_RECORDS_CONCURRENCY=4: This environment variable in mod-search controls how many Kafka consumers that will be initiated to consume inventory data(produced by mod-inventory-storage). By default, 16 partitions are created by mod-search for most of the Kafka topics. With 4 instances of mod-search, it made sense to set the environment variable to 4 so that 16 consumers will consume from 16 partitions. These consumers are responsible for saving inventory data from Kafka into mod-search’s database schema.
KAFKA_REINDEX_RANGE_INDEX_CONCURRENCY=6: This environment variable in mod-search controls the number of Kafka consumers that will consume “reindex ranges”. These ranges are retrieved from mod-search database schema, transformed then saved into Opensearch.
- {env}.ALL.search.reindex.range-index Partition Count=24: Additional partitions were added to this topic to match concurrency settings in mod-search. With 4 instances of mod-search, 24 partitions are needed to meet KAFKA_REINDEX_RANGE_INDEX_CONCURRENCY=6.
Full reindex should be initiated with modified index settings
POST /search/index/instance-records/reindex/full HTTP/1.1 Content-Type: application/json { "numberOfReplicas": 0, "refreshInterval": "-1" }
After reindex completion index settings should be reverted to previous state
PUT /search/index/settings HTTP/1.1 Content-Type: application/json { "resourceName": "instance", "indexSettings": { "numberOfReplicas": 1, "refreshInterval": "1" } }

4.5 Example Configurations

Database Instance	Reindex Duration (10M records)

Database Instance	Reindex Duration (10M records)
db.r6g.xlarge	2h 25m
db.r6g.2xlarge	1h 50m
db.r6g.4xlarge	1h 22m
db.r6g.8xlarge	1h 25m

For 32M records, expect 3+ hours with similar scaling.

5. Troubleshooting and FAQs

5.1 Common Issues

Partial Failures (e.g., some entities fail):
- Use the "Retry Failed Ranges" endpoint to resume only failed parts. Usually it resolves the issue. Manual data correction may be required beforehand. Check database for failed ranges ids, find logs for those ids
Deadlocks Detected:
- Avoid running multiple reindexes in parallel.
Reindex stops progressing, processed merge/upload ranges count is lower than total and does not grow:
- Issue may happen when reindex is triggered while previous reindex run hasn’t finished. Restart reindex.

5.2 Monitoring

Use /search/index/instance-records/reindex/status to track progress
Monitor OpenSearch and database metrics for resource bottlenecks.

5.3 FAQs

Q: Can I run reindex for multiple tenants at once?
A: No, only one reindex per environment is supported at a time.
Q: Is it safe to run data import during reindex?
A: It is possible, but may impact performance. Monitor closely.

References and resources

Change log

2025-05-29: Initial release of the documentation.

Reindexing in mod-search: Full, Upload, and Failed Ranges