...
AWS recommends at least 5MB worth of data passed into the bulk api allow efficient processing of data. mod-search currently send less than 800KB in general. It is such a small size because mod-search is limited to 50 records when retrieving instance Ids from Kafka. The poll record count was reduced to 50 due to a downstream issue of mod-search querying mod-inventory-storage for the latest instance, the query(via CQL) appends IDs to the URI of the HTTP call[MSEARCH-75]. Querying more than 50 records throws an HTTP exception of having an HTTP call with a URI that is too long.
...