error from logs
WARN Fetcher [Consumer clientId=consumer-DI_INVENTORY_HOLDING_CREATED.mod-inventory-18.0.4-33, groupId=DI_INVENTORY_HOLDING_CREATED.mod-inventory-18.0.4] Received unknown topic or partition error in fetch for partition kcp1.Default.fs09000000.DI_INVENTORY_HOLDING_CREATED-0
@timestamp1642416090190
KIWI | KIWI (with OL) | KIWI with partitions N# 2 | KIWI with partitions N# 4 | ||
|---|---|---|---|---|---|
5K MARC Create | PTF - Create 2 | 5 min, 8 min | 8 min | 5 min | 5,7 min |
5K MARC Update | PTF - Updates Success - 1 | 11 min, 13 min | 6 min | 7 min | 6 min |
10K MARC Create | PTF - Create 2 | 11 min , 14 min | 12 min | 10 min | 16 min |
10K MARC Update | PTF - Updates Success - 1 | 22 min, 24 min | 15 min | 11 min | failed |
| 25K MARC Create | PTF - Create 2 | 23 mins, 25 mins, 26 mins | 24 min | 23 min | 25 min |
| 25K MARC Update | PTF - Updates Success - 1 | 1 hour 20 mins (completed with errors) *, 56 mins | 40 min | failed | failed |
| 50K MARC Create | PTF - Create 2 | Completed with errors, 1 hr 40 mins | 43 min | - | |
| 50K Update | PTF - Updates Success - 1 | 2 hr 32 mins (job stuck at 76% completion) | 1hr 4min | - |
number of partitions - 2
memory usage
org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit cannot be completed since the consumer is undergoing a rebalance for auto partition assignment. You can try completing the rebalance by calling poll() and then retry the operation.
| 5,000 recordsBegan 7:33 AM today | 1/2 |
| 5,000 recordsBegan 7:59 AM today | 1/25/2022, 8:06 AM |
| 10,000 recordsBegan 8:20 AM today | 11/25/2022, 8:36 AM |
| 25,000 recordsBegan 8:45 AM today | |
| 25,000 recordsBegan 12:55 PM today | 1/25/2022, 1:20 PM |
| 50,000 recordsBegan 2:35 PM | 50,000 recordsBegan 2:35 PM |
| |
| 5,000 recordsBegan 12:11 PM today | 1/26/2022, 12:16 PM |
| 1/5,000 recordsBegan 1:13 PM today | 1/26/2022, 1:19 PM |
| 10,000 recordsBegan 1:31 PM today |
2 partitions
| CREATE 5,000 recordsBegan 8:20 AM | 1/28/2022, 8:26 AM | |
| CREATE10,000 recordsBegan 8:48 AM | 1/28/2022, 9:00 AM | |
| update 10,000 recordsBegan 9:17 AM | FAILED | |
| CREATE 25,000 recordsBegan 9:37 AM Completed with errors | 1/28/2022, 10:04 AM | fs09000000_mod_inventory_storage.item - 24996 fs09000000_mod_inventory_storage.holdings_record - 24996 fs09000000_mod_inventory_storage.instance - 25000 fs09000000_mod_source_record_storage.records_lb - 25000 |
| restart mods and clean Kafka MQ | ||
| create 25,000 recordsBegan 10:31 AM today | 1/28/2022, 10:57 AM | |
| restart mods and clean Kafka MQ | ||
50,000 recordsBegan 11:18 AM Completed with errors | 1/28/2022, 12:43 PM | fs09000000_mod_inventory_storage.item 49284 fs09000000_mod_inventory_storage.holdings_record 48684 fs09000000_mod_source_record_storage.records_lb 50000 fs09000000_mod_inventory_storage.instance 50000 |