| Table of Contents |
|---|
...
** - up to 10 items were discarded with the error: io.vertx.core.impl.NoStackTraceThrowable: Cannot get actual Item by id: org.folio.inventory.exceptions.InternalServerErrorException: Access for user 'data-import-system-user' (f3486d35-f7f7-4a69-bcd0-d8e5a35cb292) requires permission: inventory-storage.items.item.get
...
- tenant0_mod_source_record_storage.marc_records_lb = 9674629
- tenant2_mod_source_record_storage.marc_records_lb = 0
- tenant3_mod_source_record_storage.marc_records_lb = 0
- tenant0_mod_source_record_storage.raw_records_lb = 9604805
- tenant2_mod_source_record_storage.raw_records_lb = 0
- tenant3_mod_source_record_storage.raw_records_lb = 0
- tenant0_mod_source_record_storage.records_lb = 9674677
- tenant2_mod_source_record_storage.records_lb = 0
- tenant3_mod_source_record_storage.records_lb = 0
- tenant0_mod_source_record_storage.marc_indexers = 620042011
- tenant2_mod_source_record_storage.marc_indexers = 0
- tenant3_mod_source_record_storage.marc_indexers = 0
- tenant0_mod_source_record_storage.marc_indexers with field_no 010 = 3285833
- tenant2_mod_source_record_storage.marc_indexers with field_no 010 = 0
- tenant3_mod_source_record_storage.marc_indexers with field_no 010 = 0
- tenant0_mod_source_record_storage.marc_indexers with field_no 035 = 19241844
- tenant2_mod_source_record_storage.marc_indexers with field_no 035 = 0
- tenant3_mod_source_record_storage.marc_indexers with field_no 035 = 0
- tenant0_mod_inventory_storage.authority = 4
- tenant2_mod_inventory_storage.authority = 0
- tenant3_mod_inventory_storage.authority = 0
- tenant0_mod_inventory_storage.holdings_record = 9592559
- tenant2_mod_inventory_storage.holdings_record = 16
- tenant3_mod_inventory_storage.holdings_record = 16
- tenant0_mod_inventory_storage.instance = 9976519
- tenant2_mod_inventory_storage.instance = 32
- tenant3_mod_inventory_storage.instance = 32
- tenant0_mod_inventory_storage.item = 10787893
- tenant2_mod_inventory_storage.item = 19
- tenant3_mod_inventory_storage.item = 19
PTF -environment ocp3
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731 - MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- Kafka topics partitioning: - 2 partitions for DI topics
...
| Module ocp3-pvt Mon Sep 11 09:33:28 UTC 2023 | Task Def. Revision | Module Version | Task Count | Mem Hard Limit | Mem Soft limit | CPU units | Xmx | MetaspaceSize | MaxMetaspaceSize | R/W split enabled |
|---|---|---|---|---|---|---|---|---|---|---|
| mod-remote-storage | 13 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-remote-storage:2.0.3 | 2 | 4920 | 4472 | 1024 | 3960 | 512 | 512 | false |
| mod-agreements | 8 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-agreements:5.5.2 | 2 | 1592 | 1488 | 128 | 968 | 384 | 512 | false |
| mod-data-import | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-mod-data-import:2.7.1 | 1 | 2048 | 1844 | 256 | 1292 | 384 | 512 | false |
| mod-search | 30 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-search:2.0.1 | 2 | 2592 | 2480 | 2048 | 1440 | 512 | 1024 | false |
| mod-authtoken | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-authtoken:2.13.0 | 2 | 1440 | 1152 | 512 | 922 | 88 | 128 | false |
| mod-configuration | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-mod-configuration:5.9.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | false |
| mod-inventory-storage | 1 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory-storage:26.1.0-SNAPSHOT.665 | 0 | 2208 | 1952 | 1024 | 1440 | 384 | 512 | false |
| mod-circulation-storage | 15 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-circulation-storage:16.0.1 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | false |
| mod-source-record-storage | 11 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-mod-source-record-storage:5.6.7 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | false |
| mod-calendar | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-calendar:2.4.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | false |
| mod-inventory | 12 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-inventory:20.0.6 | 2 | 2880 | 2592 | 1024 | 1814 | 384 | 512 | false |
| mod-circulation | 9 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-mod-circulation:23.5.6 | 2 | 2880 | 2592 | 1536 | 1814 | 384 | 512 | false |
| mod-di-converter-storage | 8 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-di-converter-storage:2.0.5 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | false |
| mod-pubsub | 8 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-pubsub:2.9.1 | 2 | 1536 | 1440 | 1024 | 922 | 384 | 512 | false |
| mod-users | 8 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-mod-users:19.1.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | false |
| mod-patron-blocks | 8 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron-blocks:1.8.0 | 2 | 1024 | 896 | 1024 | 768 | 88 | 128 | false |
| mod-source-record-manager | 9 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-source-record-manager:3.6.4 | 2 | 5600 | 5000 | 2048 | 3500 | 384 | 512 | false |
| nginx-edge | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-nginx-edge:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | false |
| mod-quick-marc | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-quick-marc:3.0.0 | 1 | 2288 | 2176 | 128 | 1664 | 384 | 512 | false |
| nginx-okapi | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/nginx-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 0 | 0 | 0 | false |
| okapi-b | 8 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/okapi:5.okapi:5.0.1 | 3 | 1684 | 1440 | 1024 | 922 | 384 | 512 | false |
| mod-feesfines | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-feesfines:18.2.1 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | false |
| mod-patron | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-patron:5.5.2 | 2 | 1024 | 896 | 128 | 768 | 88 | 128 | false |
| mod-notes | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/mod-notes:5.0.1 | 2 | 1024 | 896 | 128 | 952 | 384 | 512 | false |
| pub-okapi | 7 | 579891902283.dkr.ecr.us-east-1.amazonaws.com/folio/pub-okapi:2023.06.14 | 2 | 1024 | 896 | 128 | 768 | 0 | 0 | false |
Methodology/Approach
Test 1: Manually tested 100k, 250k, and 500k records files started one by one on one tenant only.
...