| Table of Contents |
|---|
Overview
This document contains the results of testing Data Import for MARC Bibliographic records in the Orchid release to detect the range of duration of each DI test for ncp5.
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
...
Duration ncp5 test1 | Duration ncp5 test2 | Duration ncp5 test3 | Duration ncp5 test4 | Duration ncp5 test5 | Results | AVG | Max Dev from AVG | R/W split enabled * | R/W split enabled ** | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
1K MARC Create | PTF - Create 2 | 34 sec | 38 sec | 30 sec | 31 sec | 31 sec | Completed | 33 sec | 15.8% | 30 sec | 29 sec |
| 5K MARC Create | PTF - Create 2 | 2 min 27 sec | 2 min 21 sec | 4 min 42 sec | 2 min 21 sec | 2 min 21 sec | Completed | 2 min 50 sec | 65.4% | 2 min 23 sec | 2 min 3 sec |
10K MARC Create | PTF - Create 2 | 4 min 47 sec | 4 min 34 sec | 4 min 50 sec | 4 min 41 sec | 4 min 41 sec | Completed | 4 min 43 sec | 2.6% | 5 min 12 sec | 3 min 58 sec |
| 25K MARC Create | PTF - Create 2 | 11 min 29 sec | 11 min 27 sec | 12 min | 11 min 43 sec | 12 min 40 sec | Completed | 11 min 52 sec | 6.7% | 11 min 45 sec | 10 min 5 sec |
| 50K MARC Create | PTF - Create 2 | 23 min 02 sec | 22 min 14 sec | 23 min 40 sec | 23 min 55 sec | 23 min 47 sec | Completed | 23 min 20 sec | 2.5% | 23 min 36 sec | 20 min 46 sec |
| 100K MARC Create | PTF - Create 2 | 48 min 36 sec | 47 min 47 sec | 47 min 50 sec | 50 min 37 sec | 49 min | Completed | 48 min 46 sec | 3.8% | 49 min 28 sec | 44 min 18 sec |
| 5K MARC Update | PTF - Updates Success - 1 | 2 min 48 sec | 2 min 45 sec | ||||||||
| 10K MARC Update | PTF - Updates Success - 1 | 5 min 23 sec | 5 min 23 sec | ||||||||
| 25K MARC Update | PTF - Updates Success - 1 | 14 min 12 sec | 14 min 19 sec | ||||||||
| 50K MARC Update | PTF - Updates Success - 1 | 27 min 52 sec | 28 min | ||||||||
| 100K MARC Update | PTF - Updates Success - 1 | 57 min 41 sec | 55 min |
* - enabled for
- mod-data-import
- mod-source-record-storage
- mod-source-record-manager
- mod-di-converter-storage
...
Infrastructure ncp5
PTF -environment ncp5
- 10 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
2 database instances, one reader, and one writer
Name API Name Memory GIB vCPUs max_connections R6G Extra Large db.r6g.xlarge 32 GiB 4 vCPUs 2731 - MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- Kafka topics partitioning: - 2 partitions for DI topics
...
Modules | Version | Task Definition | Running Tasks | CPU | Memory | MemoryReservation | MaxMetaspaceSize | Xmx |
|---|---|---|---|---|---|---|---|---|
| mod-inventory-storage | 26.0.0 | 12 | 2 | 2048 | 4096 | 3690 | 512 | 3076 |
| mod-inventory | 20.0.4 | 1 | 2 | 1024 | 2880 | 2592 | 512 | 1814 |
| mod-source-record-storage | 5.6.7 | 27 | 2 | 2048 | 5600 | 5000 | 512 | 3500 |
| mod-quick-marc | 3.0.0 | 6 | 1 | 128 | 2288 | 2176 | 512 | 1664 |
| mod-source-record-manager | 3.6.4 | 18 | 2 | 2048 | 5600 | 5000 | 512 | 3500 |
| mod-di-converter-storage | 2.0.5 | 8 | 2 | 128 | 1024 | 896 | 128 | 768 |
| mod-data-import | 2.7.1 | 10 | 1 | 256 | 2048 | 1844 | 512 | 1292 |
| okapi | 5.0.1 | 8 | 3 | 1024 | 1684 | 1440 | 512 | 922 |
| nginx-okapi | 2023.06.14 | 7 | 2 | 128 | 1024 | 896 | - | - |
| pub-okapi | 2023.06.14 | 7 | 2 | 128 | 1024 | 896 | - | 768 |
...