| Table of Contents | ||
|---|---|---|
|
...
Increasing the number of connections for mod-srs & mod-srm to 500 (even to 100) causes database memory usage errors.
Test Runs
| Aurora PostgreSQL Changes | DI duration with 30 mod-srs & mod-srm connections | Status | DI duration with 500 mod-srs & mod-srm connections | ACU | Average ACUs per test | Mem (GiB) | Price per hour $ | Baseline price per month $ | Additional expenses |
| Serverless v2 (0.5 - 128 ACUs) | 1 hour 3 min | Completed with errors | stopped due to DB error | 0.5 | 84 | 1 | 0.09(per 0.5 ACU) | 64.80 | +Price per additionally used ACUs |
| Serverless v2 (32 - 128 ACUs) | 34 min | Completed | 20 min hanging 0% completed DB error | 32 | 99 | 64 | 0.18(per 1 ACU) | 4,147.20 | +Price per additionally used ACUs |
| db.r6g.8xlarge | 18 min | Completed | stopped due to DB error | - | - | 256 | 3.597 | 2,589.84 | No additional expenses |
| db.r6g.4xlarge | 29 min | Completed | stopped due to DB error | - | - | 128 | 1.798 | 1,294.56 | No additional expenses |
| db.r6g.2xlarge | 50 min | Completed | stopped due to DB error | - | - | 64 | 0.889 | 640.00 | No additional expenses |
| db.r6g.xlarge | 2 hours -20% done. | Could be completed with errors due to DB error | stopped due to DB error | - | - | 32 | 0.45 | 324.00 | No additional expenses |
| Serverless v2 (16 - 96 ACUs) | 45 min | Completed | - | 16 | 63 | 32 | 0.18(per 1 ACU) | 2,073.60 | +Price per additionally used ACUs |
| Serverless v2 (8 - 96 ACUs) | 36 min (25 min when started right after the first one) | Completed | - | 8 | 83 | 16 | 0.18(per 1 ACU) | 1,036.80 | +Price per additionally used ACUs |
* Note that the average ACU utilization for the database is approximately 4 ACUs without any activities in the environment. And each job, test, or any activity will cause additional ACUs utilization.
...
- The scaling rate for an Aurora Serverless v2 DB instance depends on its current capacity. The higher the current capacity (minimal ACU), the faster it can scale up. If you need the DB instance to quickly scale up to a very high capacity, consider setting the minimum capacity to a value where the scaling rate meets your requirement. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.setting-capacity.html
When DB is already scaled up DI duration decreases( for example from 36 to 25 min when started right after the first one for Serverless v2 (8 - 96 ACUs)
- Although it is highly benefitted to consider the permissible minimum ACU (0.5) (because the baseline
price per month is the smallest) as the minimum capacity, it will have challenges in terms of scaling the capacity. More info https://dev.to/aws-builders/a-guide-on-selecting-amazon-aurora-serverless-and-provisioned-database-cluster-1e1h
For easier calculation of database price use https://calculator.aws/#/addService/AuroraPostgreSQL. Need to place average ACU utilization per month to get more accurate results.
...
On these graphs, we can observe gaps that correspond to DB issues with increased connections for mod-srs and mod-srm to 500.
Appendix
Infrastructure
Records count :
- mod_source_record_storage.marc_indexers = 1028022472
PTF -environment cptf2
- 6 m6i.2xlarge EC2 instances located in US East (N. Virginia)us-east-1
1 database instance one writer only
- MSK ptf-kakfa-3
- 4 m5.2xlarge brokers in 2 zones
Apache Kafka version 2.8.0
EBS storage volume per broker 300 GiB
- auto.create.topics.enable=true
- log.retention.minutes=480
- default.replication.factor=3
- 2 partitions per DI topic