AWS/Kubernetes tags for billing
- kseniia_dubniak
Existing tags for monitoring costs in Kubecost
EKS tags:
- kubernetes_cluster = cluster_name - added to all EKS resources created by terraform-aws-eks Terraform module
RDS, MSK, ES tags:
- kubernetes_cluster = cluster_name (ex. folio-dev, folio-perf)
- kubernetes_namespace = namespace_name (ex. volaris, folijet)
- kubernetes_label_team = team_name (ex. volaris, folijet)
- kubernetes_service = name_of_service (ex., ES-Dashboard)
Resources created by Terraform
| Resource Name | Existing Tags | Add Tags | Notes |
|---|---|---|---|
| Cluster folder | |||
| EKS Cluster | Terraform = true Team = kitfox Project = folio | Terraform = true Team = kitfox Project = folio Env = folio-testing Region = us-west-2 | service tag cannot add https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf#L46 |
| Node Group | Terraform = true Team = kitfox Project = folio Name = folio-testing | Terraform = true Team = kitfox Project = folio Name = folio-testing Env = folio-testing Region = us-west-2 Service = node-group ? | Service tag in file terraform\rancher\cluster\eks.tf try to add tags. https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/node_groups.tf#L223 |
| EC2 | Terraform = true Team = kitfox Project = folio Name = folio-testing | Terraform = true Team = kitfox Project = folio Name = folio-testing Env = folio-testing Region = us-west-2 | https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf#L65 service tag cannot add |
| IAM roles | Terraform = true Team = kitfox Project = folio Name = load-balancer-controller-role | Terraform = true Team = kitfox Project = folio Name = load-balancer-controller-role Env = folio-testing Region = us-west-2 Service = iam-role | In file terraform\rancher\cluster\iam.tf in tags block add Service tag. |
| Network folder | |||
| VPC | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc Env = folio-testing Region = us-west-2 Service = vpc | Service tag https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/v3.14.0/main.tf#L31 |
| Private subnet | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-private-us-west-2c | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-private-us-west-2c Env = folio-testing Region = us-west-2 Service = subnet | In file terraform\rancher\network\main.tf in block private_subnet_tags add service tag |
| Public subnet | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-public-us-west-2c | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-public-us-west-2c Env = folio-testing Region = us-west-2 Service =subnet | In file terraform\rancher\network\main.tf in block public_subnet_tags add service tag |
| Database subnet | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-db-us-west-2c | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-db-us-west-2c Env = folio-testing Region = us-west-2 Service = subnet | In file terraform\rancher\network\main.tf in block database_subnet_tags add service tag |
| Elastic IP | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-nat-eip | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-nat-eip Env = folio-testing Region = us-west-2 Service = elastic-ip | In terraform\rancher\network\main.tf in aws_eip resource add tag service in tag block |
| Project folder | |||
| Security Group | Environment = dev Name = allow_es Terraform = true | Env = folio-testing Name = allow_es Terraform = true Region = us-west-2 Service = security_group | Changes in file terraform\rancher\project\elasticsearch.tf and in file terraform\rancher\project\kafka.tf |
| Amazon OpenSearch | Name = es-perf-folijet Terraform = true Service = ElasticSearch Version = 7.10 | Name = es-perf-folijet Terraform = true Service = ElasticSearch Version = 7.10 Region = us-west-2 Env = folio-testing | Changes in file terraform\rancher\project\elasticsearch.tf |
| Amazon MSK | Name = KAFKA-PERF-bulk-edit service = kafka | Name = kafka-bulk-edit Terraform = true Service = msk Version = 7.10 Region = us-west-2 Env = folio-testing | Changes in file terraform\rancher\project\kafka.tf |
| Amazon RDS | Terraform = true Environment = dev | Terraform = true Env = folio-testing Service = rds Region = us-west-2 | Changes in terraform\rancher\project\postgresql.tf Tags do not match with Terraform code. |
Terraform, Team, Project, Env, and Region are set up in variable.tf files in each folder.
Cluster folder
Module code https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf
Network module
Module docs https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/v3.14.0/main.tf
Monitor Kubernetes part costs
For monitoring costs in Kubernetes, we can use the Kubecost app.
Here is the official documentation for installation and configuration.
Using this tool we can monitor many resources by pods or PVC, namespaces.
Also, Kubecost used ElasticSearch, so we can use the Grafana dashboard https://grafana.com/grafana/dashboards/11270-kubecost/
Each team has a separate namespace in the cluster. So we can monitor team costs by namespace filter.
Kubecost link https://folio-testing-kubecost.ci.folio.org/allocations.html
Monitor by Namespace:
Report CSV file example: cumulative-cost-for-last-7-days-by-namespace-hiding-idle-1663240778166.csv
Kubecost configuration
SPOT instances
Kubecost will reconcile your spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be pulled hourly by integrating directly with the AWS spot feed.
For enabling hourly integration for SPOT:
- Create a bucket for SPOT logs
- Create user and grant read access to the bucket (copy API key and secret)
- In terraform\rancher\cluster\kubecost.tf file add values:
Additional info:
https://guide.kubecost.com/hc/en-us/articles/4407595928087#spot-data-feed-integration
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html
Authentification using AWS Cognito
Manually steps:
For cost saving, we use one User Pool for all our clusters, but different App Clients.
Before deploying Kubecost check that Kubecost user pool exists in AWS Cognito or create a new one.
Configuration for the user account, policies and others can be set up for project purposes.
Create a domain name using Cognito. The name prefix must be unique. In our configuration using folio-kubecost.
App client creates and configured by Terraform code automatically.