...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Terraform = true
Team = kitfox
Project = folio
...
Terraform = true
Team = kitfox
Project = folio
Env = folio-testing
Region = us-west-2
...
service tag cannot add
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf#L46
...
Terraform = true
Team = kitfox
Project = folio
Name = folio-testing
...
Terraform = true
Team = kitfox
Project = folio
Name = folio-testing
Env = folio-testing
Region = us-west-2
Service = node-group ?
...
Service tag in file terraform\rancher\cluster\eks.tf try to add tags.
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/node_groups.tf#L223
...
Terraform = true
Team = kitfox
Project = folio
Name = folio-testing
...
Terraform = true
Team = kitfox
Project = folio
Name = folio-testing
Env = folio-testing
Region = us-west-2
...
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf#L65
service tag cannot add
...
Terraform = true
Team = kitfox
Project = folio
Name = load-balancer-controller-role
...
Terraform = true
Team = kitfox
Project = folio
Name = load-balancer-controller-role
Env = folio-testing
Region = us-west-2
Service = iam-role
...
Terraform = true
Team = kitfox
Project = folio
Name = folio-rancher-vpc
...
Terraform = true
Team = kitfox
Project = folio
Name = folio-rancher-vpc
Env = folio-testing
Region = us-west-2
Service = vpc
Service tag https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/v3.14.0/main.tf#L31
...
Terraform = true
Team = kitfox
Project = folio
...
Existing tags for monitoring costs in Kubecost
EKS tags:
- kubernetes_cluster = cluster_name - added to all EKS resources created by terraform-aws-eks Terraform module
RDS, MSK, ES tags:
- kubernetes_cluster = cluster_name (ex. folio-dev, folio-perf)
- kubernetes_namespace = namespace_name (ex. volaris, folijet)
- kubernetes_label_team = team_name (ex. volaris, folijet)
- kubernetes_service = name_of_service (ex., ES-Dashboard)
Resources created by Terraform
| Resource Name | Existing Tags | Add Tags | Notes | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Cluster folder | |||||||||
| EKS Cluster | Terraform = true Team = kitfox Project = folio | Terraform = true Team = kitfox Project = folio | Name = folio-rancher-vpc-private-us-west-2cEnv = folio-testing Region = us-west-2 | In file terraform\rancher\network\main.tf in block private_subnet_tags add service tag | Public subnet | service tag cannot add https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf#L46 | |||
| Node Group | Terraform = true Team = kitfox Project = folio Name = folio- | rancher-vpc-public-us-west-2ctesting | Terraform = true Team = kitfox Project = folio Name = folio- | rancher-vpc-public-us-west-2ctesting Env = folio-testing Region = us-west-2 Service = | subnetIn node-group ? | Service tag in file terraform\rancher\ | networkcluster\ | mainin block | try to add tags. https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/node_groups.tf#L223 |
| EC2 | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-db-us-west-2cTerraform = truetesting | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-db-us-west-2ctesting Env = folio-testing Region = us-west-2 Service = subnet | In file terraform\rancher\network\main.tf in block database_subnet_tags add service tag | Elastic IP https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf#L65 service tag cannot add | |||||
| IAM roles | Terraform = true Team = kitfox Project = folio Name = folioload-rancherbalancer-vpccontroller-nat-eiprole | Terraform = true Team = kitfox Project = folio Name = folioload-rancherbalancer-vpccontroller-nat-eiprole Env = folio-testing Region = us-west-2 Service = elasticiam-iprole | In file terraform\rancher\networkcluster\mainiam.tf in aws_eip resource add tag service in tag block | ||||||
| Project folder | |||||||||
| Security Group | Environment = dev Name = allow_es Terraform = true | Env = folio-testing Name = allow_es Terraform = truetags block add Service tag. | |||||||
| Network folder | |||||||||
| VPC | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc Env = folio-testing Region = us-west-2 Service = security_group | Changes in file terraform\rancher\project\elasticsearch.tf and in file terraform\rancher\project\kafka.tf | Amazon OpenSearch | Name = es-perf-folijet Terraform = true Service = ElasticSearch Version = 7.10 | Name = es-perf-folijetvpc | Service tag https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/v3.14.0/main.tf#L31 | ||
| Private subnet | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-private-us-west-2c | Terraform = true Service Team = ElasticSearchkitfox Version Project = 7.10folio Region = Name = folio-rancher-vpc-private-us-west-22c Env = folio-testing Changes inRegion = us-west-2 Service = subnet | In file terraform\rancher\projectnetwork\elasticsearchmain.tf | Amazon MSK | Name = KAFKA-PERF-bulk-edit service = kafka | Name = kafka-bulk-edit in block private_subnet_tags add service tag | |||
| Public subnet | Terraform = true Service Team = mskkitfox Version Project = 7.10folio Region = Name = folio-rancher-vpc-public-us-west-22cEnv | Terraform = folio-testing | Changes in file terraform\rancher\project\kafka.tf | Amazon RDS | Terraform = true Environment = dev | Terraform = truetrue Team = kitfox Project = folio Name = folio-rancher-vpc-public-us-west-2c Env = folio-testing Service = rds Region = us-west-2 Changes in Service =subnet | In file terraform\rancher\projectnetwork\postgresqlmain.tf Tags do not match with Terraform code. | ||
Terraform, Team, Project, Env, and Region are set up in variable.tf files in each folder.
Cluster folder
Module code https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf
Network module
Module docs https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/v3.14.0/main.tf
Monitor Kubernetes part costs
For monitoring costs in Kubernetes, we can use the Kubecost app.
Here is the official documentation for installation and configuration.
Using this tool we can monitor many resources by pods or PVC, namespaces.
Also, Kubecost used ElasticSearch, so we can use the Grafana dashboard https://grafana.com/grafana/dashboards/11270-kubecost/
In files pipelines-shared-library\resources\helm can add Team label for each module:
Kubecost link https://folio-testing-kubecost.ci.folio.org/allocations.html
Monitor by Namespace:
Report CSV file example: cumulative-cost-for-last-7-days-by-namespace-hiding-idle-1663240778166.csv
...
| in block public_subnet_tags add service tag | |||
| Database subnet | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-db-us-west-2c | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-db-us-west-2c Env = folio-testing Region = us-west-2 Service = subnet | In file terraform\rancher\network\main.tf in block database_subnet_tags add service tag |
| Elastic IP | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-nat-eip | Terraform = true Team = kitfox Project = folio Name = folio-rancher-vpc-nat-eip Env = folio-testing Region = us-west-2 Service = elastic-ip | In terraform\rancher\network\main.tf in aws_eip resource add tag service in tag block |
| Project folder | |||
| Security Group | Environment = dev Name = allow_es Terraform = true | Env = folio-testing Name = allow_es Terraform = true Region = us-west-2 Service = security_group | Changes in file terraform\rancher\project\elasticsearch.tf and in file terraform\rancher\project\kafka.tf |
| Amazon OpenSearch | Name = es-perf-folijet Terraform = true Service = ElasticSearch Version = 7.10 | Name = es-perf-folijet Terraform = true Service = ElasticSearch Version = 7.10 Region = us-west-2 Env = folio-testing | Changes in file terraform\rancher\project\elasticsearch.tf |
| Amazon MSK | Name = KAFKA-PERF-bulk-edit service = kafka | Name = kafka-bulk-edit Terraform = true Service = msk Version = 7.10 Region = us-west-2 Env = folio-testing | Changes in file terraform\rancher\project\kafka.tf |
| Amazon RDS | Terraform = true Environment = dev | Terraform = true Env = folio-testing Service = rds Region = us-west-2 | Changes in terraform\rancher\project\postgresql.tf Tags do not match with Terraform code. |
Terraform, Team, Project, Env, and Region are set up in variable.tf files in each folder.
Cluster folder
Module code https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v18.26.6/main.tf
Network module
Module docs https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/v3.14.0/main.tf
Monitor Kubernetes part costs
For monitoring costs in Kubernetes, we can use the Kubecost app.
Here is the official documentation for installation and configuration.
Using this tool we can monitor many resources by pods or PVC, namespaces.
Also, Kubecost used ElasticSearch, so we can use the Grafana dashboard https://grafana.com/grafana/dashboards/11270-kubecost/
Each team has a separate namespace in the cluster. So we can monitor team costs by namespace filter.
Kubecost link https://folio-testing-kubecost.ci.folio.org/allocations.html
Monitor by Namespace:
Report CSV file example: cumulative-cost-for-last-7-days-by-namespace-hiding-idle-1663240778166.csv
Kubecost configuration
SPOT instances
Kubecost will reconcile your spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be pulled hourly by integrating directly with the AWS spot feed.
For enabling hourly integration for SPOT:
- Create a bucket for SPOT logs
- Create user and grant read access to the bucket (copy API key and secret)
- In terraform\rancher\cluster\kubecost.tf file add values:
Additional info:
https://guide.kubecost.com/hc/en-us/articles/4407595928087#spot-data-feed-integration
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html
Authentification using AWS Cognito
Manually steps:
For cost saving, we use one User Pool for all our clusters, but different App Clients.
Before deploying Kubecost check that Kubecost user pool exists in AWS Cognito or create a new one.
Configuration for the user account, policies and others can be set up for project purposes.
Create a domain name using Cognito. The name prefix must be unique. In our configuration using folio-kubecost.
App client creates and configured by Terraform code automatically.
| Info |
|---|
| Please, pay attention. If you change User Pool or domain prefix name change it also in code (terraform\rancher\cluster\kubecost.tf file). |