Cross Account AWS Managed Prometheus
Overview
A common pattern is to have a separate AWS Management Account to contain monitoring, logs etc but this can be complex to setup in AWS because of the requirement for cross-account IAM.
Managed Prometheus
AWS have a blog https://aws.amazon.com/blogs/opensource/setting-up-cross-account-ingestion-into-amazon-managed-service-for-prometheus/ but fail to supply a git repository with the terraform and kubernetes yaml for implementing this. I have created that repo:
In this example there are 2 AWS accounts Prod and Dev that have EKS clusters, and a Management AWS account with the managed prometheus workspaces for Prod and Dev and a AWS Managed Grafana that can access both workspaces. AWS Managed Prometheus is charged by the amount of data so not more costly to having multiple workspaces.
For more details see https://neillwturner.medium.com/aws-managed-prometheus-and-grafana-58d136bc2091
Managed Grafana
At time of writing AWS Managed Grafana is not supported by terraform so needs to be done manually in the AWS console see https://aws.amazon.com/blogs/mt/amazon-managed-grafana-getting-started/.
AWS Managed Grafana is charged by the number of users who logon each month so it’s cheaper to have just one Managed Grafana.
AWS recommends the Kubernetes apiserver(12006) dashboard for API server See https://grafana.com/grafana/dashboards/12006. This can be imported into Managed Grafana and when connected to the managed prometheus data source:
Alert Manager
Alert Manager is deployed to the Kubernetes clusters from the helm chart https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus
The alerts are configured in the values for the helm chart. It good to setup alerts for service levels for key EKS Kubernetes metrics for:
- EKS Control Plane
- Cluster State Metrics
- Resource Metrics
See https://epsagon.com/development/monitoring-amazon-eks/ for some ideas.
See https://gist.github.com/neillturner/45915fdbfb3359d7d98b97fee281eadb for an example of alerting rules.
CloudWatch Container Insights Monitoring
Container Insights on Amazon EKS and Kubernetes when setup shows some good pre-built dashboards in the AWS console. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html It automates the discovery of Prometheus metrics from containerized systems and workloads. Ingests custom metrics in CloudWatch, includes pre-built dashboards.
Confusingly there are several ways to setup this. Setting it up as a daemonset that runs on each node seems to be the way to get the performance information required. Setup by following Set up the CloudWatch agent to collect cluster metrics in https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-metrics.html
