Setup AWS Centralised Logs with Terraform
AWS ElasticSearch Service, Lambda and CloudWatch logs can be used to provide a simple and cost-effective centralised log management service.
+------------+ +----------+ +----------------+
| Server | |Cloudwatch| lambda | ElasticSearch | clean
| Instance | +---> Logs +----------> Service <-----
++----------++ | | | | |lambda
||Cloudwatch|| | +----------+ | |
|| Agent |-----+ | |
|+----------+| +----------+ lambda | +--------+ |
+------------+ |ELB Logs +----------> | Kibana | |
| (S3) | | | | |
| | | +----^---+ |
+----------+ +---------|------+
|
|
+----------------|------+
|Client +---------+ |
|Work- |Proxy (optional)
|station +----^----+ |
| | |
| +----+----+ |
| | | |
| | Browser | |
| | | |
| +---------+ |
+-----------------------+The sample terraform Terraform AWS Centralised Logs shows how simple it is to create a centralised logging using AWS ElasticSearch service, lambda and CloudWatch logs. It calls 3 terraform registry modules:
- egarbi/es-cluster — Create the AWS elastic search cluster.
- neillturner/lambda-es-cleanup — Lambda to cleanup at 1am each morning delete old logs data.
- neillturner/elb-logs-to-elasticsearch — Lambda to load ELB logs from S3 to elasticsearch cluster.
1. Server instances need to be setup to send logs to AWS CloudWatch logs via the the AWS CloudWatch logs agent. See CloudWatch logs GettingStarted
2. Currently manually in the AWS console each CloudWatch log needs to be configured to call the AWS supplied lambda to load the data into the AWS ElasticSearch cluster. See Streaming CloudWatch Logs Data to Amazon ElasticSearch Service
3. The AWS ElasticSearch service should be setup in a private VPC so it cannot be accessed via the public internet.
4. Additional security can be setup by using a proxy:
- How to Control Access to Your Amazon ElasticSearch Service Domain
- Configuring AWS ElasticSearch/Kibana Proxy
- Setting up Kibana proxy for AWS ElasticSearch
5. Finally the lambdas have limitations in the amount of data they can process. To overcome limitations use Kinesis Firehose.
See:
