Earlier, Custom metrics and application logs are pushed to CloudWatch using the CloudWatch monitoring scripts. They are now deprecated. AWS recommends using the CloudWatch agent to collect metrics and logs.
AWS CloudWatch service has the capability to store custom logs and process metrics generated from application instances.
Use cases for custom metrics and logs:
- EC2 instance custom metrics (Disk, Memory, etc)can be pushed to CloudWatch.
- Webserver (Nginx, Apache, etc) access or error logs can be pushed to Cloudwatch logs which acts as central log management for applications running on AWS.
Pushing Custom Metrics & Application Logs To AWS Cloudwatch Workflow
- Create a custom EC2 IAM role with CloudWatch log write access
- Install CloudWatch agent & Pushing Custom Metrics to CloudWatch
- Configure the log path in the CloudWatch agent log configuration file.
- Start the CloudWatch agent with the log configuration file.
- Validate logs in the CloudWatch console.
Create a custom EC2 IAM role with CloudWatch log write access
Create and add a custom EC2 IAM role to the instance. This IAM role will have policies with write access to the CloudWatch service so that all the logs from EC2 instances can be shipped to CloudWatch.
Attach the following policy to the Role.
| { “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Action”: [ “logs:CreateLogGroup”, “logs:CreateLogStream”, “logs:PutLogEvents”, “logs:DescribeLogStreams” ], “Resource”: [ “arn:aws:logs:*:*:*” ] } ] } |
Install CloudWatch agent & Pushing custom metrics to Cloudwatch
Use the following commands – Amazon Linux 2
| #!/bin/bash sudo yum install amazon-cloudwatch-agent -y cd /opt/aws/amazon-cloudwatch-agent/etc sudo wget https://metrics.s3.ap-southeast-2.amazonaws.com/instancemetrics.json sudo chmod 777 instancemetrics.json sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/instancemetrics.json systemctl restart amazon-cloudwatch-agent.service |
Use the following commands – Ubuntu
| #!/bin/bash sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb sudo dpkg -i -E ./amazon-cloudwatch-agent.deb sudo apt-get update && sudo apt-get install collectd -y cd /opt/aws/amazon-cloudwatch-agent/etc sudo wget https://metrics.s3.ap-southeast-2.amazonaws.com/instancemetrics.json sudo chmod 777 instancemetrics.json sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/instancemetrics.json sudo service amazon-cloudwatch-agent start |
You can view the custom metrics in the CloudWatch Console (Metrics → All Metrics → Custom namespaces → Custom Metrics).
Configure the log path in the CloudWatch agent log configuration file
- Go to /opt/aws/amazon-cloudwatch-agent/etc
- You can find log-config.json
- Modify the log file path in the following and Paste it in the log-config.json
| { “agent”: { “metrics_collection_interval”: 10, “run_as_user”: “root” }, “logs”: { “logs_collected”: { “files”: { “collect_list”: [ { “file_path”: “/var/log/httpd/access*”, “log_group_name”: “apache-access-logs”, “log_stream_name”: “{instance_id}” }, { “file_path”: “/var/log/httpd/error*”, “log_group_name”: “apache-error-logs”, “log_stream_name”: “{instance_id}” } ] } } } } |
Start the CloudWatch agent with the log configuration file
Use the following command to start the CloudWatch agent with the modified log-config.json file.
| sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/log-config.json -s |
Validating Logs in the Cloudwatch Console
Once the setup is done, you can view all the configured logs under the CloudWatch Console.
- Go to Logs → Log groups and you will see the log group mentioned in the log configuration file.
- Select the log group and you can see the instance ID mentioned in the config.
