{"id":1674,"date":"2022-11-10T10:30:19","date_gmt":"2022-11-10T10:30:19","guid":{"rendered":"http:\/\/cloudaliv.com\/stage\/?p=1674"},"modified":"2023-08-24T13:34:10","modified_gmt":"2023-08-24T13:34:10","slug":"pushing-custom-metrics-application-logs-to-cloudwatch-using-cloudwatch-agent","status":"publish","type":"post","link":"https:\/\/cloudaliv.com\/stage\/pushing-custom-metrics-application-logs-to-cloudwatch-using-cloudwatch-agent\/","title":{"rendered":"AWS CloudWatch: Custom Metrics & Logs"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

<\/p>\n

Earlier, Custom metrics and application logs are pushed to CloudWatch using the CloudWatch monitoring scripts. They are now deprecated. AWS recommends using the CloudWatch agent to collect metrics and logs.<\/p>\n

\"\"<\/p>\n

<\/p>\n

<\/p>\n

AWS CloudWatch service has the capability to store custom logs and process metrics generated from application instances.<\/p>\n

<\/p>\n

<\/p>\n

Use cases for custom metrics and logs:<\/p>\n

<\/p>\n

<\/p>\n

    \n
  1. EC2 instance custom metrics (Disk, Memory, etc)can be pushed to CloudWatch.<\/li>\n
  2. Webserver (Nginx, Apache, etc) access or error logs can be pushed to Cloudwatch logs which acts as central log management for applications running on AWS.<\/li>\n<\/ol>\n

    <\/p>\n

    <\/p>\n

    Pushing Custom Metrics & Application Logs To AWS Cloudwatch Workflow<\/strong><\/h6>\n

    <\/p>\n

    <\/p>\n

      \n
    1. Create a custom EC2 IAM role with CloudWatch log write access<\/li>\n
    2. Install CloudWatch agent & Pushing Custom Metrics to CloudWatch<\/li>\n
    3. Configure the log path in the CloudWatch agent log configuration file.<\/li>\n
    4. Start the CloudWatch agent with the log configuration file.<\/li>\n
    5. Validate logs in the CloudWatch console.<\/li>\n<\/ol>\n

      <\/p>\n

      <\/p>\n

      \"\"<\/figure>\n

      <\/p>\n

      <\/p>\n

      Create a custom EC2 IAM role with CloudWatch log write access<\/strong><\/h6>\n

      <\/p>\n

      <\/p>\n

      Create and add a custom EC2 IAM role to the instance. This IAM role will have policies with write access to the CloudWatch service so that all the logs from EC2 instances can be shipped to CloudWatch.<\/p>\n

      <\/p>\n

      <\/p>\n

      Attach the following policy to the Role.<\/p>\n

      <\/p>\n

      <\/p>\n

      \n\n\n\n
      {
        “Version”: “2012-10-17”,
        “Statement”: [
          {
            “Effect”: “Allow”,
            “Action”: [
              “logs:CreateLogGroup”,
              “logs:CreateLogStream”,
              “logs:PutLogEvents”,
              “logs:DescribeLogStreams”
          ],
            “Resource”: [
              “arn:aws:logs:*:*:*”
          ]
        }
      ]
      }<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

      <\/p>\n

      <\/p>\n

      Install CloudWatch agent & Pushing custom metrics to Cloudwatch<\/strong><\/h6>\n

      <\/p>\n

      <\/p>\n

      Use the following commands – Amazon Linux 2<\/strong><\/p>\n

      <\/p>\n

      <\/p>\n

      \n\n\n\n
      #!\/bin\/bash
      sudo yum install amazon-cloudwatch-agent -y
      cd \/opt\/aws\/amazon-cloudwatch-agent\/etc
      sudo wget https:\/\/metrics.s3.ap-southeast-2.amazonaws.com\/instancemetrics.json
      sudo chmod 777 instancemetrics.json
      sudo \/opt\/aws\/amazon-cloudwatch-agent\/bin\/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:\/opt\/aws\/amazon-cloudwatch-agent\/etc\/instancemetrics.json
      systemctl restart amazon-cloudwatch-agent.service<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

      <\/p>\n

      <\/p>\n

      Use the following commands – Ubuntu<\/strong><\/p>\n

      <\/p>\n

      <\/p>\n

      \n\n\n\n
      #!\/bin\/bash
      sudo wget https:\/\/s3.amazonaws.com\/amazoncloudwatch-agent\/debian\/amd64\/latest\/amazon-cloudwatch-agent.deb
      sudo dpkg -i -E .\/amazon-cloudwatch-agent.deb
      sudo apt-get update && sudo apt-get install collectd -y
      cd \/opt\/aws\/amazon-cloudwatch-agent\/etc
      sudo wget https:\/\/metrics.s3.ap-southeast-2.amazonaws.com\/instancemetrics.json
      sudo chmod 777 instancemetrics.json
      sudo \/opt\/aws\/amazon-cloudwatch-agent\/bin\/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:\/opt\/aws\/amazon-cloudwatch-agent\/etc\/instancemetrics.json
      sudo service amazon-cloudwatch-agent start<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

      <\/p>\n

      <\/p>\n

      You can view the custom metrics in the CloudWatch Console (Metrics \u2192 All Metrics \u2192 Custom namespaces \u2192 Custom Metrics).<\/p>\n

      <\/p>\n

      <\/p>\n

      Configure the log path in the CloudWatch agent log configuration file<\/strong><\/h6>\n

      <\/p>\n

      <\/p>\n

        \n
      • Go to \/opt\/aws\/amazon-cloudwatch-agent\/etc<\/li>\n
      • You can find log-config.json<\/li>\n
      • Modify the log file path in the following and Paste it in the log-config.json<\/li>\n<\/ul>\n

        <\/p>\n

        <\/p>\n

        \n\n\n\n
        {
            “agent”: {
              “metrics_collection_interval”: 10,
              “run_as_user”: “root”
            },
            “logs”: {
              “logs_collected”: {
                “files”: {
                  “collect_list”: [
                    {
                      “file_path”: “\/var\/log\/httpd\/access*”,
                      “log_group_name”:  “apache-access-logs”,
                      “log_stream_name”: “{instance_id}”
                    },
                    {
                        “file_path”: “\/var\/log\/httpd\/error*”,
                        “log_group_name”: “apache-error-logs”,
                        “log_stream_name”: “{instance_id}”
                    }
                  ]
                }
              }
            }
          }<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

        <\/p>\n

        <\/p>\n

        Start the CloudWatch agent with the log configuration file<\/strong><\/h6>\n

        <\/p>\n

        <\/p>\n

        Use the following command to start the CloudWatch agent with the modified log-config.json file.<\/p>\n

        <\/p>\n

        <\/p>\n

        \n\n\n\n
        sudo \/opt\/aws\/amazon-cloudwatch-agent\/bin\/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:\/opt\/aws\/amazon-cloudwatch-agent\/etc\/log-config.json -s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

        <\/p>\n

        <\/p>\n

        Validating Logs in the Cloudwatch Console<\/strong><\/h6>\n

        <\/p>\n

        <\/p>\n

        Once the setup is done, you can view all the configured logs under the CloudWatch Console.<\/p>\n

        <\/p>\n

        <\/p>\n

          \n
        • Go to Logs \u2192 Log groups and you will see the log group mentioned in the log configuration file.<\/li>\n
        • Select the log group and you can see the instance ID mentioned in the config.<\/li>\n<\/ul>\n

          <\/p>\n

          <\/p>\n

          <\/p>\n

          <\/p>\n

          <\/p>\n

          <\/p>\n

          <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

          Earlier, Custom metrics and application logs are pushed to CloudWatch using the CloudWatch monitoring scripts. They are now deprecated. AWS recommends using the CloudWatch agent to collect metrics and logs. AWS CloudWatch service has the capability to store custom logs and process metrics generated from application instances. Use cases for custom metrics and logs: EC2 […]<\/p>\n","protected":false},"author":1,"featured_media":1679,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[12],"tags":[19],"class_list":["post-1674","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","tag-aws"],"acf":[],"_links":{"self":[{"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/posts\/1674","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/comments?post=1674"}],"version-history":[{"count":16,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/posts\/1674\/revisions"}],"predecessor-version":[{"id":4255,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/posts\/1674\/revisions\/4255"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/media\/1679"}],"wp:attachment":[{"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/media?parent=1674"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/categories?post=1674"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cloudaliv.com\/stage\/wp-json\/wp\/v2\/tags?post=1674"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}