Logstash CloudWatch input plugin Updated

We’ve previously released the Logstash CloudWatch Input plugin to fetch CloudWatch metrics from AWS. Since then we’ve realized that it’s not as complete or as configurable as we’d like it to be. So we’ve refactored it significantly, making sure that it properly supports specific metrics. We’ve also increased the flexibility around for configuring those metrics, and made it Logstash 2 compatible. So, available immediately, we’re announcing version 1.0.0 of the Logstash CloudWatch Input plugin.

Changes

Configuration

The most significant change is in how you configure the plugin. The previous version tried to give you a generic configuration for all namespaces. This failed miserable since all the different namespaces require different configurations. From version 1.0.0 the configuration options have changed, and you’ll need to set up an input per CloudWatch namespace.

The instances, tag_name and tag_value settings all fall away.

Supported Namespaces

Unfortunately it’s not possible to create a “one shoe fits all” solution for fetching metrics from AWS. We need to specifically add support for every namespace. This takes time so we’ll be adding support for namespaces as the requests for them come in and we get time to do it. Please check the metric support issues for already requested namespaces, and add your request if it’s not there yet.

Example Config

Just note that the below configuration doesn’t contain the AWS API access information. See the post on Logstash AWS options on how to set it up.

     input {
       cloudwatch {
         namespace => "AWS/EC2"
         metrics => [ "CPUUtilization" ]
         filters => { "tag:Monitoring" => "Yes" }
         region => "us-east-1"
       }
     }

     input {
       cloudwatch {
         namespace => "AWS/EBS"
         metrics => ["VolumeQueueLength"]
         filters => { "tag:Monitoring" => "Yes" }
         region => "us-east-1"
       }
     }

     input {
       cloudwatch {
         namespace => "AWS/RDS"
         metrics => ["CPUUtilization", "CPUCreditUsage"]
         filters => { "EngineName" => "mysql" } # Only supports EngineName, DatabaseClass and DBInstanceIdentifier
         region => "us-east-1"
       }
     }

See AWS Developer Guide for more information on namespaces and metrics.

The plugin is available on Logstash 1.5 and up, including Logstash 2. It can be installed using the plugin utility:

bin/plugin install logstash-input-cloudwatch

Minimum Configuration

The plugin uses the Logstash AWS config settings and requires an AIM profile that has access to CloudWatch metrics. The following policy is the minimum you need to get started with EC2 metrics:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1444715676000",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1444716576170",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        }
    ]
}

You need to specify what resources you want to track using the filters setting. The specifics differ for each namespace, but generally follows the format of the [filter][6] option in AWS calls:

EC2

input {
  cloudwatch {
    namespace => "AWS/EC2"
    metrics => [ "CPUUtilization" ]
    filters => { "tag:Monitoring" => "Yes" }
    region => "us-east-1"
  }
}

RDS

input {
  cloudwatch {
    namespace => "AWS/RDS"
    metrics => ["CPUUtilization", "CPUCreditUsage"]
    filters => { "EngineName" => "mysql" } # Only supports EngineName, DatabaseClass and DBInstanceIdentifier
    region => "us-east-1"
  }
}

EBS

input {
  cloudwatch {
    namespace => "AWS/EBS"
    metrics => ["VolumeQueueLength"]
    filters => { "tag:Monitoring" => "Yes" }
    region => "us-east-1"
  }
}

ELB

input {
  cloudwatch {
    namespace => "AWS/ELB"
    metrics => ["RequestCount", "Latency"]
    filters => { "LoadBalancerName" => "contegoview-api" }
    region => "us-east-1"
  }
}

Output

If set up successfully, you’d see events like this coming into Logstash:

{
       :timestamp => 2015-12-04 06:26:00 UTC,
    :sample_count => 5.0,
            :unit => "Percent",
         :minimum => 0.58,
         :maximum => 0.82,
             :sum => 3.33,
         :average => 0.666,
       :namespace => "AWS/EC2",
     :metric_name => "CPUUtilization",
      :start_time => "2015-12-04T08:21:35+02:00",
        :end_time => "2015-12-04T08:36:35+02:00",
          :period => 300,
     "InstanceId" => "i-cbde1234"
}

Other Configs

There’s a number of other parameters you can use to change the way the plugin operates:

region – The region in which the instances reside. If you’re not seeing any stats coming in, it might be that the region is incorrect.

interval – How often the CloudWatch API should be polled for metric statistics. Defaults to 900 seconds (15 minutes)

period – The granularity of the data points returned. Defaults to 300 seconds (5 minutes)

metrics – An array of Metric names you’d like to fetch. This is namespace specific.

statistics – An array of statistics you’d like to fetch for each metric. Defaults to SampleCount, Average, Minimum, Maximum and Sum.

The plugin hasn’t been added to the official Logstash repo yet. You can add your support or see the progress on this PR. If there’s anything missing or any issues, please report them on the Github Repo.

Happy Stashing!