Amazon’s CloudWatch service provides statistics on various metrics for AWS services. You can track CPU Utilization and credit usage on EC2, the amount of time a volume is idle on EBS, and the number of messages published, delivered and failed on SNS. Although AWS provides access to these metrics and alarms on them, you’re locked into the AWS ecosystem to use them. To import your CloudWatch metrics into your ecosystem, you can now use the Logstash CloudWatch input plugin to regularly fetch the required metrics from AWS.
Installation
The plugin is only available on Logstash 1.5, and can be installed using the plugin utility:
bin/plugin install logstash-input-cloudwatch
Minimum Configuration
The plugin uses the Logstash AWS config settings and requires an AIM profile that has access to CloudWatch metrics. The following policy is the minimum you need to get started:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1444715676000",
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics"
],
"Resource": "*"
},
{
"Sid": "Stmt1444716576170",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances"
],
"Resource": "*"
}
]
}
Other than that you only need to specify the instances to track. This can be done in two ways:
Hardcoded instances
You can specify specific instances by using the instances setting:
input {
cloudwatch {
instances => [ 'i-123456', 'i-654321' ]
}
}
Due to the dynamic nature of AWS, this is probably not the best option.
Tagged instances
An alternative to hardcoding instance ID’s, is to tag the instances you’d like to monitor, and then specifying the tag name and value in the Logstash config:
input {
cloudwatch {
tag_name => 'Monitored'
tag_values => [ 'Yes' ]
}
}
Output
If set up successfully, you’d see events like this coming into Logstash:
{
"timestamp" => 2015-06-14 20:15:00 UTC,
"sample_count" => 5.0,
"unit" => "Bytes",
"minimum" => 143802.0,
"maximum" => 443711.0,
"sum" => 1381311.0,
"average" => 276262.2,
"@version" => "1",
"@timestamp" => "2015-06-14T20:15:00.000Z",
"metric" => "NetworkOut",
"instance" => "i-123456"
}
Other Configs
There’s a number of other parameters you can use to change the way the plugin operates:
region – The region in which the instances reside. If you’re not seeing any stats coming in, it might be that the region is incorrect.
interval – How often the CloudWatch API should be polled for metric statistics. Defaults to 900 seconds (15 minutes)
period – The granularity of the data points returned. Defaults to 60 seconds (1 minute)
metrics – An array of Metric names you’d like to fetch.
statistics – An array of statistics you’d like to fetch for each metric.
The plugin hasn’t been added to the official Logstash repo yet. You can add your support or see the progress on this PR. If there’s anything missing or any issues, please report them on the Github Repo.
Jurgens du Toit
Coder. Thinker. Human. I try to write good code for a living and wrangle data as a hobby. Be sure to check out the book I'm writing: The Logstash Config Guide.
Subscribe To Our Newsletter
Join our mailing list to receive the latest news and updates from our team.
