Logstash comes with a number of plugins that interact with AWS, Amazon’s cloud infrastructure. This short guide will look at the common configuration options for these plugins.

Quick Info

Source:
/lib/logstash/plugin_mixins/aws_config.rb
Requirements:

  • AWS Account
  • AWS Credentials with access to the required service OR
  • An instance of Logstash running on EC2 with access to the required service.

TL;DR

  • If you’re running Logstash on an EC2 instance, you don’t need to do anything.
  • If you have multiple plugins using the same credentials, use ENV variables.
  • If you have multiple plugins using different credentials, use the aws_credentials_file parameter.
  • Otherwise go for the access_key_id and secret_access_key parameters.

Details

All of Logstash’s AWS related plugins try several methods to retrieve the AWS credentials it should use to authenticate itself to AWS. Here’s the options in order of priority, along with a short discussion on the usability of each option:

1. The key and secret specified in the logstash config options

The plugin first checks for the access key id and secret access key in the access_key_id and secret_access_key parameters for the specific plugin, in this case the SNS output. You’ll need to specify both:

output {
  sns {
    access_key_id => "Access Key ID"
    secret_access_key => "Access Key Secret"
  }
}

The fact that the credentials are saved in plain text in something other than your configuration management system might pose a security risk. On the other hand, if you’re using a template in your config management to create the file, it shouldn’t be a problem.

2. The key and secret specified in an external config file

If the key and secret isn’t specified as parameters, Logstash checks for a credentials file specified with aws_credentials file that should contain the key and secret. The file should contain YAML:

output {
  sns {
    aws_credentials_file => "/etc/aws_credentials.yaml"
  }
}

This is a simpler and more centralized way to handle the credentials than the first option. If you have multiple plugins using the same credentials, this is the way to go. Of all the options, it’s the easiest to manage, has the most flexibility, and you’re less likely to commit and push your AWS credentials to your repo.

# /etc/aws_credentials.yaml
:access_key_id: "12345"
:secret_access_key: "54321"

3. Environment variables

If you prefer to have your secrets as environment variables, Logstash will check the following pairs of environment variables for the key and secret.

Firstly, it checks AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Secondly, it checks AMAZON_ACCESS_KEY_ID and AMAZON_SECRET_ACCESS_KEY.

The drawback with using ENV variables is that all the plugins in the current environment will be using the same credentials. If you have different access profiles for the different services, you won’t be able to use this.

4. IAM Instance Profile

As a last gasp effort, Logstash will check for an IAM Instance Profile. This is only available if Logstash is running on an EC2 instance.

It has the same drawback as placing the key and secret as ENV variables, in that you only have one set of credentials. It’s also the easiest setup in that you have to do absolutely nothing to set it up.

Logstash plugins using AWS

Inputs

Outputs


Are the Logstash config blues getting you down? The Logstash Config Guide will pick you righ up. Check it out now!

Coder. Thinker. Human. I try to write good code for a living and wrangle data as a hobby. Be sure to check out the book I'm writing: The Logstash Config Guide.

Subscribe To Our Newsletter

Subscribe To Our Newsletter

Join our mailing list to receive the latest news and updates from our team.

You have Successfully Subscribed!

Share This