How to write a Logstash plugin

One of the killer features of Logstash is it’s extensibility: You can use various official and unofficial plugins to add the functionality you need to Logstash. Further to that, if you can’t find an existing plugin, it’s relatively simple to write your own plugin. This post will look at how to write an input plugin for Logstash. Since Logstash 1.5 plugins are standalone Ruby gems the process will seem familiar if you’ve built a gem in the past. The plugin we’ll be writing will fetch CloudWatch metrics from AWS.

TL;DR

Here’s a quick rundown of what you need to do:

1. Clone an example plugin
2. Rename the core files
3. Edit the gemspec file
4. Ensure that your Gemfile specifies JRuby
5. Do a bundle install
6. Do your development, and remember to add and run tests
7. Build the gem
8. Install the gem

On to more detail.

JRuby

Logstash runs on JRuby, so it’s plugins also need to run on JRuby. If you haven’t installed it yet, now is a good time to do so. Check out this post on setting up a Logstash development environment on how to do so.

Before we jump in

As mentioned before, a Logstash plugin is basically a ruby gem. This gem usually consists of only one class that extends a Base class provided by logstash. There’s also a number of other methods and utilities available in the code that you should be aware of:

The config method allows you to specify configuration options available to the plugin. When specified, the option will be available in the plugin class as an instance variable (prefixed with an @ sign).

The config_name needs to be called to give your plugin a unique, machine readable name in Logstash. It’s best to do this right at the top of the class.

Not all plugins use it, but the default codec for the plugin needs to be defined by adding default :codec, 'plain' to the class.

Get the Code

Elastic maintains an example plugin for each of the plugin types: inputs, outputs, filters and codecs. These provide a great base to start from, as they contain all the structure and boilerplate you need for your plugin. The first step then is to clone the plugin you need. Run the following commands in bash to create your plugin, add the example repository as a remote, and pull master from there. Note the pattern you need to use when naming your plugin: logstash-[type]-[name].

mkdir logstash-input-cloudwatch
cd logstash-input-cloudwatch
git init
git remote add example [email protected]:logstash-plugins/logstash-input-example.git
git pull example master

Setup

Once you have the example code, you need to remove and rename all the sample information and replace it with your own.

The files are all named after the example plugin. Just rename them to match your own:

mv logstash-input-example.gemspec logstash-input-cloudwatch.gemspec
mv lib/logstash/inputs/example.rb lib/logstash/inputs/cloudwatch.rb
mv spec/inputs/example_spec.rb spec/inputs/cloudwatch_spec.rb

We will now change the plugin’s details in the logstash-input-cloudwatch.gemspec file. This file contains information about the project, such as the author and website, as well as it’s dependencies. For now just change the author and project details, we’ll cover the dependencies later.

For completeness we’ll also ensure that bundler knows we should be using JRuby. Change your Gemfile to look like this:

source 'https://rubygems.org'
ruby "1.9.3", :engine => "jruby", :engine_version => "1.7.19"
gemspec

You can now install all of the plugin’s dependencies by running bundle install. Every time you change your dependencies in the gemspec file you need to rerun this command. You’ll of course need the bundler gem on your machine to do this. Install it by running gem install bundler.

A Basic Outline

All Logstash plugins follow the same basic pattern. Here’s a rough outline of a Logstash plugin. We’ll discuss the various parts later in the post:

# The class definition, inheriting the base class
class LogStash::Inputs::CloudWatch < LogStash::Inputs::Base
  # The machine readable name for the plugin
  config_name "cloudwatch"

  # If undefined, LogStash will complain, even if codec is unused.
  default :codec, "json"

  # The interval config option with a default
  config :interval, :validate => :number, :default => (60 * 15)

  # The setup phase of the plugin
  def register
    # Do once of, long running tasks here
  end

  # The actual processing phase of the plugin
  def run(queue)
    # Generate Logstash::Event's and push them to the queue
  end

  # The shutdown phase of the plugin
  def teardown
    # Close connections, clean up, etc.
  end
end

Just note that this outline is slightly different for the different types of plugins. Look at the example plugins for more information.

Development

Ok, all of the setup has been done. We can now start doing development. There’s four parts to Logstash plugin development that you need to take care of:

1. Configuration
2. Setup
3. Execution
4. Teardown

Configuration

Looking at the beginning of the plugin class, you’ll notice the various methods we discussed earlier that’s used to define the plugin. This plugin “header” will define the basic meta information, configuration and behavior for your plugin. The next step will use the config options you specified to set up the plugin. Try and make your plugin as configurable as possible. If you’re hardcoding a value, it would probably do better as a config option to allow the user to change it as needed.

Setup

Logstash will call the register method of your plugin once when initializing. This is to allow the plugin to start or initialize anything that might be needed later. Anything that’s done once and is possibly long running should be done here. Things like initializing clients, lists and connections are commonly done here. This happens before Logstash starts processing events. For the CloudWatch plugin we create CloudWatch and EC2 API clients.

Execution

Once Logstash starts processing events, it will run the run method of your plugin. The run method is passed a SizedQueue onto which you need to push new events. Events are defined by the Logstash::Event class. The decorate method adds fields and tags as defined by the Logstash config and it’s expected that you decorate events before pushing them onto the queue:

event = LogStash::Event.new({text: 'Some event text'})
decorate(event)
queue << event

The method needs to keep running until interrupted since every Logstash input runs in its own thread. For the CloudWatch input we use Stud to poll the CloudWatch API every couple of seconds:

  def run(queue)
    Stud.interval(@interval) do
      @logger.debug('Polling CloudWatch API')
      # Do the cloudwatch magic...
      event = LogStash::Event.new(cw_data)
      decorate(event)
      queue << event
    end
  end

Notice the use of the @interval configuration value, as well the @logger utility made available to the plugin by the base class.

Teardown

Logstash won’t run forever, and its good manners to clean up after yourself. Close connections, clean up files, do what ever you need to do to keep it clean in the teardown method.

Testing

The example plugin has provided boilerplate to allow for automated testing of your plugin. By writing and running tests for your plugin you can ensure that common mistakes are easily caught, and eventual changes to the Logstash API that might conflict with your plugin can easily be picked up. You run the tests by doing:

bundle exec rspec

Building

Once your development is done, and you’re ready to deploy your plugin, you need to build your plugin:

gem build logstash-input-cloudwatch.gemspec

The gem utility uses the information in your gemspec file to build a.gem` file. You can now install this gem / plugin locally.

Installation

From inside your Logstash setup run the following:

bin/plugin install /path/to/logstash-input-cloudwatch-0.1.0.gem

This will use the Logstash plugin utility to install your plugin. Be sure to give the correct path to the .gem file we generated in the build step. Confirm that the plugin has been installed by running bin/plugin list and look for your plugin name to show. You can now run Logstash, with your plugin added as an input, to properly test it.

Once happy with your local testing, you can publish the plugin on RubyGems, the public repository for ruby gems or just keep it in-house.

What Logstash plugins have you written? What plugin would you like to see? Tell us below!