How to Configure the Logstash Metrics filter

This short guide will look at the Logstash Metrics filter. It often happens that you’re not interested in actual events, but just the number of times an event occurs in a specific period of time. Logstash supports counting events and measuring their velocity rates through the metrics filter.

Quick Info

Links: Source | Documentation
Version: 1.4.2
Requirements:

• An active Logstash input.

The short version

The metrics filter can operate in one of two modes: Meter or timer. In meter mode it will count the number of events, and emit a new event containing the total number of events and the short, mid and long term rates of events at regular intervals. In timer mode it will emit a new event containing the count, short, mid and long term rates and various statistics about the number of events that passed through it at set intervals.

# Bare minimum Meter
filter {
  metric {
    meter => [ "log_events" ]
  }
}

# Bare minimum Timer
filter {
  metric {
    timer => [ "log_events", "%{request_time}" ] 
  }
}

The meter filter will create a new event every 5 seconds, giving you the following stats of events up to this point:

• log_events.count – The total number of events up to now
• log_events.rate_1m – The average number of events per minute
• log_events.rate_5m – The average number of events in 5 minutes
• log_events.rate_15m – The average number of events in 15 minutes

You’ll notice that the value of the meter option was used to name the event properties. This is useful to differentiate between different metrics.

The timer filter will create a new event every 5 seconds, giving you the same stats of the event as the meter filter above, as well as the following stats on the request_time property of the event:

• request_time.min – The smallest request_time value the filter has seen so far.
• request_time.max – The largest request_time value the filter has seen so far.
• request_time.stddev – The standard deviation of the value the filter has seen so far.
• request_time.mean – The mean of the values the filter has seen so far.
• request_time.p1 – The 1th percentile for the measured values.
• request_time.p5 – The 5th percentile for the measured values.
• request_time.p10 – The 10th percentile for the measured values.
• request_time.p90 – The 90th percentile for the measured values.
• request_time.p95 – The 95th percentile for the measured values.
• request_time.p99 – The 99th percentile for the measured values.
• request_time.p100 – The 100th percentile for the measured values.

Once again the value passed as the key of the hash is used to construct the event properties.

The longer version

The metrics filter has a couple of options with which you can tweak it’s behaviour to suit your own needs. By default Logstash will keep on counting events and report on all the events it has received so far until it is stopped. The first example shows how you can use the clear_interval setting to reset the count ever 60 minutes:

# Only count for 60 minutes
filter {
  metric {
    meter => [ "log_events" ]
    clear_interval => 3600 # This needs to be multiples of 5
  }
}

By default a new event is created every 5 seconds. This can be altered using the flush_interval option:

# Create a new event every 5 minutes
filter {
  metric {
    timer => [ "log_events", "%{request_time}" ] 
    flush_interval => 300 # This needs to be multiples of 5
  }
}

Each new event contains a lot of data. You can limit that using the rates and percentiles options:

filter {
  metric {
    timer => [ "log_events", "%{request_time}" ] 
    percentiles => [50, 95]
    rates => []
  }
}

The percentiles is only applicable to the timer metric. The rates options to both the timer and meter metrics.

All the options

meter – Array – The metered rates you want to generate. You need to specify at least one of timer or meter.

timer – Hash – Specify the timed rates as well as the value you want to measure. You need to specify at least one of timer or meter.

percentiles – Array – The percentile values you want to include in your timer metrics. Defaults to [1, 5, 10, 90, 95, 99, 100].

rates – Array – The velocity rates you want to include in your timer and meter metrics. The only possible values are 1, 5, and 15. Defaults to [1, 5, 15].

clear_interval – Number – How often, in seconds, the counters should be cleared and start over. It needs to be a multiple of 5. Defaults to -1, which means never.

flush_interval – Number – How often, in seconds, a metric event should be generated. It needs to be a multiple of 5, and defaults to 5

ignore_older_than – Number – You can use this option to exclude events that are too old. This is useful to only include near real time events. Defaults to 0 (disabled).

---

Check out the Logstash Config Guide for sound advice on configuring Logstash.