Even though Logstash is great for parsing events as they happen, you can also use it to process historical data. Normally logstash will timestamp an event with the time when the event was initially processed. This isn’t ideal when you’re trying to analyze historic data. Logstash provides the Logstash Date filter to aid in the parsing and setting of dates and timestamps.
Quick Info
Links: Source | Documentation
Version: 1.4.2
Requirements:
- An active Logstash input
The short version
The date filter parses dates using formats as defined by the Joda Time library. All you need to do is specify the field and the format it conforms to, and Logstash will timestamp the event according to the contents of the field. If the field isn’t present or not populated, it won’t update the event.
# Bare minimum
filter {
date {
match => [ "logdate", "yyyy-MM-dd HH:mm:ss" ]
}
}
This will timestamp the event if the logdate event has a field that has dates that look like this: 2015-04-27 14:15:16
The long version
The field you’re parsing for the date might not contain the timezone. You can use the timezone setting to specify the default timezone for events. Use a time zone ID from the [Joda time zones][] page:
# Default Timezone
filter {
date {
match => [ "logdate", "yyyy-MM-dd HH:mm:ss" ]
timezone => "Africa/Johannesburg"
}
}
Month and weekday names might be in a different locale. Use the locale setting to ensure that they are parsed correctly. The setting should be in the format lang_country_variant, such as eng_US_POSIX. The country and variant parts are optional, so you can get away with just setting it to eng.
# Locale
filter {
date {
match => [ "logdate", "yyyy-MM-dd HH:mm:ss" ]
locale => "eng_US_POSIX"
}
}
To make parsing easier, Logstash provides a number of predefined matchers that you can use in the match parameter:
- ISO8601
- Parses valid [ISO8601][5] dates.
- UNIX
- Match fields containing [Unix time][6] – seconds since the epoch.
- UNIX_MS
- Match fields containing [Unix time][6] – milliseconds since the epoch.
- TAI64N
- Reads [TAI64][7] timestamps
Use these instead of the custom format to ensure accuracy when parsing fields:
# Predefined matcher
filter {
date {
match => [ "logdate", "ISO8601" ]
}
}
All of the options
timezone – String – The default timezone the timestamp should be generated in. This is optional.
locale – String – The locale to use when parsing the field. The three parts you specify are passed to the Java Locale object. This is optional.
match – Array – Value pairs with the first value the field to parse, and the second value the format of the field. You need to specify at least one field and format.
target – String – The field in which the parsed timestamp should be stored. Use this if you’re using the filter to parse fields for something other than the event’s timestamp. Defaults to @timestamp
Do you need Logstash enlightenment? Try out the Logstash Config Guide for config nirvana!
Jurgens du Toit
Coder. Thinker. Human. I try to write good code for a living and wrangle data as a hobby. Be sure to check out the book I'm writing: The Logstash Config Guide.
Subscribe To Our Newsletter
Join our mailing list to receive the latest news and updates from our team.
