These days data is being generated at an immense pace: Servers, laptops, smart phones, embedded devices, applications, kitchen appliances, all of them are producing data of some kind. We set up these systems to generate the data, but we’re not always sure what to do with it. We know it’s useful, but we’re not quite sure how.
As an example: You’re the creator of a system that does wonderful things. It makes people’s lives better, it adds value and people depend on it. But it breaks. Not always, but every now and then someone does something unexpected, and the system goes down. So you set up an email to go out every time the system breaks.
In the beginning it’s great, as you’re notified every time something goes wrong, and you have all the details of the error. Your customer service improves and you sign up more users. Then your system gets really busy, and your email account is flooded. There’s so much email, that you stop reading email all together, and start slipping on customer service for your system. Users leave.
Smart person that you are, you decide to stop sending emails when something bad happens, and rather dump the errors to a text log file. Great, the email flood as been diverted and you can once again get to Inbox 0. Busy as you are, though, you sometimes forget to check the log file for critical errors. Days and even weeks go buy with your users unable to get their daily fix because you are unaware of a critical error.
You’re sitting with tons of data, but not a lot of information.
There are tons of tools to change data into information. This blog will focus on one stack of tools to address this problem: The ELK Stack: Elasticsearch, Logstash and Kibana.
Elasticsearch is a search optimized data store that will form the base of the stack. All of the data flows to Elasticsearch, and all the information is extracted using it’s search functionalities.
Logstash is a utility to ship and modify logs. It’s very versatile so it can be used to not only ship logs but also connect disparate systems.
Kibana is a Javascript interface to Elasticsearch that can be used to create simple data visualizations that will give you an insight into the data your collecting.
If you’re interested to learn more about ELK and it’s possibilities, be sure to check back here regularly, of sign up for our newsletter in the sidebar.