We all have that dream job, the one that pays enough, is in the right city and has the right perks. Yet we’re all so busy in our normal jobs that we have no time to keep a proper eye on all the job boards and postings available on the internet. To name but a few, there’s:

  1. We work remotely
  2. Gun.io
  3. Authentic Jobs
  4. Reddit’s forhire
  5. Hacker News’ monthly posts
  6. Stackoverflow careers

Ain't nobody got time for that
Let’s automate this using logstash:

Code Along

You can get a vagrant box with everything you need to follow this blog post in this repo.


Most of the job sites come with RSS feeds, so let’s use that along with Logstash’s RSS input as the first step. First clone the rss input to the inputs folder of logstash:

cd /opt/logstash/inputs
clone https://github.com/coolacid/logstash-rss.git

And set up the input:

input {
  rss {
    url => "http://www.authenticjobs.com/rss/custom.php?terms=&type=1,2,6&cats=&onlyremote=1&location="
    interval => 6000

Logstash will now check the RSS feed for new entries every 6000 seconds. Adjust as neeeded.


Every job posting matching the search criteria will now be piped through Logstash. Let’s filter these further using the 9 filter:

filter {
  ruby {
    code => "
test = event['title'].match(/senior|ruby/i) 
  && event['message'].match(/remote/i)
event.cancel unless test
(event["tags"] ||= []) << "sinatra" if event['message'].match(/sinatra/i)
(event["tags"] ||= []) << "rails" if event['message'].match(/rails/i)

This let’s us throw away posts that might not match our criteria, as well as categorize each posting according to what we want.


And, for simplicity’s sake, we email the posting to ourself:

output {
  if "rails" in [tags] or "sinatra" in [tags] {
    email {
      to => '[email protected]'
      subject => 'New Job! %{title}'
      htmlbody => "<h3>Tags:</h3><p>%{tags}</p><h3>Posting:</h3>%{message}"
      options => {
          port => 1025

We’ve set it up to use the default mailcatcher config to test it. Customize as needed.


You now run logstash as follows so that it pick’s up the custom input plugin:

$ /opt/logstash/bin/logstash -f /etc/logstash/conf.d --pluginpath /opt/logstash/inputs/lib

Et voila, you’ll be notified of every matching job posting. We can, of course, use Elasticsearch to do even deeper text analysis of the job posts. More on this in another post.

Subscribe To Our Newsletter

Subscribe To Our Newsletter

Join our mailing list to receive the latest news and updates from our team.

You have Successfully Subscribed!

Share This