Searching for that perfect job: Scraping a RSS feed using Logstash

We all have that dream job, the one that pays enough, is in the right city and has the right perks. Yet we’re all so busy in our normal jobs that we have no time to keep a proper eye on all the job boards and postings available on the internet. To name but a few, there’s:

Let’s automate this using logstash:

Code Along

You can get a vagrant box with everything you need to follow this blog post in this repo.

Input

Most of the job sites come with RSS feeds, so let’s use that along with Logstash’s RSS input as the first step. First clone the rss input to the inputs folder of logstash:

cd /opt/logstash/inputs
clone https://github.com/coolacid/logstash-rss.git

And set up the input:

input {
  rss {
    url => "http://www.authenticjobs.com/rss/custom.php?terms=&type=1,2,6&cats=&onlyremote=1&location="
    interval => 6000
  }
}

Logstash will now check the RSS feed for new entries every 6000 seconds. Adjust as neeeded.

Filter

Every job posting matching the search criteria will now be piped through Logstash. Let’s filter these further using the 9 filter:

filter {
  ruby {
    code => "
test = event['title'].match(/senior|ruby/i) 
  && event['message'].match(/remote/i)
event.cancel unless test
(event["tags"] ||= []) << "sinatra" if event['message'].match(/sinatra/i)
(event["tags"] ||= []) << "rails" if event['message'].match(/rails/i)
"
  }
}

This let’s us throw away posts that might not match our criteria, as well as categorize each posting according to what we want.

Output

And, for simplicity’s sake, we email the posting to ourself:

output {
  if "rails" in [tags] or "sinatra" in [tags] {
    email {
      to => '[email protected]'
      subject => 'New Job! %{title}'
      htmlbody => "<h3>Tags:</h3><p>%{tags}</p><h3>Posting:</h3>%{message}"
      options => {
          port => 1025
      }
    }
  }
}

We’ve set it up to use the default mailcatcher config to test it. Customize as needed.

Finally

You now run logstash as follows so that it pick’s up the custom input plugin:

$ /opt/logstash/bin/logstash -f /etc/logstash/conf.d --pluginpath /opt/logstash/inputs/lib

Et voila, you’ll be notified of every matching job posting. We can, of course, use Elasticsearch to do even deeper text analysis of the job posts. More on this in another post.

Subscribe To Our Newsletter

Join our mailing list to receive the latest news and updates from our team.