Aggregation Station June 29th, 2007

Duplicating efforts is bad, so I often start by googling, "rails functionality_i_want". Often helpful, but not always. "ruby functionality_i_want" is easily overlooked. I might end up doing ActiveRecord mapping, but there is much greater diversity in this space. That is how I found FeedNormalizer, my news feed parsing library of choice.

FeedNormalizer is consistant across feed types, allows modifying and adding parsers without changing your application, and can sanitize elements with html for you.

So how do you use this miracle brew?

  1. Install
    gem install feed-normalizer
  2. Off to the races
    require 'open-uri'
require 'rubygems'
require 'feed-normalizer'

file = open('http://monki.geemus.com/feed/')
feed = FeedNormalizer::FeedNormalizer.parse file
# Pull whatever fields you might need
p feed.title
p feed.entries
# etc
    Quick and easy. See this intro for details about how this works.

  3. As a fun addition, you can use headers check for updates before downloading the whole feed.
    # open the feed
data = open('http://monki.geemus.com/feed/')
# grab the relevant meta data
last_modified = data.meta['last-modified']
etag = data.meta['etag']

# use the feed
feed = FeedNormalizer::FeedNormalizer.parse data
# etc

# now ask nicely for updates
begin
  data = open('http://monki.geemus.com/feed/', 
               {'If-Modified-Since' => last_modified, 
               'If-None-Match' => etag})
rescue OpenURI::HTTPError
  # No updates, so feed was not downloaded.
else
  # Updates
  feed = FeedNormalizer::FeedNormalizer.parse data
end
Pretty slick, saves publisher bandwidth and your processing time. Everybody wins!

Go forth and aggregate!

tags: gems

ruby gem sources June 7th, 2007

I recently realized that rubygems doesn't update itself of its own accord. Which is to say:

gem update

Updates installed gems and not the gem system itself. To do that you need:

gem update --system

Which leads me to what was driving me crazy(a recurring theme?), which may be unrelated to updating...

Gem sources. I added the gem repository from rubyonrails.com to try the capistrano 2.0 preview. Cap 2 works fine, but I didn't intend to update rails to newer-than-1.2. I tried removing the rubyonrails gem repository from the list (see: gem help sources), but I was still getting newer versions...

Thankfully the explanation is simple. When I install gems I use sudo. The root user has a different source list from other users.
So if you normally type:
sudo gem install some_gem_name
You will also want to type:
sudo gem sources remove some_gem_repository_url

Easy to overlook, I think I got lucky figuring it out. Hope others have better luck, go go googlebot etc.

tags: gems