Simple DB caching for Heroku

Heroku is a great platform. I like the style of the page, I appreciate the documentation and you can start up for free! One thing that I miss a lot is decent caching. The readonly filesystem eats up a lot of flexibility.

I played around with HTTP caching and Herokus Varnish works really well. The problem is that my app loads a lot of stuff from different 3rd party services like Twitter, so every new visitor will have all the load time on his first visit. Not a surprise that New Relic indicates that request times were ‘Unacceptable’…

I would like to check out the ‘Memcached Basic’ plugin of Heroku, but I did not manage to get into the private beta. So there was no other option than implementing a DB cache.

There is just one requirement that I have. Load stuff from a 3rd party service only if it’s expired. For simplicity, expired means, that the data is older than a predefined interval. In my test environment I like to use a shorter period than in production, so I define the interval in the environment files:

# config/environments/development.rb
CACHE_TIME = 30.seconds

# config/environments/production.rb
CACHE_TIME = 10.hours

A simple key-data pair is enough for my needs, because I always have a unique key for the values I want to cache. I am using Marshal.dump/Marshal.load for serialization, as they play well with anonymous inner classes that YAML can’t deal with. Encoding the data Base64 helps working around some SQLITE issues with serialized data strings:

# app/models/storage.rb
class Storage < ActiveRecord::Base
  
  validates_presence_of :key, :data
  
  def data=(data)
    write_attribute :data, ActiveSupport::Base64.encode64(Marshal.dump(data))
  end
  
  def data
    Marshal.load(ActiveSupport::Base64.decode64(read_attribute :data))
  end
  
end

The actual caching logic is embeded in my application controller. I provide a simple cache method, that can be called with a block. The block contains the remote call that I want to cache and is only executed if there is no data stored for the given key or the stored data is expired:

  # app/controllers/application_controller.rb
  def cache(key, &to_cache)
    from_db = Storage.first(:conditions => {:key => key})
    if from_db.nil? || from_db.updated_at < Time.new - CACHE_TIME
      data = (yield to_cache).collect{|t|t}
      return [] if data.nil? || data.empty?
      from_db = (from_db || Storage.new)
      from_db.key = key
      from_db.data = data
      from_db.save!
    end
    instance_variable_set :"@#{key.to_s}", from_db.data
  end

Finally the data is pushed into an instance varaible, so that I have access to it within my views.

Caching is now as simple as this:

  # cache all twitter posts and make them accessible via @tweets
  cache(:tweets){Helper::twitter_posts}

This little tweak noteable improved the response time of my app:

  This week:
  Apdex Score: 0.700.5 (Fair)

  Last week:
  Apdex Score: 0.060.5 (Unacceptable)

Sugar on rails!

Savon vs. Handsoap: Accessing a WSDL

This documentation is deprecated, please have a look at savonrb.com!

Both clients provide an interface to work with a WSDL. While the Handsoap WSDL support is hidden in some helper class, WSDLs are a first class citizen in Savon. The code for printing out the available SOAP actions looks like this:

require "handsoap/parser"
wsdl = Handsoap::Parser::Wsdl.read(@wsdl_uri)
wsdl.bindings.each {|binding| binding.actions.each{|action| p action.name }}
p Savon::Client.new("some_wsdl").wsdl.soap_actions

The Handsoap parser class is part of a Rails generator. The generator can be used for creating a Handsoap service class skeleton and tests:

$ script/generate handsoap http://www.thomas-bayer.com/axis2/services/BLZService?wsdl

      exists  app
      exists  app/models
      create  app/models/blz_service.rb
      exists  test
      exists  test/integration
      create  test/integration/blz_service_test.rb
----
Endpoints in WSDL
  You should copy these to the appropriate environment files.
  (Eg. `config/environments/*.rb`)
----
# wsdl: http://www.thomas-bayer.com/axis2/services/BLZService?wsdl
BLZ_SERVICE_ENDPOINT = {
  :uri => 'http://www.thomas-bayer.com:80/axis2/services/BLZService',
  :version => 2
}
----

The skeleton provides method stubs for adding request parameters and result parsing:

def get_bank
  soap_action = ''
  response = invoke('tns:getBank', soap_action) do |message|
    raise "TODO"
  end
end

Heroku with ruby-aaws

The first step of migrating my Rails app to Heroku included only a reduced feature set.

I completely removed all parts dealing with the Amazon-API, because ruby-aaws could not be installed as a gem on Heroku. The ruby-aaws gem is build on Ruby >= 1.8.7 while Heroku is running 1.8.6 #fail.

I asked Ian Macdonald, the creator of ruby-aaws, if there might be a workaround for this problem. Ian told me that this should not be a blocker and pointed me to some things I should take care of.

How to make ruby-aaws run with 1.8.6

Here are the steps that I performed to get things running:

# install gemsonrails to freeze the gem into the app
sudo gem install gemsonrails

# go to your rails folder
cd rails/yourapp

# install gemsonrails for your app
gemsonrails

# freeze the app
rake gems:freeze GEM=ruby-aaws

This will freeze the gem into vendor/gems/. Then I removed the tests and the examples, because they are not necessary and contain code that won’t work.
The app won’t start up until I fixed some little load path error:

# vendor/gems/ruby-aaws-0.7.0/init.rb
# add amazon to the path
require_options = ["ruby-aaws", "ruby/aaws", "amazon"]

The ruby-aaws library expects some login-credentials for the Amazon-API which are usually stored in a ~/.amazonrc file. Since Heroku gives no access to a user home (at least I did not manage to get access to it), I worked around it by putting the file in the RAILS_ROOT and adding some glue code in Amazon::Config:

# vendor/gems/ruby-aaws-0.7.0/lib/amazon.rb
# add rails-file
config_files << File.join(RAILS_ROOT, '.amazonrc') if defined?(RAILS_ROOT)

The next thing to fix was the usage of String.bytesize, which is not available in 1.8.6:

string.gsub( /([^a-zA-Z0-9_.~-]+)/ ) do
  # replace .bytesize with .size
  '%' + $1.unpack( 'H2' * $1.size ).join( '%' ).upcase
end

Voilà!

Heroku with custom domain

I recently decided to switch my Rails hosting platform. I started Rails using a shared hosting solution from domainFACTORY. That solution did not quite fill my needs so i decided to give Heroku a try.

Switching to Heroku is an easy taks, at least if you know what you’re doing…

Using Heroku

The heroku documentation is pretty decent. You will find almost everything very well explained.

So I provide just a little wrapup of the steps I went through for integrating an existing Rails application:

# install heroku gem
gem install heroku

# go to your rails folder
cd rails/yourapp

# skip git initialization if your app is already under versioncontrol
git init
git add .
git commit -m "new app"

# create a heroku app with the name of your app unless you already have one
# the first call to heroku automatically prompts for your ssh public key
# enter your heroku and ssh credentials
# heroku will automatically add heroku git repository as a remote
heroku create yourappname

# check that heroku is added
git remote

# add heroku remote manually if it's missing
git remote add heroku git@heroku.com:yourappname.git

# push your changes to heroku and automatically start your app with this
git push heroku master

# check logs 
# if the app crashes, most of the times 
# it is a missing .gems file that heroku uses for gem-management
heroku logs

Configuring the domain

Once the app is up and running, the next step is to get the custom domain working with Heroku. Heroku provides serveral solutions for this task. The most flexible way seems to be the Heroku-Zerigo integration.

You only have to add the Zerigo name servers to your domain as an NS (name server) naming entry. For my domain www.phoet.de this looks like this:

www.phoet.de	NS	 	a.ns.zerigo.net
www.phoet.de	NS	 	b.ns.zerigo.net
www.phoet.de	NS	 	c.ns.zerigo.net
www.phoet.de	NS	 	d.ns.zerigo.net
www.phoet.de	NS	 	e.ns.zerigo.net

It takes some time until the changes are propagated through all the name servers. You can check it using the host command:

host www.phoet.de
# www.phoet.de is an alias for proxy.heroku.com.
# proxy.heroku.com has address 75.101.163.44
# proxy.heroku.com has address 174.129.212.2
# proxy.heroku.com has address 75.101.145.87

The Heroku addons take care of the rest, no setup or whatever:

# add the custom-domain addon
heroku addons:add custom_domains:basic

# add the zerigo-dns addon
heroku addons:add zerigo_dns

# add your domain to heroku
heroku domains:add www.yourdomain.com

Magic!

I write these lines, because I did not manage to do this withouth the fast Heroku support.