Simple DB caching for Heroku

Heroku is a great platform. I like the style of the page, I appreciate the documentation and you can start up for free! One thing that I miss a lot is decent caching. The readonly filesystem eats up a lot of flexibility.

I played around with HTTP caching and Herokus Varnish works really well. The problem is that my app loads a lot of stuff from different 3rd party services like Twitter, so every new visitor will have all the load time on his first visit. Not a surprise that New Relic indicates that request times were ‘Unacceptable’…

I would like to check out the ‘Memcached Basic’ plugin of Heroku, but I did not manage to get into the private beta. So there was no other option than implementing a DB cache.

There is just one requirement that I have. Load stuff from a 3rd party service only if it’s expired. For simplicity, expired means, that the data is older than a predefined interval. In my test environment I like to use a shorter period than in production, so I define the interval in the environment files:

# config/environments/development.rb
CACHE_TIME = 30.seconds

# config/environments/production.rb
CACHE_TIME = 10.hours

A simple key-data pair is enough for my needs, because I always have a unique key for the values I want to cache. I am using Marshal.dump/Marshal.load for serialization, as they play well with anonymous inner classes that YAML can’t deal with. Encoding the data Base64 helps working around some SQLITE issues with serialized data strings:

# app/models/storage.rb
class Storage < ActiveRecord::Base
  
  validates_presence_of :key, :data
  
  def data=(data)
    write_attribute :data, ActiveSupport::Base64.encode64(Marshal.dump(data))
  end
  
  def data
    Marshal.load(ActiveSupport::Base64.decode64(read_attribute :data))
  end
  
end

The actual caching logic is embeded in my application controller. I provide a simple cache method, that can be called with a block. The block contains the remote call that I want to cache and is only executed if there is no data stored for the given key or the stored data is expired:

  # app/controllers/application_controller.rb
  def cache(key, &to_cache)
    from_db = Storage.first(:conditions => {:key => key})
    if from_db.nil? || from_db.updated_at < Time.new - CACHE_TIME
      data = (yield to_cache).collect{|t|t}
      return [] if data.nil? || data.empty?
      from_db = (from_db || Storage.new)
      from_db.key = key
      from_db.data = data
      from_db.save!
    end
    instance_variable_set :"@#{key.to_s}", from_db.data
  end

Finally the data is pushed into an instance varaible, so that I have access to it within my views.

Caching is now as simple as this:

  # cache all twitter posts and make them accessible via @tweets
  cache(:tweets){Helper::twitter_posts}

This little tweak noteable improved the response time of my app:

  This week:
  Apdex Score: 0.700.5 (Fair)

  Last week:
  Apdex Score: 0.060.5 (Unacceptable)

Sugar on rails!

Writing your own DSL with Ruby

I am a big fan of Ruby. There are so many beautiful libraries out there and most of them are based on some kind of domain specific language. Take builder as an example:

  Builder::XmlMarkup.new.person { |b| b.name("Jim"); b.phone("555-1234") }
  #=> <person><name>Jim</name><phone>555-1234</phone></person>

Generating XML in this manner is pretty cool! There is no crazy XML editor in the world that gives you as much flexibility as this straight forward DSL. I think that a good DSL makes coding feel natural.

HOWTO DSL

If you did read Ruby best practicies or stuff like that, this article won’t bring anything new to you. In case you did not, you should start right away!

Anyhow, there are some simple rules behind writing a good DSL or API:

  • let the user choose how to use it
  • make options optional
  • make use of an option hash for defaults
  • make use of scoped blocks
  • make use of instance_eval
  • consider implementing method_missing

I am going to explain these rules for you, so that you can start writing your own cool DSL and help make programming Ruby even more fun.

the Regexp DSL

I think that regular expressions are a great tool and they are ubiquitous in Ruby. Regexps are a first class citizen in Ruby and so you get a lot of built-in power for free. There are also some decent Regexp guides on the web waiting for you:

Regular expressions are a framework of their own, embedded into lots of programming languages. The biggest problem that I have with the Regexp DSL is missing readability. Take a look at the monster that lives in the dark pit of URI::REGEXP::PATTERN:

/
        ([a-zA-Z][-+.a-zA-Z\d]*):                     (?# 1: scheme)
        (?:
           ((?:[-_.!~*'()a-zA-Z\d;?:@&=+$,]|%[a-fA-F\d]{2})(?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*)              (?# 2: opaque)
        |
           (?:(?:
             \/\/(?:
                 (?:(?:((?:[-_.!~*'()a-zA-Z\d;:&=+$,]|%[a-fA-F\d]{2})*)@)?  (?# 3: userinfo)
                   (?:((?:(?:(?:[a-zA-Z\d](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.)*(?:[a-zA-Z](?:[-a-zA-Z\d]*[a-zA-Z\d])?)\.?|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|\[(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:(?:[a-fA-F\d]{1,4}:)*[a-fA-F\d]{1,4})?::(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))?)\]))(?::(\d*))?))?(?# 4: host, 5: port)
               |
                 ((?:[-_.!~*'()a-zA-Z\d$,;:@&=+]|%[a-fA-F\d]{2})+)           (?# 6: registry)
               )
             |
             (?!\/\/))                              (?# XXX: '\/\/' is the mark for hostport)
             (\/(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*(?:\/(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*)*)?              (?# 7: path)
           )(?:\?((?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*))?           (?# 8: query)
        )
        (?:\#((?:[-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]|%[a-fA-F\d]{2})*))?            (?# 9: fragment)
      /xn

You don’t need to write such complex expressions to get to the point where you don’t understand the Regexp you wrote just 2 minutes ago. That’s why Ruby provides an extended mode that allows you to insert spaces, newlines and comments in the pattern:

  def test_multiline_regex_with_comments
    assert_match(%r[a # the a
                    n # the n
                    ]x, 'anna')
  end

This is nothing that feels natural to me… So I started working on my own regular expression DSL.

lessons learned from Rebuil

It is always a good idea to write a DSL. There is no better way of becoming a domain expert than implementing a custom language for that specific domain. The other great thing is that you get in depth knowledge of the key Ruby features that help you creating crisp APIs and neat frameworks.

The vision that I have about using Regexps is a very descriptive one. There are probably some people that would call it verbose, but there is always a tradeoff:

  exp = rebuil.many.group('rebuil', :cool_name)
  puts "hello #{exp.match('hello world with rebuil')[:cool_name]} world!" 
  #=> hello rebuil world

This code basically creates the Regexp ’.*(rebuil)’. The cool thing is that you get named groups like in Ruby 1.9 for accessing them with descriptive symbols instead of an index.

let the user choose how to use it

The first advice for creating a DSL is important if you want to make your library sexy for other developers. Since most hackers have different coding styles, they prefer different types of expressions. A good example for this are the different ways one can create Rebuil objects:

  # standard approach
  Rebuil::Expression.new.group('a')

  # helper within object
  rebuil.group('a')

  # helper with a block
  rebuil do |exp|
    exp.group('a')
  end

  # helper with some starting pattern
  rebuil("").group('a')

As all implemented Rebuil methods return the object instance itself, one can chain method calls for convenience.

make options optional

The rebuil method can take a string as an argument for the start of the expression. As you can see in the example above, this argument is optional.

A well designed API does not force the user to provide arguments that are not mandatory.

make use of an option hash for defaults

If you end up with a lot of optional parameters in your method signature, you should consider using an optional parameter hash as the last method argument. Using carefully picked keys for the parameters gives the impression of named parameters wich improves readablity greatly:

  def some_method(man, da, tory, options={:some=>'defaults'})
    ...
  end

make use of scoped blocks

Providing scopes is another way to improve the readability of your code. Ruby makes it easy to do scope based programming with blocks:

  # no scope
  exp = rebuil
  exp.group('a')
  exp.characters('a')
  exp.many

  # better with a block
  rebuil do |exp|
    exp.group('a')
    exp.characters('a')
    exp.many
  end

This can be achieved by just yielding a Rebuil::Expression if a block is given.

make use of instance_eval

Blocks can also be used to reduce the code you will have to write. The scoped block example above can be rewritten with less code:

  # smooth with instance_eval
  rebuil do
    group('a')
    characters('a')
    many
  end

Evaluating the block in the context of the current object is as simple as passing it on to Rubies instance_eval. The only drawback is that you can’t access stuff from your current scope like member variables. But there is a way to make both solutions work, just ask the block for the number of arguments:

  def rebuil(expression="", &block)
    ...
    # block.arity returns the number of expected arguments of the block
    block.arity < 1 ? re.instance_eval(&block) : block.call(re) if block_given?
    ...
  end

consider implementing method_missing

Highly dynamic DSLs like Builder follow a different approach. They implement method_missing. The great advantage is that the user can posibly call anything on your object. The domain logic lies within your implementation of method_missing, which passes the name of the originally called method, the parameters and an optional block. Since regular expressions are deterministic, this dynamic approach does not suite Rebuil well.

With great power comes great responsibility. Always try to find the most appropriate implementation for your specific domain.

And don’t forget the sugar!

Buildr – The build system that doesn’t suck

If you are a Java guy like me, you are probably doing a lot of ugly tasks in your everday work. One of these tasks is build management.

The Java community tries to tackle builds with standard tools like Ant and “all-in-one” solutions like Maven. Both are based on huge XML configuration files that are hard to read, painful to maintain and impossible to test. Even though these tools are mainstream, they are quite buggy. Just have a look at all the rants about maven you find on the net.

Java builds are known to become very complex over time. You have to compile things found here, include stuff from there, resolve dependencies in a remote repository, add something to the version control, […even more crazy stuff…]. Addressing such complex and dynamic tasks with static structures is just a stupid idea. What you really want to do is scripting!

scripting builds

Using scripting languages is a natural way to address build tasks. If you are close to the operating system you can do most things easily. Trust your admin, the shell is your friend!

The simplest way to get scripting power is to use bash scripts, which has the drawback that it’s not interoperable and you will always have to show consideration for WINDOW$ users…

There are portable frameworks that try to integrate the flexibilty of scripting languages like Ruby with the power of Ant or Maven. Have a look at Gradle, GMaven, Sbt, Raven, Gant or Antwrap as examples.
About one year ago I stumbled over another build tool that claims the slogan The build system that doesn’t suck which I found quite promising!

dynamic builds with Buildr

Buildr is an expressive Ruby DSL for managing builds based on Rake and Ant. Defining a Java project that follows the default Apache directory structure is dead simple:

# buildfile.rb
desc 'define a project called simple'
define 'simple' do
  info 'create a simple.jar from this project'
  package :jar
end

After installing the Buildr Gem you can call built-in or custom Buildr tasks on your project. This is done in the same way Rake would do it:

# install the Buildr gem
gem install buildr

# check buildr version
buildr -V
=> Buildr 1.3.5 (JRuby 1.4.0)

# build your Java project: compile, test, package
buildr package

Since Buildr is based on sensible defaults, it knows where to look for your class files, resources and tests. Before creating the jar, it compiles everything, executes the tests and then zips up the bytecode into a distributable.

Buildr does not support every build feature by default, but it’s easy to extend. Buildr at its core is just a bunch of tasks, chained in a predefined order. You can easily write your own tasks and plug them into the build to fit your needs. Buildr provides method hooks that allow you to integrate your tasks into the build cycle.

If this small example makes you say “Awesome!”, you should have a look at the best documentation of an open source framework I have ever seen. There are some sophisticated examples of how to use Buildr hosted on github as well.

Build, don’t goof up XML!