noSQL – Rails models with SOAP

Using a DB is a natural thing for a Rails developer. Since Rails is a database driven application framework, that does not come as a big surprise. But there are times where environmental constraints do not allow the freedom to use the weapon of choice…

Imagine a legacy Java SOA landscape that provides tons of webservices but does not permit access to a transaction DB. Sounds phoney? Ask your local J2EE consultant!

Working around this constraint, it would be great if one could just wire a SOAP service into Rails as a backing of model data. Using Rails without a database is a little bit tricky, especially if you don’t want to forego the power of ActiveRecord!

so why use Rails then?

There are a lot of people that would say “Why don’t you use Sinatra instead?”.

First of all, most Ruby developers know how to use Rails. The Rails community is large, lively and a great resource for knowledge. Features like REST come for free and nobody want’s to miss model validations. In general, Rails plugins are lazy programmers best friend!

working with ActiveForm

A simple way to get your SOAP backed noSQL model working with ActiveRecord::Validations is probably by using ActiveForm. It provides validations for non ActiveRecord models and is available on github.

You can install the Rails plugin via:

# (re)install from git as a plugin
script/plugin install --force git://github.com/remvee/active_form.git

Using the plugin in your code is simple. Inherit from ActiveForm instead of ActiveRecord::Base:

# app/models/blog.rb
class Blog < ActiveForm
  column :title
  column :message, :type => :text
  
  validates_presence_of :title, :message
  [...]
end

It’s possible to remove all evidence of database connectivity. Just kick ActiveRecord from the list of Rails frameworks and re-add it as a gem (this step is not necessary, so you might skip it and work with Rails sqlite3 default):

# config/environment.rb
Rails::Initializer.run do |config|
  [...]
  config.gem "activerecord", :version => '2.3.5'
  [...]
  config.frameworks -= [ :active_record, :active_resource, :action_mailer ]
  [...]
end

Doing so will allow you to delete the database.yml file in your application.

Savon for multi-tier persistence

Accessing an enterprise SOAP service with Savon is easy and integrating Savon into a Rails model requires just two steps:

  • implementing a to_hash method
  • implementing a save hook

Since Savon communication is based on data hashes, you have to provide a thin mapping layer to convert your model into a request hash that matches your SOAP interface:

# app/models/blog.rb
  def to_hash
    { :data => {:title=>title, :message=>message} }
  end

Pushing the data to the webservice requires some custom ‘persistence’ code to be implemented. A good place for that code should be in one of the model’s save hooks:

# app/models/blog.rb
  def after_save
    client = Savon::Client.new "http://localhost:8080/"

    client.post! do |soap|
      soap.namespace = "urn:savon:blog"
      soap.body = to_hash
    end
  end

Overwriting the after_save method is a neat way to let the model code be readable for other Rails developers. Sticking to conventions is a best practice and reduces complexity greatly!

more information?

There is a working example using a local soap4r server available on github.

no SQL - no problem!

Simple DB caching for Heroku

Heroku is a great platform. I like the style of the page, I appreciate the documentation and you can start up for free! One thing that I miss a lot is decent caching. The readonly filesystem eats up a lot of flexibility.

I played around with HTTP caching and Herokus Varnish works really well. The problem is that my app loads a lot of stuff from different 3rd party services like Twitter, so every new visitor will have all the load time on his first visit. Not a surprise that New Relic indicates that request times were ‘Unacceptable’…

I would like to check out the ‘Memcached Basic’ plugin of Heroku, but I did not manage to get into the private beta. So there was no other option than implementing a DB cache.

There is just one requirement that I have. Load stuff from a 3rd party service only if it’s expired. For simplicity, expired means, that the data is older than a predefined interval. In my test environment I like to use a shorter period than in production, so I define the interval in the environment files:

# config/environments/development.rb
CACHE_TIME = 30.seconds

# config/environments/production.rb
CACHE_TIME = 10.hours

A simple key-data pair is enough for my needs, because I always have a unique key for the values I want to cache. I am using Marshal.dump/Marshal.load for serialization, as they play well with anonymous inner classes that YAML can’t deal with. Encoding the data Base64 helps working around some SQLITE issues with serialized data strings:

# app/models/storage.rb
class Storage < ActiveRecord::Base
  
  validates_presence_of :key, :data
  
  def data=(data)
    write_attribute :data, ActiveSupport::Base64.encode64(Marshal.dump(data))
  end
  
  def data
    Marshal.load(ActiveSupport::Base64.decode64(read_attribute :data))
  end
  
end

The actual caching logic is embeded in my application controller. I provide a simple cache method, that can be called with a block. The block contains the remote call that I want to cache and is only executed if there is no data stored for the given key or the stored data is expired:

  # app/controllers/application_controller.rb
  def cache(key, &to_cache)
    from_db = Storage.first(:conditions => {:key => key})
    if from_db.nil? || from_db.updated_at < Time.new - CACHE_TIME
      data = (yield to_cache).collect{|t|t}
      return [] if data.nil? || data.empty?
      from_db = (from_db || Storage.new)
      from_db.key = key
      from_db.data = data
      from_db.save!
    end
    instance_variable_set :"@#{key.to_s}", from_db.data
  end

Finally the data is pushed into an instance varaible, so that I have access to it within my views.

Caching is now as simple as this:

  # cache all twitter posts and make them accessible via @tweets
  cache(:tweets){Helper::twitter_posts}

This little tweak noteable improved the response time of my app:

  This week:
  Apdex Score: 0.700.5 (Fair)

  Last week:
  Apdex Score: 0.060.5 (Unacceptable)

Sugar on rails!

Savon vs. Handsoap: Accessing a WSDL

This documentation is deprecated, please have a look at savonrb.com!

Both clients provide an interface to work with a WSDL. While the Handsoap WSDL support is hidden in some helper class, WSDLs are a first class citizen in Savon. The code for printing out the available SOAP actions looks like this:

require "handsoap/parser"
wsdl = Handsoap::Parser::Wsdl.read(@wsdl_uri)
wsdl.bindings.each {|binding| binding.actions.each{|action| p action.name }}
p Savon::Client.new("some_wsdl").wsdl.soap_actions

The Handsoap parser class is part of a Rails generator. The generator can be used for creating a Handsoap service class skeleton and tests:

$ script/generate handsoap http://www.thomas-bayer.com/axis2/services/BLZService?wsdl

      exists  app
      exists  app/models
      create  app/models/blz_service.rb
      exists  test
      exists  test/integration
      create  test/integration/blz_service_test.rb
----
Endpoints in WSDL
  You should copy these to the appropriate environment files.
  (Eg. `config/environments/*.rb`)
----
# wsdl: http://www.thomas-bayer.com/axis2/services/BLZService?wsdl
BLZ_SERVICE_ENDPOINT = {
  :uri => 'http://www.thomas-bayer.com:80/axis2/services/BLZService',
  :version => 2
}
----

The skeleton provides method stubs for adding request parameters and result parsing:

def get_bank
  soap_action = ''
  response = invoke('tns:getBank', soap_action) do |message|
    raise "TODO"
  end
end

Heroku with ruby-aaws

The first step of migrating my Rails app to Heroku included only a reduced feature set.

I completely removed all parts dealing with the Amazon-API, because ruby-aaws could not be installed as a gem on Heroku. The ruby-aaws gem is build on Ruby >= 1.8.7 while Heroku is running 1.8.6 #fail.

I asked Ian Macdonald, the creator of ruby-aaws, if there might be a workaround for this problem. Ian told me that this should not be a blocker and pointed me to some things I should take care of.

How to make ruby-aaws run with 1.8.6

Here are the steps that I performed to get things running:

# install gemsonrails to freeze the gem into the app
sudo gem install gemsonrails

# go to your rails folder
cd rails/yourapp

# install gemsonrails for your app
gemsonrails

# freeze the app
rake gems:freeze GEM=ruby-aaws

This will freeze the gem into vendor/gems/. Then I removed the tests and the examples, because they are not necessary and contain code that won’t work.
The app won’t start up until I fixed some little load path error:

# vendor/gems/ruby-aaws-0.7.0/init.rb
# add amazon to the path
require_options = ["ruby-aaws", "ruby/aaws", "amazon"]

The ruby-aaws library expects some login-credentials for the Amazon-API which are usually stored in a ~/.amazonrc file. Since Heroku gives no access to a user home (at least I did not manage to get access to it), I worked around it by putting the file in the RAILS_ROOT and adding some glue code in Amazon::Config:

# vendor/gems/ruby-aaws-0.7.0/lib/amazon.rb
# add rails-file
config_files << File.join(RAILS_ROOT, '.amazonrc') if defined?(RAILS_ROOT)

The next thing to fix was the usage of String.bytesize, which is not available in 1.8.6:

string.gsub( /([^a-zA-Z0-9_.~-]+)/ ) do
  # replace .bytesize with .size
  '%' + $1.unpack( 'H2' * $1.size ).join( '%' ).upcase
end

Voilà!

Heroku with custom domain

I recently decided to switch my Rails hosting platform. I started Rails using a shared hosting solution from domainFACTORY. That solution did not quite fill my needs so i decided to give Heroku a try.

Switching to Heroku is an easy taks, at least if you know what you’re doing…

Using Heroku

The heroku documentation is pretty decent. You will find almost everything very well explained.

So I provide just a little wrapup of the steps I went through for integrating an existing Rails application:

# install heroku gem
gem install heroku

# go to your rails folder
cd rails/yourapp

# skip git initialization if your app is already under versioncontrol
git init
git add .
git commit -m "new app"

# create a heroku app with the name of your app unless you already have one
# the first call to heroku automatically prompts for your ssh public key
# enter your heroku and ssh credentials
# heroku will automatically add heroku git repository as a remote
heroku create yourappname

# check that heroku is added
git remote

# add heroku remote manually if it's missing
git remote add heroku git@heroku.com:yourappname.git

# push your changes to heroku and automatically start your app with this
git push heroku master

# check logs 
# if the app crashes, most of the times 
# it is a missing .gems file that heroku uses for gem-management
heroku logs

Configuring the domain

Once the app is up and running, the next step is to get the custom domain working with Heroku. Heroku provides serveral solutions for this task. The most flexible way seems to be the Heroku-Zerigo integration.

You only have to add the Zerigo name servers to your domain as an NS (name server) naming entry. For my domain www.phoet.de this looks like this:

www.phoet.de	NS	 	a.ns.zerigo.net
www.phoet.de	NS	 	b.ns.zerigo.net
www.phoet.de	NS	 	c.ns.zerigo.net
www.phoet.de	NS	 	d.ns.zerigo.net
www.phoet.de	NS	 	e.ns.zerigo.net

It takes some time until the changes are propagated through all the name servers. You can check it using the host command:

host www.phoet.de
# www.phoet.de is an alias for proxy.heroku.com.
# proxy.heroku.com has address 75.101.163.44
# proxy.heroku.com has address 174.129.212.2
# proxy.heroku.com has address 75.101.145.87

The Heroku addons take care of the rest, no setup or whatever:

# add the custom-domain addon
heroku addons:add custom_domains:basic

# add the zerigo-dns addon
heroku addons:add zerigo_dns

# add your domain to heroku
heroku domains:add www.yourdomain.com

Magic!

I write these lines, because I did not manage to do this withouth the fast Heroku support.