Tag: noSQL

Heroku Cedar Background Jobs for free!

I’m using Heroku mostly for playing around with latest technology and hosting free apps. Since Heroku changed their pricing model due to the introduction of the new process model some of my apps changed from free to paid, especially those that had some background jobs or nightly crons (I really did not get, why this happend).

Full Stack Background Processes

If you want to run a resque worker and a clockwork process within your web-app, this becomes a costly thing, even if those are just running some minor jobs in the night, because you need to pay for 2 additional dynos.
You could achieve this through defining multiple processes in your Procfile:

# Procfile
web: bundle exec rails server thin -p $PORT
worker: QUEUE=* bundle exec rake environment resque:work
clock: bundle exec clockwork app/clock.rb

heroku scale worker=1 clock=1

Sharing Addon Connections

As Heroku officially announced in their latest newsletter, it’s possible to share connections between multiple apps. In my case, this would be a connection to a redis key-value store, that is provided by Redistogo. It’s as simple as copying the environment configuration for the addon over to the second app:

heroku config | grep DATABASE_URL  --app sushi
=> DATABASE_URL   => postgres://lswlm...

heroku config:add DATABASE_URL=postgres://lswlm... --app sushi-analytics
=> Adding config vars: DATABASE_URL => postgres://lswlm...m/ldfoiusfsf
=> Restarting app... done, v74.

Going Freemium

So the solution for getting back to a free worker setup is combining 3 Heroku apps and a shared Redis connection through Redistogo:

heroku apps:create freemium-web --stack cedar --remote heroku
git push heroku master

heroku apps:create freemium-worker --stack cedar --remote worker
git push worker master

heroku apps:create freemium-clock --stack cedar --remote clock
git push clock master

heroku scale web=1 --app=freemium-web
heroku scale web=0 worker=1 --app=freemium-worker
heroku scale web=0 clock=1 --app=freemium-clock

heroku addons:add redistogo:nano --app=freemium-web

heroku config:add `heroku config -s --app=freemium-web|grep redis` --app=freemium-worker
heroku config:add `heroku config -s --app=freemium-web|grep redis` --app=freemium-clock

I created an example project running a Rails 3 application with a mounted Resque web, a Resque worker and a clock process with Clockwork.

Even simpler

If you are just looking for a simple solution of running a background process have a look at crony, a bootstrap project using rufus-scheduler.

Using the Redis addon on Heroku

I am always playing around with new addons offered by Heroku. My latest discovery was the Redis addon that is provided by Redistogo. The addon is probably in private beta (docs are still on beta), but since they put up a link to it on their site, I managed to install it to my personal website that runs in the cloud.

Redis is “an advanced key-value store” and has some features that make it a perfect match for a cache! I use caching extensively on my site and keep on trying out new ways to do it to circumvent Heroku’s readonly filesystem.

Like Memcache, Redis provides the ability to set a time to live (ttl) on a key. This comes in handy, if you have data that expires in a short period of time, like 3rd party data from Twitter etc.

Caching with Redis

Accessing Redis is very simple, since it is a text based protocol. The command reference is straight forward and there is a simple Ruby wrapper available:

require "redis"
redis = Redis.new
redis.set "foo", "bar"
# => "OK"
redis.get "foo"
# => "bar"

The redis-store gem already provides a Rails 3 compatible Cache Store implementation, but I needed some more configuration points, especially the ttl.

That’s why I wrote my own Rails 3 Redis Cache, also a great way to get used to the way of working with Redis and the Redistogo addon.

Using Rails Redis Cache

There is some configuration needed for Rails to pick up the new cache store. If you want to use different or no caching for test, development and production, you should put the config in your environment files:

# config/environemnts/production.rb
config.action_controller.perform_caching = true
config.cache_store = ActiveSupport::Cache::RailsRedisCache.new(:url => ENV['REDISTOGO_URL'])

If there is a Redis server available in all environments, you can put it in your environment file:

# config/environment.rb
ActionController::Base.cache_store = ActiveSupport::Cache::RailsRedisCache.new(:url => ENV['REDISTOGO_URL'])

The caching parts are mostly in my controllers:

@tweets = cache("tweets", :expires_in => 30.seconds){ Twitter::Search.new(...) }

The store is using the basic Rails cache store implementation which is broken in the Rails 3.0.0.beta1 version that runs on Heroku, so I added a monkey-patch for that using edge Rails.

Redis on localhost

Installing and running Redis on Mac OS X is really simple:

brew install redis
redis-server

There is also a commandline client available for direct access:

redis-cli
redis> set "foo" "bar"
OK
redis> get "foo"
"bar"

It’s key value stores, stupid!

noSQL – Rails models with SOAP

Using a DB is a natural thing for a Rails developer. Since Rails is a database driven application framework, that does not come as a big surprise. But there are times where environmental constraints do not allow the freedom to use the weapon of choice…

Imagine a legacy Java SOA landscape that provides tons of webservices but does not permit access to a transaction DB. Sounds phoney? Ask your local J2EE consultant!

Working around this constraint, it would be great if one could just wire a SOAP service into Rails as a backing of model data. Using Rails without a database is a little bit tricky, especially if you don’t want to forego the power of ActiveRecord!

so why use Rails then?

There are a lot of people that would say “Why don’t you use Sinatra instead?”.

First of all, most Ruby developers know how to use Rails. The Rails community is large, lively and a great resource for knowledge. Features like REST come for free and nobody want’s to miss model validations. In general, Rails plugins are lazy programmers best friend!

working with ActiveForm

A simple way to get your SOAP backed noSQL model working with ActiveRecord::Validations is probably by using ActiveForm. It provides validations for non ActiveRecord models and is available on github.

You can install the Rails plugin via:

# (re)install from git as a plugin
script/plugin install --force git://github.com/remvee/active_form.git

Using the plugin in your code is simple. Inherit from ActiveForm instead of ActiveRecord::Base:

# app/models/blog.rb
class Blog < ActiveForm
  column :title
  column :message, :type => :text
  
  validates_presence_of :title, :message
  [...]
end

It’s possible to remove all evidence of database connectivity. Just kick ActiveRecord from the list of Rails frameworks and re-add it as a gem (this step is not necessary, so you might skip it and work with Rails sqlite3 default):

# config/environment.rb
Rails::Initializer.run do |config|
  [...]
  config.gem "activerecord", :version => '2.3.5'
  [...]
  config.frameworks -= [ :active_record, :active_resource, :action_mailer ]
  [...]
end

Doing so will allow you to delete the database.yml file in your application.

Savon for multi-tier persistence

Accessing an enterprise SOAP service with Savon is easy and integrating Savon into a Rails model requires just two steps:

  • implementing a to_hash method
  • implementing a save hook

Since Savon communication is based on data hashes, you have to provide a thin mapping layer to convert your model into a request hash that matches your SOAP interface:

# app/models/blog.rb
  def to_hash
    { :data => {:title=>title, :message=>message} }
  end

Pushing the data to the webservice requires some custom ‘persistence’ code to be implemented. A good place for that code should be in one of the model’s save hooks:

# app/models/blog.rb
  def after_save
    client = Savon::Client.new "http://localhost:8080/"

    client.post! do |soap|
      soap.namespace = "urn:savon:blog"
      soap.body = to_hash
    end
  end

Overwriting the after_save method is a neat way to let the model code be readable for other Rails developers. Sticking to conventions is a best practice and reduces complexity greatly!

more information?

There is a working example using a local soap4r server available on github.

no SQL - no problem!