Heroku Cedar Background Jobs for free!

I’m using Heroku mostly for playing around with latest technology and hosting free apps. Since Heroku changed their pricing model due to the introduction of the new process model some of my apps changed from free to paid, especially those that had some background jobs or nightly crons (I really did not get, why this happend).

Full Stack Background Processes

If you want to run a resque worker and a clockwork process within your web-app, this becomes a costly thing, even if those are just running some minor jobs in the night, because you need to pay for 2 additional dynos.
You could achieve this through defining multiple processes in your Procfile:

# Procfile
web: bundle exec rails server thin -p $PORT
worker: QUEUE=* bundle exec rake environment resque:work
clock: bundle exec clockwork app/clock.rb
heroku scale worker=1 clock=1

Sharing Addon Connections

As Heroku officially announced in their latest newsletter, it’s possible to share connections between multiple apps. In my case, this would be a connection to a redis key-value store, that is provided by Redistogo. It’s as simple as copying the environment configuration for the addon over to the second app:

heroku config | grep DATABASE_URL  --app sushi
=> DATABASE_URL   => postgres://lswlm...

heroku config:add DATABASE_URL=postgres://lswlm... --app sushi-analytics
=> Adding config vars: DATABASE_URL => postgres://lswlm...m/ldfoiusfsf
=> Restarting app... done, v74.

Going Freemium

So the solution for getting back to a free worker setup is combining 3 Heroku apps and a shared Redis connection through Redistogo:

heroku apps:create freemium-web --stack cedar --remote heroku
git push heroku master

heroku apps:create freemium-worker --stack cedar --remote worker
git push worker master

heroku apps:create freemium-clock --stack cedar --remote clock
git push clock master

heroku scale web=1 --app=freemium-web
heroku scale web=0 worker=1 --app=freemium-worker
heroku scale web=0 clock=1 --app=freemium-clock

heroku addons:add redistogo:nano --app=freemium-web

heroku config:add `heroku config -s --app=freemium-web|grep redis` --app=freemium-worker
heroku config:add `heroku config -s --app=freemium-web|grep redis` --app=freemium-clock

I created an example project running a Rails 3 application with a mounted Resque web, a Resque worker and a clock process with Clockwork.

Even simpler

If you are just looking for a simple solution of running a background process have a look at crony, a bootstrap project using rufus-scheduler.

Migrating an existing App to Heroku Celadon Cedar Stack

It’s currently not possible to do an automated migration from or to the Heroku Celadon Cedar Stack which started in May. The only help that you get from Heroku is this:

Migrating From Bamboo to Cedar

Before migrating to Bamboo, you should make sure your app runs on Ruby 1.9.2. If your app is not already deployed successfully to bamboo-mri-1.9.2, you should do that first, and then come back and try Cedar.

Once you’re sure your app is fully compatible with Ruby 1.9.2, create an app on the Cedar stack and push to it. Currently, stack:migrate does not work for moving apps to Cedar.

Since there seem to be no blogposts of how to do all the stuff with an existing application I will try to cover everything that I did to do the migration manually in this article.

Cedar is different

If you are looking at the new stack you will notice that a lot has changed with Cedar. Most of the Heroku CLI has changed to reflect the new possibilites that come with the Cedar stack. Since Cedar introduces a new way of handling processes on Heroku, most of the commands are now streamlined with this:

# old
heroku console
heroku rake db:migrate
# new 
heroku run console
heroku run rake db:migrate

# have a look at running processes
heroku ps

# look at logs of different processes and tail the output
heroku logs --ps web -t

Preparing the Migration

I started by cloning my current app from Heroku into another directory and creating a new Cedar app on Heroku:

git clone git@heroku.com:hamburg-on-ruby.git legacy
cd legacy
heroku create --stack cedar

Adapting Cedar changes

The docs for creating a Rails 3 app on Cedar reflect some of the changes that have been introduced with Cedar.

One of those is that PostgreSQL is now recommended for local development and you need to have the pg-gem in your Gemfile. Otherwise you will get errors while running rake commands:

heroku run rake db:migrate

  Running rake db:migrate attached to terminal... up, run.1
  (in /app)
  rake aborted!
  no such file to load -- pg

Since I developed my app with SQLite3 and I don’t want to learn, install and manage ANOTHER Database I split up the database stuff for development, test and production:

# Gemfile
group :production do
  gem "pg"
end

group :development, :test do
  gem "sqlite3-ruby", :require => "sqlite3"
end

and exclude production dependencies from my local environment:

# put this into your .profile
alias bd="bundle --without=production"

It’s also recommended to serve Rails through thin webserver and manage the application with the Procfile manager foreman:

# Gemfile
gem "foreman"
gem "thin"
# Procfile
web: bundle exec rails server thin -p $PORT
foreman start

After going through those chages you should be able to see your application on localhost:

open http://localhost:5000

The migration process

If the application is running on your development machine you can start the core migration process by putting your legacy application into maintenance mode, so that no database changes will be made:

heroku maintenance:on
# stop worker, crons or whatever might be changing the db

Database Migration

After you completely halted everything on your dyno, you can start dumping the database to your development machine (I thought it would be neat to do a direct migration from the legacy database to the new Cedar database, but I did not get it working). You need to install the taps-gem in order to do any dumping. Taps will dump the database into your development database unless you provide an connection string to another database:

# use sqlite for dumping and do it into a separate file
heroku db:pull sqlite://dump.db

# mv the file to your Cedar application and push it
heroku db:push sqlite://dump.db

Config variables

If you have custom config variables in your application it’s pretty easy to move them into the new environment:

# list all config values of the legacy app in console format
heroku config -s

# replace the newlines with whitespaces and
# append all the configuration variables of 
# your new Cedar app at the end or remove the
# ones that you do not want to migrate!

# add the list of values to your Cedar app
heroku config:add KEY=VALUE KEY2=VALUE2 [...]

Installing addons

It’s also most likely that you want to use the same addons than in your legacy application:

# list your addons in the legacy app
heroku addons

# install all addons separately in the Cedar app
heroku addons:add apigee:basic
heroku addons:add cron:daily
[...]

Sendgrid

If you are using the Sendgrid Addon you should be aware that the automatic configuration is not working anymore the Cedar Stack! You need to configure it in your app:

  # config/environments/production.rb
  config.action_mailer.raise_delivery_errors = true
  config.action_mailer.smtp_settings = {
    :address        => "smtp.sendgrid.net",
    :port           => "25",
    :authentication => :plain,
    :user_name      => ENV['SENDGRID_USERNAME'],
    :password       => ENV['SENDGRID_PASSWORD'],
    :domain         => ENV['SENDGRID_DOMAIN']
  }

Switching domains

Before you do the last step, you should have a look if everything is up and running:

heroku open

If everything is fine, you can move all the domains from the legacy application to your new and shiny Cedar Stack application:

# list domains in legacy app
heroku domains

# and remove them
heroku domains:remove hamburg.onruby.de
heroku domains:remove onruby.de
[...]

# re-add them to Cedar application
heroku domains:add hamburg.onruby.de
heroku domains:add onruby.de
[...]

Looking at the result is quite disappointing, because nothing has changed for the user, but everything is new and shiny from the inside!

If you have any other pointers of what you need to do for migrating your application please feel free to add a comment!

Generating PDF from HTML using DocRaptor on Heroku

There comes a time one has to create PDFs for a Rails application. Searching the web will most likely bring you to libraries like PDF Kit and Wicket PDF that use wkhtmltopdf as a driver.

If your app is hosted on Heroku you wonder weather wkhtmltopdf is available so that you can use one of these awesome libraries. Searching the Heroku docs, you will probably come to same conclusion: nothing on there!

As the Heroku support states:
You’re correct that there’s no official documentation. Most of our customers seem to be using wkhtmltopdf, generally with pdfkit. I do hope to document this usage soon.

http://github.com/jdpace/PDFKit
http://code.google.com/p/wkhtmltopdf/

If you want us to offer a Prince add-on, I encourage you to write them inviting them to check out our Add-on Provider Programme: http://addons.heroku.com/provider

Of course, you can always purchase your own license and run it on EC2 or another server of your choosing.

PrinceXML

One of the libraries that were used at my last job was PrinceXML, which did a great jog generating PDFs from HTML pages. It supports most of the HTML and CSS stuff, passing the ACID2 test. PrinceXML has some additional CSS attributes that enable you to configure additional PDF specific layout settings.

DocRaptor

Since PrinceXML is a commercial product, Heroku won’t support it and I did not find anything on the web, that would offer PDF generation as a service. Asking the PrinceXML Forum I found out about DocRaptor. These guys provide a service to convert HTML to XLS or PDF over a webservice interface, extactly what I was looking for. As an additional bonus, they just implemented a gem for supporting the Heroku Add-on interface. Mail to Expected Behavior Support if you want to participate in the private beta.

Improvements

DocRaptor offers a great service, but they are still in early development. There were some Issues that were resolved recently. If you are a user of PrinceXML you probably know the —baseurl option that allows usage of relative paths for images and stylesheets. DocRaptor adds support for command line options just yet. The feature for generating a PDF from a given URL is even better!

Using DocRaptor from Rails

The latest DocRaptor documentation for the Heroku Add-on is decent and it provides some nice examples.

PDF from raw HTML

Here is what I did to get it running on my Rails 3 project:

# Gemfile
gem "doc_raptor", "0.1.1"

# mime_types.rb
Mime::Type.register_alias "application/pdf", :pdf

# your_controller.rb
def your_pdf_action
  respond_to do |format|
    format.pdf do
      data = DocRaptor.create(:name => 'DocRaptor.pdf', :document_content => render_to_string, :document_type => "pdf", :prince_options => {:baseurl => 'http://nofail.de'})
      send_data data, :type => 'application/pdf', :filename => 'DocRaptor.pdf'
    end
  end
end

If you registered the pdf mime-type you will have to provide an additional layout for this. I added some PrinceXML specific parameters to the styles to make it a fullscreen PDF. One thing that is essential for making the stylesheets work is the media => ‘screen, print’ settings:

// application.pdf.haml
= stylesheet_link_tag 'style', :media => 'screen, print'
// you can provide a base tag for images and stylesheets
// %base{:url=>'http://blog.nofail.de'}

%style
  @page { size: A4 }
  @page { margin: 0px }
  @page { border: none }
  @page { padding: 0px }
  @page { prince-shrink-to-fit: auto }

PDFs from an URL

The simplest solution for generating a PDF is to send an url to the service, so you can re-use all your view logic:

data = DocRaptor.create(:name => "DocRaptor.pdf", :document_url => "http://blog.nofail.de", :document_type => "pdf")
send_data data, :type => 'application/pdf', :filename => "DocRaptor.pdf"

One caveat though, you got to have at least two dynos to serve the additional request from DocRaptor!

See a working example on my homepage.

Generating PDF form HTML without the hassle, thanks to DocRaptor!

Rails, getting started without the hassle

I just changed jobs and am now a Rails developer at tolingo.com, which is an online translation broker. When I started out working on my new desk, I had to setup my iMac development environment. There are tons of articles of how to compile/install/run stuff like MySQL, to get you started on OS X, but I think all one really needs is Homebrew and RVM.

Homebrew

Homebrew is a Ruby based packaging tool for Mac and once you start using it, you immediately hate yourself for having wasted time on MacPorts

“Homebrew is the easiest and most flexible way to install the UNIX tools Apple didn’t include with OS X.”

This quote is from the official website and I guess they are absolutely right!

Formula

Homebrew is build around formulas. They describe how a package should be loaded from the web and installed on your system. It also cares about package dependencies, paths and all the other ugly stuff:

require 'formula'

class Wget < Formula
  homepage 'http://www.gnu.org/wget/'
  url 'http://ftp.gnu.org/wget-1.12.tar.gz'
  md5 '308a5476fc096a8a525d07279a6f6aa3'

  def install
    system "./configure --prefix=#{prefix}"
    system 'make install'
  end
end

You can easily install packages from the shell with brew:

brew install wget

Homebrew puts all the packages into ‘/usr/local’, so that it won’t interfer with other components of your system. To get your packages working, you need to include it into your $PATH. If you have any problems running something, Homebrew comes with the doctor command, that scans for problems in your setup!

Installation

Just download Homebrew to your system and update once a while:

# install homebrew via curl
sudo mkdir -p /usr/local && sudo chown -R $USER /usr/local && curl -Lsf http://bit.ly/9H4NXH | tar xvz -C/usr/local --strip 1

# update homebrew
brew update

Git, MySQL, Sphinx and more

What else do you need? Just search for it or get more infos with info!

These are the packages that I needed for development:

# install mysql and set it up
brew install mysql
mysql_install_db
# add mysqld as launch agent
cp /usr/local/Cellar/mysql/#{MYSQL_VERSION}/com.mysql.mysqld.plist ~/Library/LaunchAgents
launchctl load -w ~/Library/LaunchAgents/com.mysql.mysqld.plist

# install git
brew install git git-flow

# add git bash completion (find path to your git with 'brew info git')
ln -s /usr/local/Cellar/git/#{GIT_VERSION}/etc/bash_completion.d/git-completion.bash ~/.git-completion.bash
source .git-completion.bash

# install sphinx search-deamon
brew install sphinx

# aspell with all spellings
brew install aspell --all

# libxml and imagemagick for sprites
brew install libxml2 imagemagick

RVM the Ruby Version Manager

RVM is a command line tool for managing your local Ruby environments, you can get some more information on the RVM homepage and in earlier articles.

Quick start with installing RVM to your machine:

# install rvm via curl !!! FOLLOW RVM INSTRUCTIONS !!!
bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )

# download and compile latest 1.8.7
rvm install 1.8.7

# create a .rvmrc file in your app's base directory
echo "rvm use 1.8.7@#{YOUR_APP} --create" > #{YOUR_APP}/.rvmrc
# execute it by cd-ing to your app's directory
cd #{YOUR_APP}

Now you can work on your app with a custom gem environment. Unless you are using Bundler, this is probably what you want for installing and removing gems painlessly.

Cucumber with Celerity

Behavior driven development with Cucumber works nicely with Celerity, a JRuby implementation of a headless browser using HtmlUnit and it’s companion a Ruby wrapper called Culerity. Culerity has recently been updated with some configuration points for registering your local JRuby environment:

# jruby config für culerity (from http://rvm.beginrescueend.com/integration/culerity/)
rvm install jruby
rvm use jruby@celerity --create
gem install celerity
rvm wrapper jruby@celerity celerity jruby
# add to .profile
export JRUBY_INVOCATION="$(readlink "$(which celerity_jruby)")"

If you are experiencing any weird Broken Pipe errors (like me), have a look at this issue.

This is just an example of how you can setup your Rails development environment. Comments on this topic are appreciated!

Using the Redis addon on Heroku

I am always playing around with new addons offered by Heroku. My latest discovery was the Redis addon that is provided by Redistogo. The addon is probably in private beta (docs are still on beta), but since they put up a link to it on their site, I managed to install it to my personal website that runs in the cloud.

Redis is “an advanced key-value store” and has some features that make it a perfect match for a cache! I use caching extensively on my site and keep on trying out new ways to do it to circumvent Heroku’s readonly filesystem.

Like Memcache, Redis provides the ability to set a time to live (ttl) on a key. This comes in handy, if you have data that expires in a short period of time, like 3rd party data from Twitter etc.

Caching with Redis

Accessing Redis is very simple, since it is a text based protocol. The command reference is straight forward and there is a simple Ruby wrapper available:

require "redis"
redis = Redis.new
redis.set "foo", "bar"
# => "OK"
redis.get "foo"
# => "bar"

The redis-store gem already provides a Rails 3 compatible Cache Store implementation, but I needed some more configuration points, especially the ttl.

That’s why I wrote my own Rails 3 Redis Cache, also a great way to get used to the way of working with Redis and the Redistogo addon.

Using Rails Redis Cache

There is some configuration needed for Rails to pick up the new cache store. If you want to use different or no caching for test, development and production, you should put the config in your environment files:

# config/environemnts/production.rb
config.action_controller.perform_caching = true
config.cache_store = ActiveSupport::Cache::RailsRedisCache.new(:url => ENV['REDISTOGO_URL'])

If there is a Redis server available in all environments, you can put it in your environment file:

# config/environment.rb
ActionController::Base.cache_store = ActiveSupport::Cache::RailsRedisCache.new(:url => ENV['REDISTOGO_URL'])

The caching parts are mostly in my controllers:

@tweets = cache("tweets", :expires_in => 30.seconds){ Twitter::Search.new(...) }

The store is using the basic Rails cache store implementation which is broken in the Rails 3.0.0.beta1 version that runs on Heroku, so I added a monkey-patch for that using edge Rails.

Redis on localhost

Installing and running Redis on Mac OS X is really simple:

brew install redis
redis-server

There is also a commandline client available for direct access:

redis-cli
redis> set "foo" "bar"
OK
redis> get "foo"
"bar"

It’s key value stores, stupid!