Asset Pipeline with Heroku Continued

This is a follow up post to yesterday’s article on how I work with asset pipeline for a Heroku hosted app.

While the setup described from the previous article works. It leaves a few sticking points.

  1. Before I do assets precompile, I have to either export the necessary environment variables manually, or set them up in my .zshrc globally.
  2. It’s hard to precompile assets for different deployment targets, e.g. staging and production.
  3. I have to remember to run the assets clean rake task to keep my local dev env sane.

So … I did some follow up work. The README I wrote for the specific project is as follow.

Asset Compiling

Assets are precompiled locally and automatically uploaded to S3 with the help from the asset_sync gem.

After step into the project root directory, simply calling a custom ap() function, such as

⚡ ap staging

to compile the assets and upload the compiled assets to S3.

For this to work, RVM on local development machine is required. The handy function ap() is defined in .rvmrc under the project root.

The argument passed into the ap() function should be a valid Heroku Git remote name.

For example, we have staging setup on Heroku because of the following git remote output.

⚡ git remote -v
origin  git@bitbucket.org:xxxxx/xxxxx.git (fetch)
origin  git@bitbucket.org:xxxxx/xxxxx.git (push)
staging git@heroku.com:xxxxx.git (fetch)
staging git@heroku.com:xxxxx.git (push)

This means we will need to create a .projectrc.staging file under the project root. A projectrc template is provided, .projectrc.sample. 3 config variables need to be filled in.

Note that FOG_DIRECTORY is the Amazon S3 bucket name. To mute fog’s noisy output during assets precompile, name the S3 bucket using only alphabets and . is recommended.

The same 3 variables are also set on Heroku and can be found by doing

⚡ heroku config

The ap() function does the following

  1. precompile assets locally
  2. upload the compiled assets under /public/assets to the corresponding S3 buckets
  3. purge local compiled assets from /public/assets, with the exception of /public/assets/manifest.xml, which is pushed to Heroku.

Now to finish it up, here’s my .rvmrc

rvm ruby-1.9.2-p290@my_project --create

ap() {
  local projectrc=".projectrc.$1"

  if [ -f "$projectrc" ]; then
    source "$projectrc"

    bundle exec rake assets:precompile
    git add public/assets/manifest.yml -f
    git commit -m 'updated asset manifest'
    bundle exec rake assets:clean && git checkout public/assets/manifest.yml
  else
    echo "Missing '$projectrc'. Check README for details."
  fi
}

And here’s the .projectrc.sample

export FOG_DIRECTORY=XXXXXXXXXXXXX
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX

Hope this makes sense and helps.

Published: 2012-07-12

Asset Pipeline with Heroku

Rails 3’s asset pipeline is a great way to organise javascript, css and images. One app I’m working on at the moment deploys to Heroku and doing the asset precompile on Heroku could be slow. When deploying to Heroku, the time spent on doing asset precompile is unnecessary and unbearable, especially if the new changes have nothing to do with assets …

The best solution I can find is http://icelab.com.au/articles/asset-pipeline-tips/. However it’s quite high level, so here’s a step by step instruction on how I accomplished the task.

First, add the asset_sync gem to your Gemfile inside the assets group, and bundle install it

gem "asset_sync"

Go to /config/application.rb, append any custom js and/or css files to the precompile list

config.assets.precompile += %w(corporate.css corporate.js reset.css)

Edit /config/environments/production.rb to add the asset_host option

config.action_controller.asset_host = "http://#{ENV['FOG_DIRECTORY']}.s3.amazonaws.com"

Run the following to generate the asset_sync initialis(z)er. Tweak the config as your wish.

rails g asset_sync:install --provider=AWS

Next up, add all necessary AWS credentials as ENV vars

export FOG_DIRECTORY=xxxxxxxx
export AWS_ACCESS_KEY_ID=xxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxx

One thing to notice is the FOG_DIRECTORY value. It is the bucket name you created on S3. I was using _ in the bucket name, asset_sync or fog to be specific was complaining about the name. Updating the bucket names to use . instead of _ muted all the noises.

I also configured the above 3 ENV vars on Heroku, but I think only FOG_DIRECTORY is required.

Now, precompiling assets locally by doing

bundle exec rake assets:precompile

The above will put all the compiled assets in /public/assets. Now we need to make git happy.

First, stage and commit /public/assets/manifest.yml. Then edit .gitignore by adding

#Ignore compiled assets except the manifest
public/assets/
!public/assets/manifest.yml

This will make git track the asset manifest, but ignore all the other compiled asset files (coz they are already uploaded to S3 as a post asset compilation step by asset_sync).

All good, all working… But wait a second … Now when developing locally, compiled assets from /public/assets are all loaded … trouble !!!

At the moment, I’m doing what’s suggested

bundle exec rake assets:clean && git checkout public/assets/manifest.yml

That’s all.

Published: 2012-07-11

Pagination with Kaminari

I haven’t written any blog posts for a while. I was either busy hacking Objective-C or just being sick instead …

Here’s one Ruby thing I did today. Nothing fancy, just pagination using Kaminari

After adding the Kaminari gem in the Gemfile and bundle install, we start from the controller. All we needed to do is to chain the page(params[:page]) method call after the ordered scope

def typed_requests
  @requests = Request.ordered.page(params[:page])
end

Next up, adding the pagination HTML to the view

= paginate @requests, :remote => true

To finish it off, some coffeescript to listen on the new pager links

jQuery ->
  $('.pagination a').live 'click', (e) ->
    e.preventDefault()
    unless $(this).parent().attr('class').match(/active|disabled/)
      $.get $(this).attr('href'), (data) ->
        $(".container").html(data)

A side note, yesterday, Symfony master Fabien Potencier wrote a post PHP is much better than you think. It stirred up quite some controversy. As someone who worked with PHP for a good 6 to 7 years before moved on doing Ruby, my response is simple. Putting aside how good or bad PHP and its entire ecosystem is, the ease of use from Ruby and all these Ruby gems beats PHP hands down …

Published: 2012-07-05

Ruby (1.9.3) Psych YAML parser issue

Ruby 1.9.3 uses a more restrict YAML parser, Psych, instead of the old Syck parser. This caused me issues when using Settingslogic. Even doing what suggested by Settingslogic to set the YAML parser back to Syck didn’t help.

A simple test script

Running it under Ruby 1.9.3 outputs this

Using: psych
{"defaults"=>{"cool"=>{"bang"=>"wow", "fruit"=>"apple"}}, "development"=>{"cool"=>{"fruit"=>"banana"}}}
Using: syck
{"defaults"=>{"cool"=>{"bang"=>"wow", "fruit"=>"apple"}}, "development"=>{"cool"=>{"fruit"=>"banana"}}}

You could see “development” lost its “bang” hash key.

There are ways to get around the issue by re-building Ruby with libyaml flag. I consider this as harmful than helpful. So I reverted my application.yml back to its very dumb form, no default options, no merging, just duplicate all settings for all environments. It’s definitely stupid to do so, but for now, it’s considered as a temporary workaround.

Published: 2012-01-29

Ruote workflow engine with RabbitMQ (Part 2)

I explained how I setup Ruote and RabbitMQ in last post. From there on, you basically already have a working system. All there’s left is to have external services subscribe to the RabbitMQ job queues, parse the workitem JSON, do whatever and post the modified/amended JSON back to RabbitMQ, default queue is named “ruote_workitems”, which is what the RuoteAMQP receiver listens on.

In this post, I’ll demonstrate how a DaemonKit generated daemon script consumes the RabbitMQ job messages and how we construct a DaemonKit compatible Ruote process definition.

Generate DaemonKit project

rvm 1.9.3@daemonkit_ruote
gem install daemon-kit
daemon-kit daemon -i ruote
cd ruote
bundle

Make daemon listen to correct AMQP queues by editing ruote.yml config

defaults: &defaults
  amqp:
    queues:
      - ldap_job
      - email_job

Edit participant class by adding the following 2 methods

def ldap
  workitem["ldap"] = "done"
  # just to demonstrate the AMQP message carries the ID fields and it can be consumed
  workitem["original_cust_id"] = workitem["cust_id"]
end

def email
  workitem["email"] = "done"
end

Let’s start everything up

rabbitmq-server
cd /my-ruotekit-enabled-rails-project && rails s
RAKE_TASK=true rake ruote:run_worker
cd /my-daemonkit-ruote-project && bin/daemon

Now let’s work out how we wire everything up by carefully crafting a process definition.

Go to http://localhost:3000/_ruote/processes/new in your browser and enter the following as process definition

Ruote.process_definition :name => 'suspend_account', :revision => '0.1' do
  sequence do
    ldap :command => "/sample/ldap"
    # slot in editor to check ldap process result status
    email :command => "/sample/email"
    # slot in editor to check email process result status
    notifier :forget => true # this could be email or final results back in notification queue
  end
end

Enter the following into the workitem fields box

{"cust_id": 111}

Then launch it! If everything goes well(it should!), you’ll see no processes listed under the ruotekit app interface. And you’ll have 4 queues listed under RabbitMQ’s management interface.

  • email_job, 0 job
  • ldap_job, 0 job
  • notify_job, 1 job
  • ruote_workitems, 0 job

By popping the message out of the notify_job queue, you’ll see the serialised JSON string contains ldap done and email done key-value pairs. It shows me the job is done, and all participants are exercised correctly.

Now, we go back a step and explain a little on the process definition we crafted.

If a remote participant will be processed by a DaemonKit daemon script, we need to give each of those participants a hash, which contains a :command key. The value for :command key needs to follow a certain convention too. The convention is “/participant_class_name/method_name”. DaemonKit will classify and constantize “participant_class_name” in to ParticipantClassName object, and “method_name” will be sent to it. So in our example, when the daemon script sees command “/sample/ldap”, it’ll invoke a call to “Sample#ldap”.

The value we entered into the workitem fields textbox is simply be merged into the initial workitem hash, and it’ll be sent along to all participants.

That is it. Hope it all makes sense, and please tweet me if you found I’ve got any bits laid out wrongly or explained wrongly.

Published: 2012-01-20

Ruote workflow engine with RabbitMQ (Part 1)

At MYOB, I am assigned the task to build a component based workflow application. There are several requirements this application has to address. Here are a few

  • Decoupled components
  • Resilient workflow
  • Configurable workflow
  • Every single piece of the application needs to be scalable

A messaging based system becomes somewhat a natural choice to address the decoupling, distributable and scalable issue. While in the hunt of a good workflow system, I was pointed to a Ruby gem called Ruote by @jmettraux. Ruote has also got several companion gems. One of them is ruote-amqp.

Rails Sample Ruote AMQP is the most complete example I could find on Github. This excellent sample project almost covers all aspect. However its Rails default README file doesn’t do its justice. I had to dig hard to figure out how all the pieces are put together.

Enough mumbling from me. The following are the steps I implemented to get an end-to-end Ruote + RabbitMQ going on my MBP. As prerequisites, I have Homebrew and RVM installed.

Install RabbitMQ

This is rather simple with the help from Homebrew.

brew install rabbitmq
rabbitmq-plugins enable rabbitmq_management
rabbitmq-server

Now you can monitor your running RabbitMQ instance from the browser at http://127.0.0.1:55672/mgmt/ (login using guest - guest)

Setup RuoteKit enabled Rails project

Create a new rails app called lcp

rails new lcp

Step in the project root, create .rvmrc

rvm 1.9.3@lcp --create

Run bundle install after amending Gemfile with

gem "ruote", "~> 2.2.0"
gem "ruote-kit", "~> 2.2.0.3"
gem "ruote-amqp", "~> 2.2.0"

Add ruote kit initializer. As you could see, the file shown is a work in progress.

# config/initializers/ruote-kit.rb

# TODO: move setting values into a config file
AMQP.settings[:host] = '127.0.0.1'
#AMQP.settings[:vhost] = '/'
#AMQP.settings[:user] = 'guest'
#AMQP.settings[:pass] = 'guest'

# run ruote engine without worker
# run ruote rake task for workers, rake ruote:run_worker
RuoteKit.engine = Ruote::Engine.new(RUOTE_STORAGE)

if ENV["RAKE_TASK"]
  RuoteAMQP::Receiver.new(RuoteKit.engine)
else # don't register participants in rake tasks
  RuoteKit.engine.register do
    participant :ldap, RuoteAMQP::ParticipantProxy, :queue => "ldap_job"
    participant :email, RuoteAMQP::ParticipantProxy, :queue => "email_job"
    participant :notifier, RuoteAMQP::ParticipantProxy, :queue => "notify_job", :forget => true
    #participant :editor

    # register the catchall storage participant named '.+'
    catchall
  end
end

# when true, the engine will be very noisy (stdout)
RuoteKit.engine.context.logger.noisy = false

Add ruote worker rake task

# lib/tasks/ruote.rake

namespace :ruote do
  desc "Run a worker thread for ruote"
  task :run_worker => :environment do
    RuoteKit.run_worker(RUOTE_STORAGE)
  end
end

Last not least, mount RuoteKit to our Rails app by adding the following to the routes file

# ruote-kit
match '/_ruote' => RuoteKit::Application
match '/_ruote/*path' => RuoteKit::Application

This completes first half of the story. We now have a running RabbitMQ instance with admin monitoring, and a RuoteKit enabled Rails app. The Rails app has several RuoteAMQP participants registered, a ruote rake task ready to run Ruote workers, and of course the RuoteKit rack app that allows us to manually launch Ruote jobs.

In the next post, I’ll complete the full picture by implementing a DaemonKit generated Ruote daemon script, which processes the AMQP jobs published by the ruote worker running on the Rails app side.

But for now, PEACE!

Published: 2012-01-20

Ruby basics - Array

Here are some basic Ruby array methods that I should learn to use.

**& and ** Array intersection and union
a = [1, 2, 3]
b = [2, 3, 4]
a & b               #=> [2, 3]
a | b               #=> [1, 2, 3, 4]

collect and select

a = [1, 2, 3, 4]
a.collect(&:odd?)   #=> [true, false, true, false]
a.select(&:odd?)    #=> [1, 3]

delete_if and reject

a = [1, 2, 3, 4]
a.reject(&:odd?)    #=> [2, 4]
a                   #=> [1, 2, 3, 4]
a.delete_if(&:odd?) #=> [2, 4]
a                   #=> [2, 4]

push, pop, shift and unshift

a = [1, 2, 3, 4]
a.push 5            #=> [1, 2, 3, 4, 5]
a                   #=> [1, 2, 3, 4, 5]
a.pop               #=> 5
a                   #=> [1, 2, 3, 4]
a.unshift 0         #=> [0, 1, 2, 3, 4]
a                   #=> [0, 1, 2, 3, 4]
a.shift             #=> 0
a                   #=> [1, 2, 3, 4]
Published: 2012-01-02

Ruby basics - String

One night, I was studying some code written by Aleksey. I spotted his usage of Ruby string’s “%” notation. It shows me how elegant Ruby can be, given knowing all those Ruby basics.

To confess, I learned (and still learning) Ruby from learning Rails. At this moment, I defintely think it’s a bad idea! Basics should always be taught/learned before the magical frameworks. So, I’ll start a series of posts to go through some of these basics.

Ruby string % notation

%Q - Interpolated string. Character used after %Q will be used as the new string delimeter, hence it should be balanced. This makes normal string delimeters such as “ and ‘ become auto-escaped. If the new delimeter also appears in the string, it only needs to be manually escaped if it’s unbalanced.

%Q[Don't need to escape " and '. Need to escape \[ manually, but not [ and ]]

%r - Similar to %Q, but constructs a Regexp object. No need to escape those slashes!

%W - Feed it with a string of words separated by spaces, get back an array of words.

%x - Interpolated shell commands. Same as using backticks.

Published: 2011-12-23

New years resolutions

Another year almost goes passed by. Time for doing this new years resolutions thing again.

Let’s look back first, and see what I have accomplished in 2011.

  • Work hard on my MYOB works. Make the project we’re working on a big success!
  • Work smart on my freelance works. Try getting recurring income.
  • Rails 3 is out for a while now. I WILL make a pet RoR app this year!
  • Get myself into Test Driven Development and study Behaviour Driven Development.
  • Keep spending quality time with family. At least 2 road trips.
  • Take over the duty of sending Oscar to his childcare (that is 3 mornings a week).

Okay! 5 out of 6 items are stiked out. Not too bad!

Now, look forward. In 2012, here are the things I want to make happen.

  • I’ll be re-joining MYOB and work on a new project. I want a big success on that!
  • Keep working on freelance recurring income. I believe things will work out this year.
  • Finish off my Rails 3 side project.
  • Keep doing morning and afternoon childcare runs (3 days a week).
  • Family time! Again, at least 2 family trips.
  • EDITED: Do exercise to lose weight.

This is it! I’ll see how things are tracking by the end of 2012!

Published: 2011-12-22

Install RVM Ruby on Ubuntu 11.10

Because OS X crashes every 2nd day, I thought it’s time to give Ubuntu another good spin. First task is to setup my Ruby working environment.

On a fresh Ubuntu 11.10 VM, I firstly installed curl and git via apt-get

sudo apt-get install curl git-core

I then installed RVM using the normal RVM installation command

bash < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)

Then I tried to install Ruby 1.9.3, I ran into all sorts of issues, readline not available, yaml failed to build, etc. After some fooling around, I found that all I needed to do was to RTFM! Running the following gives me all required packages needed to install an MRI (and others) ruby…

rvm requirements

Believe or not, after doing the sudo apt-get install line from rvm requirements, RVM is happy as!

Lesson learnt: before start blaming a tool (especially a free one!), let’s check what the README says first!

Published: 2011-12-16