What does `bundle exec` do?

14 Jul 2015

A colleague asked me why sometimes we’ll run a command that’s provided by a Ruby gem (e.g. “rspec”) and it fails unless we type bundle exec before it.

What Bundler gives us

Bundler is a Ruby gem that’s a little special because it manages other Ruby gems. It does basically three things:

  1. It does a topological sort of the gems listed in your Gemfile to print a complete graph of your project’s dependencies into Gemfile.lock
  2. It retrieves the gems your project needs from the source(s) you’ve defined
  3. And it makes it so when you require files the right file is loaded.

The importance of ENV in Ruby

The two most popular Ruby version managers are RVM and rbenv (though I strongly recommend chruby). They work in totally different ways (RVM overwrites your cd shell builtin function!) but the output of all these tools is the same: They export environment variables so your Ruby code operates on a particular version of Ruby with a particular set of gems.

Here’s a diff[1] of the environment when we’re using one version of Ruby versus another:

$ diff <(rvm use 2.2 && env) <(rvm use 2.1 && env)
1c1
< Using /Users/jackdanger/.rvm/gems/ruby-2.2.1
---
> Using /Users/jackdanger/.rvm/gems/ruby-2.1.5
13,14c13,14
< GEM_HOME=/Users/jackdanger/.rvm/gems/ruby-2.2.1
< GEM_PATH=/Users/jackdanger/.rvm/gems/ruby-2.2.1:/Users/jackdanger/.rvm/gems/ruby-2.2.1@global
---
> GEM_HOME=/Users/jackdanger/.rvm/gems/ruby-2.1.5
> GEM_PATH=/Users/jackdanger/.rvm/gems/ruby-2.1.5:/Users/jackdanger/.rvm/gems/ruby-2.1.5@global
19c19
< IRBRC=/Users/jackdanger/.rvm/rubies/ruby-2.2.1/.irbrc
---
> IRBRC=/Users/jackdanger/.rvm/rubies/ruby-2.1.5/.irbrc
37c37
< MY_RUBY_HOME=/Users/jackdanger/.rvm/rubies/ruby-2.2.1
---
> MY_RUBY_HOME=/Users/jackdanger/.rvm/rubies/ruby-2.1.5
42c42
< PATH=/Users/jackdanger/.rvm/gems/ruby-2.2.1/bin:...
---
> PATH=/Users/jackdanger/.rvm/gems/ruby-2.1.5/bin:...
51c51
< RUBY_VERSION=ruby-2.2.1
---
> RUBY_VERSION=ruby-2.1.5
103c103
< rvm_ruby_string=ruby-2.2.1
---
> rvm_ruby_string=ruby-2.1.5

Looking at just the keys, you can see that RVM sets a little bit of internal stuff but mostly it just sets the PATH for running commands like ruby and rspec and it sets GEM_HOME and GEM_PATH and such:

$ diff <(rvm use 2.2 >/dev/null && env) <(rvm use 2.1 >/dev/null && env) | \
awk -F= '{print $1}' | awk '{print $2}' | sort | uniq

GEM_HOME
GEM_PATH
IRBRC
MY_RUBY_HOME
PATH
RUBY_VERSION
rvm_ruby_string

Now, I wish I could tell you that the Ruby core source had a unified way of resolving dependencies from these environmental variables. Nothing could be further from the truth. Rubygems is a project outside of Ruby core that is periodically imported in to the main Ruby trunk as a vendored dependency. It reads GEM_HOME and GEM_PATH and such. And then Bundler is a separate project that overloads the Rubygems functionality.

And all of this song and dance is to make it possible for you to type require 'my_file' and the right my_file.rb or my_file.bundle[2] or my_file.so is read from disk, parsed, and evaluated line-by-line.

The actual implementation is full of edge cases but actually kind of simple to understand: require and gem are just methods in Ruby, not reserved keywords. It works like this:

If you wanted you could add another package manager on top of this that undefines the existing methods and does something totally different.

Right, but why do we need “bundle exec”?

Let’s use the same diff trick to compare a process’s environment with and without bundle exec:

$ diff <(bundle exec env) <(env)
< _ORIGINAL_GEM_PATH=/Users/jackdanger/.rvm/gems/ruby-2.2.0:/Users/jackdanger/.rvm/gems/ruby-2.2.0@global
< RUBYOPT=-rbundler/setup
< RUBYLIB=/Users/jackdanger/.rvm/gems/ruby-2.2.0/gems/bundler-1.10.5/lib

That -rANYTHING is the same as evaluating require "ANYTHING" in Ruby source code. It’s weird that there’s no space but it, hilariously, allows there to be an ubygems.rb so ruby -rubygems works.

RUBYOPT is the flag that’s passed to Ruby when invoked. For example, the following two lines are equivalent:

$ ruby -rrails -e 'puts defined?(Rails)'
$ RUBYOPT=-rrails ruby -e 'puts defined?(Rails)'

And RUBYLIB is the default load path that Ruby searches in when you require something:

$ echo 'puts :hi_mom!' > myfile.rb
jackdanger $ ruby -rmyfile -e ''
/Users/jackdanger/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- myfile (LoadError)
        from /Users/jackdanger/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
jackdanger $ RUBYLIB=. ruby -rmyfile -e ''
hi_mom!

Bundler’s bundle exec requires the Bundler setup file which lets Bundler do all it’s file-finding hacks for when you later require something. But you may find in some cases that if your environment variables are already set up just right the requires might just work without Bundler’s help. Even then it doesn’t hurt to use bundle exec so you always know you’re using the right versions of everything from the Gemfile you have defined.

Do I have to keep typing this?

No, you don’t. You could hide this command behind aliases:

for cmd in rspec ruby rubocop rails; do
  alias $cmd="bundle exec $cmd"
done

Special thanks to Agis for reviewing this post and providing critical feedback.



Footnotes:

1. ^ That odd <() notation I’m using for executing subcommands is pretty useful. Check this out:

$ cat <(sleep 3 && date) <(sleep 3 && date)
Mon Jul 13 18:32:09 EDT 2015
Mon Jul 13 18:32:09 EDT 2015

cat is short for ‘concatenate’ and was originally written to squish two files together (cat file1 file2 > file3). What I’m doing here is concatenating the output of two different subcomands. We can see how this works if I print the arguments to cat to the screen rather than trying to read them and then print them:

$ echo <(sleep 3 && date) <(sleep 3 && date)
/dev/fd/11 /dev/fd/12

The terminal not only creates subprocesses for the stuff in parentheses but makes a new file descriptor that’s readable by any program (/dev/fd/{n}) and swaps that in as the argument to the current command. So it’s the same as if I’d done this:

$ date > date1
$ date > date2
$ echo date1 date2

The bonus here is that the subcommands are executed in parallel, hence the datestamps matching on the above example. This allowed me to diff the relatively slow rvm use && env subcommand in half the time as would normally be required.

2. ^ .bundle is a packaging format for Mach executables on OS X. .so is for shared objects (binaries meant to be used as libraries rather than executed directly).


Please if you found this post helpful or have questions.