Doing software development with a Continuous Integration process involves a lot of automated testing. Doing tests consumes precious time so we decided to do some tinkering to make our test suite load and execute faster.
As your test suite grows, it takes longer to run the tests, and soon you have a problem because your test suite is running for over 10 minutes. Now you have two options: reduce the number of tests or optimize the test execution time.
Impulse speed
A few months ago, we wrote about how we do Continuous Integration in Ruby on Rails. Back then, a test suite for one of our projects (around 1400 tests) ran for about 7 minutes. Today, an even larger test suite (around 1700 tests) runs for 2 and a half minutes. How did we do it?
The mighty Zeus
Every time you call a rake task, your Rails application has to be loaded. The time it takes to load the application depends heavily on the size of the application and the number of gems the application is using. In our example, it was around 10 seconds. At first, 10 seconds doesn’t sound that bad, but when you realize you are calling rake tasks all the time and you have to wait for 10 seconds every time, you start to feel unproductive.
So we started using Zeus. Zeus speeds up loading time of your Rails application by preloading your application only once and then monitoring file changes in your application directory and doing reloads in the background. Now you can call a rake task with: Zeus rake task and it’ll take less than one second to start because the Rails application is already preloaded.
Running a large test suite with Zeus isn’t a big win (we cut off the first 10 seconds). It’s most useful when you’re doing TDD because you run your tests numerous times while developing. If you have to wait 10 seconds every time, it’s just not doable.
Just take this simple model spec with 9 tests as an example:
time rspec spec/models/balance_spec.rb
0m7.355s
time zeus rspec spec/models/balance_spec.rb
0m0.930s
Fig 1. Coming up to Warp 4!
Spring maybe?
With the latest release of Rails 4.1, the Spring application preloader was introduced as a part of the standard Rails environment. The idea behind Spring is the same as behind the mighty Zeus – preload your Rails application to speed up development. We tried using Spring on our projects and it works almost equally awesome. We favor Zeus primarily because it’s noticeably faster (roughly 30% faster), but we are eagerly awaiting any future reversals.
Running tests in parallel
The thing that actually makes a huge difference when running a large test suite is running tests in parallel. Modern CPUs have lots of cores, which makes utilizing those cores for testing very effective.
Luckily, there is a gem called Parallel Tests which does just that. The gem creates multiple test databases (one for each core) and then distributes all tests across those cores and runs them in parallel.
You can even combine parallel tests with Zeus, using a gem called Zeus Parallel Tests. With this combination, we reduced the running time of our test suite by over 50%.
time rspec
1m50.843s
time zeus parallel_rspec
0m52.924s
Fig 2. Approaching Warp 7!
Mock, extract and refactor
When a test is slow, it probably means that the model you’re testing is not optimized and you should be thinking about refactoring.
A typical antipattern is creating fat models with a lot of business logic and then wondering why tests are slow. It’s because you are testing things which don’t need to be tested (e.g. ActiveRecord) but mocked, and you’re needlessly hitting the database.
Let’s show this using a simple example:
class PriceList < ActiveRecord::Base
has_many :products
def total_price
products.sum(&:price)
end
end
require ’spec_helper’
describe PriceList do
let(:price_list){ create(:price_list) }
describe ’#total_price’ do
before do
create(:product, price_list: price_list, price: 100)
create(:product, price_list: price_list, price: 200)
end
it { expect(price_list.total_price).to eq(300) }
end
end
time rspec spec/model/price_list_spec.rb
0m8.396s
Now let’s try to refactor our model and extract that #total_price functionality to a separate class so we can test it in isolation:
class PriceSheet
def total_price(items)
items.map(&:price).inject(:+)
end
end
class PriceList < ActiveRecord::Base
has_many :products
def total_price
PriceSheet.new.total_price(products)
end
end
require File.expand_path("../../../lib/price_sheet.rb", __FILE__)
describe PriceSheet do
describe ’#total_price’ do
let(:item_one) { double(price: 200) }
let(:item_two) { double(price: 100) }
it { expect(PriceSheet.new.total_price([item_one, item_two])).to eq(300) }
end
end
time rspec spec/lib/price_sheet_spec.rb
0m1.530s
time zeus rspec spec/lib/price_sheet_spec.rb
0m0.270s
Now we don’t need to require the spec_helper (which loads Rails), we can mock the items and avoid hitting the database.
Fig 3. Warp 9. Engage!
We like this approach, not only because our tests are faster, but also because it drives us to write better code and avoid writing fat models.
Mocking external services
One important thing to remember is – don’t let your tests access any external web services. There are several reasons why this is bad:
- It really slows down tests
- Tests will fail if you don’t have internet connectivity
- Hitting third party services will unnecessarily use API request limits
This can be solved by disabling all remote connections in your tests, mocking the external service functionality and testing it in isolation. We use the WebMock gem for this.
