Real World Unit Test Problems

At Bloc, we value the automated testing of our code. We allocate time for writing automated tests of our code whenever we estimate the time it takes to complete a feature.

While the emphasis on writing tests makes our code more robust and helps us catch errors/bugs before we ship new code, we often find that testing can slow us down. One complaint inevitably seems to come up in teams focused on high code coverage:

"Why do my tests take so long to run?!"

Recently, we've focused on digging into that question at Bloc and it's led to some surprising conclusions. One of our working hypotheses has been that we're creating too many unnecessary objects when we set up our tests.

Unit testing in an ideal world

Before we talk about how we unit test at Bloc, let's just make sure we're all on the same page with respect to what a unit test is. I'm a huge fanboy of Sandi Metz and subscribe to her approach towards unit testing.

A unit test is meant to test an object's public interface. The subject of unit tests should be the messages sent to a given Object, with the assertions being made on either what the Object sends back or the public side-effects created by the Object. If we follow those principles (cliffnotes here), the resulting tests will be lightweight, fast, easy to understand, and will reinforce good Object Oriented Design.

Unit testing in our Rails application

We often see examples of unit testing in a pure Ruby context. It always leaves me wondering -- why don't the unit tests that I write for the Bloc code base look like the simple, clean tests I see in talks, books, or blog posts?

Part of the reason is that our “unit tests” aren’t actually acting as unit tests. Most often, we call the tests for our models and services "unit tests" when we actually deviate from that by testing the integration/interaction between multiple objects in those tests. So the question I have to ask is -- how do we get our Rails tests to align more closely with clean, academic examples?

Case study

Recently, we were creating a new service object to calculate the number of days a student has been frozen in our program. This FrozenDaysCalculator takes a complex object (an EnrollmentChain with at least two associated Enrollments), analyzes the EnrollmentChain, and returns the number of days frozen. You can think of the EnrollmentChain as a chain of events describing a student's full enrollment experience, whereas each Enrollment is a state in that experience: active, frozen, active again.

The first unit test I wrote for this service was:

describe '#total_frozen_days' do  
  context 'when a chain has never been frozen' do
    it 'returns 0' do
      calculator = Services::FrozenDaysCalculator.new(chain: enrollment_chain)
      expect(calculator.total_frozen_days).to eql(0)
    end
  end
end  

This test's intentions are pretty clear, but you'll notice there's one thing missing -- we didn't define the enrollment_chain variable anywhere. To define the EnrollmentChain and associated Enrollments, we turned to our handy factories to assist in quick setup.

describe '#total_frozen_days' do  
  context 'when a chain has never been frozen' do
    it 'returns 0' do
      enrollment_chain = create(:enrollment_chain, standing: 'current')
      create(:enrollment, :extended, chain: enrollment_chain)
      create(:enrollment, :latest, chain: enrollment_chain)

      calculator = Services::FrozenDaysCalculator.new(chain: enrollment_chain)
      expect(calculator.total_frozen_days).to eql(0)
    end
  end
end  

Before we use any factory, we look to make sure that it has the desired attributes. What we found when we looked at the Enrollment factory dismayed us a bit (slightly edited for the sake of brevity):

FactoryGirl.define do  
  factory :enrollment do
    birth_date { 1.week.ago.beginning_of_week - 2.days }
    course { create(:course) }
    course_start_date { 1.week.ago.beginning_of_week }
    mentor { create(:user, :mentor) }
    status 'active'
    ...    
  end
end  

Even with that small snapshot of the contents of the factory, it’s clear that two objects, course and mentor, are being created that aren't necessary for testing the FrozenDaysCalculator! And since we have to create at least two instances of Enrollment to appropriately test the service object, that means at least four superfluous objects would be created for this simple test.

When we tried to find a way to create an Enrollment instance without an associated Course (the FrozenDaysCalculator has no interest in Courses), we found that the Enrollment would raise an error. This signals a strong code smell that Enrollment is too tightly coupled to the Course model and provides direction on how we might start the decoupling process.

After this entire investigation, did we find out why our tests take so long to run? In truth, it's not clear due to the complexity of the problem. While the creation of associated objects may not have a dramatic impact on how long it takes to run a single test, it's reasonable to assume that unnecessary object creation has a significant impact in the aggregate. One happy outcome of this investigation has been increased team vigilance towards creating only the necessary objects in order to make an assertion. We'll have to continue our investigation identifying other reasons why our test suite takes so long to run.

On the other hand, I feel more comfortable answering why our unit tests aren't like the more academic examples we find in books and blogs posts: When objects in your code base do not adhere to SOLID object oriented design patterns, the unit tests that get written for those objects begin to look more like integration tests rather than unit tests. Integration tests have their own sets of concerns and strategies that ought to be addressed in a future blog post, but I believe that they don't align neatly with the same strategies as we might adopt for unit tests. Because we were writing integration tests while calling them "unit tests", it's reasonable to expect that "unit tests" we've written look drastically more complex than academic examples.

On a happy note, a useful heuristic emerged from this investigation. When we find base factories that require associated object creation, it's likely that there are high levels of coupling at play and we will not be able to test those objects' public interfaces easily.

On the flip side, writing unit tests that only use simple, barebones objects can help expose tight object coupling/object oriented anti-patterns!

By remaining conscientious of how painful it is to write unit tests and by maintaining bare-bone base factories, we anticipate a higher level of code quality in our code base.

Hack the planet!