Styles of unit testing

June 9, 2016

The topic described in this article is part of my Unit Testing Pluralsight course.

In this post, I’ll describe different styles of unit testing and compare them in terms of their value proposition.

Styles of unit testing and their value proposition

There are 3 major styles of unit testing. The first one is functional, where you feed an input to the system under test (SUT) and check what output it produces:

Functional style

Obviously, this style only works for SUTs that don’t generate side effects. The yellow slash here depicts the point at which the examination is done.

The second one is state verification:

State verification

With it, you verify the state the SUT ended up getting into after the operation is completed.

And finally, the third one is collaboration verification:

Collaboration verification

Here, you focus on collaborations between the SUT and its neighbors: you check that all collaborators got invoked in a correct order and with correct parameters.

Now, let’s look at these styles from a value proposition perspective. In the last post, we defined what a valuable test is. It is a test that:

Has a high chance of catching a regression bug.
Has a low chance of producing a false positive.
Provides fast feedback.

Do the styles of unit testing I mentioned above have the same value proposition? They don’t. Although all three provide a fast feedback and, if not used for covering trivial functionality, have a high chance of catching a regression error, the second component differs significantly.

The functional style has the best protection against false positives. Among all other parts of the SUT, its input and output tend to change less frequently during refactorings. You can alter the internal implementation of the SUT completely and tests written in a functional way will not raise a false alarm as long as the signature of the method under test stays in place.

Note that there’s a difference between an input in a functional sense and a collaborator. While we pass both of them to the SUT the same way, the input value is immutable, whereas the collaborator either maintains its own internal state which can change over time or refers to an external dependency, for example to a database. Here is an example of the former:

public double Calculate(double x, double y)
{
    return x * x + y * y + Math.Sqrt(Math.Abs(x + y));
}

And here’s an example of a collaborator:

public double Calculate(ICalculator calculator, double x, double y)
{
    calculator.Push(x);
    calculator.Push(y);
    return calculator.CalculateFormula();
}

In the first sample, both the inputs and the output are immutable. The instance of ICalculator, on the other hand, does contain mutable state.

The second style of unit testing - state verification - is more prone to false positives but is still good enough as long as we verify the SUT’s state via its public API and don’t try to analyze its content via reflection. This, of course, requires you as a developer to think carefully about the parts of the SUT you expose publicly. You shouldn’t reveal its implementation details as that would introduce a tight coupling between the tests and the SUT.

As long as the SUT’s encapsulation is not violated, state verification is a good approximation to the functional style of unit testing. It’s still unlikely to produce false positives because the SUT’s public API tends to stay in place in the face of refactoring or addition of new functionality.

The following is an example of such a SUT:

public class Customer
{
    public string Name { get; private set; }
    public CustomerStatus Status { get; private set; }
 
    public Customer(string name)
    {
        Name = name;
        Status = CustomerStatus.Pending;
    }
}

There’s a valuable piece of domain knowledge to test here. That is, new customers must reside in a pending status. The verification can be performed via the Customer’s public API - its Status property:

[Fact]
public void New_customer_is_in_pending_state()
{
    var customer = new Customer("John Doe");
 
    Assert.Equal(CustomerStatus.Pending, customer.Status);
}

What about the third style - collaboration verification? This is where the value of unit tests starts to degrade.

The collaborations the SUT goes through in order to achieve its goal are not part of its public API. Therefore, binding unit tests to the SUT’s collaborations introduces coupling between the tests and the SUT’s implementation details. Such coupling makes the tests produce a lot of false positives as the collaboration pattern tends to change often during refactorings. And that, in turn, diminishes the value of such tests and the overall return of investments in them.

The problem with mocks and with the mockist approach in general is that they aim at verifying collaborations and often do that in a way that ties unit tests to the implementation details of the SUT.

Another problem with the mockist approach is that it encourages destructive decoupling - the situation in which you decouple parts of your code base so much that they lose their intrinsic cohesion. That is what I believe DHH referred to when he wrote about test-induced design damage.

With mocks, your class diagram often starts looking like this:

Test-induced design damage

Lots of connections - often introduced without necessity - result in an architecture with a large amount of cyclic dependencies which make it hard to grasp and understand the bigger picture.

The reason here is that classes are too fine-grained to consider them separate agents. Treating classes as collaborators makes the overall architecture too verbose and fragile.

Overall, the use of unit tests with mocks has the worst value proposition because of the bad signal/noise ratio. They do help catch regressions but they do that at the expense of producing lots of false positives. It defeats the whole purpose of unit testing: having a solid test suite which you can trust and rely upon.

The use of mocks is just a sign of a problem, of course, not the problem itself. However, it’s a very strong sign as it almost always signalizes an issue with your approach to unit testing. If you use mocks, you most likely couple your unit tests to the SUT’s implementation details.

There are a few legitimate use cases for mocks, though, but not in the context of unit testing. They can be useful in integration testing when you want to substitute a volatile dependency you don’t control. Also, mocks as an instrument also can be quite beneficial if you use them to create stubs. More details on that in the next post.

By the way, there is another style of unit testing: property-based testing which you can view as the functional style on steroids. It has essentially the same traits but does its work even better due to checking multiple inputs and outputs at once.

Summary

Seems that it’s time to end the post and I didn’t even get to the architectural changes I promised in the last article. I’ll cover them in the next post, along with the stuff I teased here (legitimate use cases for mocks).

Let’s summarize this one with the following:

There are 3 styles of unit testing: functional (output verification), state verification, and collaboration verification.
Functional style is the best one in terms of its value proposition as it has the lowest chance of producing false positives.
State verification is the second best choice. As long as you encapsulate your domain classes well and unit test them against their public API, you should be good.
Collaboration verification has the worst value proposition because such tests produce a lot of false positives.
Treating your classes as collaborators also encourages you to introduce lots of unnecessary dependencies which makes the overall architecture more complex and harder to grasp.