Assertion messages in tests

December 16, 2019

In this post, we’ll talk about whether or not you should use assertion messages in tests.

I received an interesting question from a fellow reader that I’d like to elaborate on:

I have a question about Assert messages - should I use the overload that contains the message parameter and use it to pass a string describing why the Assert failed?

The answer to this question comes down to two components:

Test readability — How easy it is to understand what the test is doing.
Ease of diagnostics — How easy it is to understand why the test fails.

Let’s discuss each of them separately

Test readability

People often use assertion messages to help team members and their future selfs to understand what’s going on in the test. Let’s take the following example:

[Test]
public void Hiring_a_new_team_member()
{
    var company = new Company();
    var person = new Person(UserType.Customer);

    company.HireNewMember(person);

    Assert.AreEqual(UserType.Employee, person.Type); // no assertion message
    Assert.AreEqual(1, company.NumberOfEmployees); // no assertion message
}

Instead of the straight-up assertion, you could also indicate the reason why the test’s assertions validate what they validate:

[Test]
public void Hiring_a_new_team_member()
{
    var company = new Company();
    var person = new Person(UserType.Customer);

    company.HireNewMember(person);

    Assert.AreEqual(UserType.Employee, person.Type, "Person must become an employee after hiring");
    Assert.AreEqual(1, company.NumberOfEmployees, "Number of employees must increase");
}

Such assertions help, but they come at a price. These messages require you to

Spend time writing them
Maintain them moving forward

This set of pros and cons is the same as with code comments. And, just as with comments, my advice to you is: don’t write assertion messages for the pure purpose of test readability. If you feel that the test isn’t clear without assertion messages, try to refactor it instead. In the long run, it’s easier to make the test self-explanatory than to keep it in sync with assertion messages (and it’s just a matter of time when the two go out of sync).

Introduce assertion messages only when absolutely necessary — when you can’t improve test readability in any other way. But even then, err on the side of not writing them.

The easiest way to get a quick win in test readability is to switch to a human-readable assertion notation. For example, NUnit has a special constraint-based assert model that helps you write your assertions like this:

[Test]
public void Hiring_a_new_team_member()
{
    var company = new Company();
    var person = new Person(UserType.Customer);

    company.HireNewMember(person);

    Assert.That(person.Type, Is.EqualTo(UserType.Employee));
    Assert.That(company.NumberOfEmployees, Is.EqualTo(1));
}

Or you can use my personal favorite Fluent Assertions:

[Test]
public void Hiring_a_new_team_member()
{
    var company = new Company();
    var person = new Person(UserType.Customer);

    company.HireNewMember(person);

    person.Type.Should().Be(UserType.Employee);
    company.NumberOfEmployees.Should().Be(1);
}

Such assertions read as plain English, which is exactly how you want all your code to read. We as humans prefer to absorb information in the form of stories. All stories adhere to this specific pattern:

[Subject] [action] [object].

For example,

Bob opened the door.

Here, Bob is a subject, opened is an action, and the door is an object. The same applies to code.

This version

company.NumberOfEmployees.Should().Be(1);

reads better than

Assert.AreEqual(1, company.NumberOfEmployees);

precisely because it follows the story pattern.

By the way, the OOP paradigm has become a success partly because of this readability benefit. With OOP, you, too, can structure the code in a way that reads like a story.

The ease of diagnostics

The other angle to look at assertion messages from is the ease of diagnostics. In other words, the ease of grasping why the test fails without looking into that test’s code. This is useful when reading results of a CI build.

In terms of diagnostics, follow this guideline: if you can easily re-run a test locally, this test doesn’t need an assertion message. This is true for all unit tests (since they don’t work with out-of-process dependencies), but less so for integration and end-to-end tests.

Test Pyramid

As you go higher in the test pyramid, you may find yourself needing more detailed information because integration (and especially end-to-end) tests are slower and you might not be able to re-run them at will.

But even with integration and end-to-end tests, there are ways to make the diagnostics easier without resorting to assertion messages:

Make the test verify a single unit of behavior — When a test checks only one thing, it’s often trivial to deduce what have gone wrong. (This is not always applicable to end-to-end tests since you may want such tests to verify how multiple units of behavior work back to back).
Name your unit tests properly — The ideal test name describes the application behavior in business terms, such that even a non-programmer would understand it.

And don’t forget that even without custom assertion messages, you still have messages the unit testing framework generates for you.

For example, a failure in the following assertion:

person.Type.Should().Be(UserType.Employee);

gives you this error message:

Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: Employee
Actual:   Customer

The combination of such framework-generated messages and human-readable test names makes 90% of custom assertion messages worthless even from the ease of diagnostics standpoint. The only exception is long-running end-to-end tests. They often contain multi-step verifications, and so it makes sense to use additional assertion messages to understand which of those steps failed. You shouldn’t have many of such end-to-end tests, though.

Of course, to take advantage of the framework-generated failure messages, you need to avoid generic Boolean comparisons like this:

(person.Type == UserType.Employee).Should().BeTrue();

Because they result in the following failure message:

Xunit.Sdk.TrueException: Assert.True() Failure
Expected: True
Actual:   False

Which doesn’t help with diagnostics at all.

Summary

Do you often feel lazy when it comes to writing assertion messages? Well, now you can justify it and refer your colleagues to this article.

"I’m glad it has a name"

Joking aside, though, here’s a summary:

There are two components to the use of assertion messages:
- Test readability (how easy it is to understand what the test is doing).
- Ease of diagnostics (how easy it is to understand why the test fails during a CI build).
From a test readability perspective, assertion messages are code comments. Instead of relying on them, refactor tests to be self-documenting.
In terms of ease of diagnostics, a better alternative to assertion messages is:
- Making tests verify a single unit of behavior
- Naming tests in business terms
The only exception is long-running end-to-end tests