The most important TDD rule

By Vladimir Khorikov

In the previous articles, we discussed what causes the pain while writing unit tests (mocks), and how to make TDD painless (get rid of the mocks). Today, I want to set the groundwork and discuss why mocks actually cause so much pain to us, developers.

The most important TDD rule

A quick note before we start: in this series, I refer to both test-first and code-first unit testing approaches as TDD for the sake of brevity. It’s not quite accurate, though, as TDD is about the former, not just writing tests in general. Just keep in mind that I use the TDD term as a synonym to unit testing.

Okay, so what is the most important TDD rule? The rule that entails all the stuff we were talking about in this article series? It’s this: unit tests should operate concepts that are one level of abstraction above the code these unit tests verify. Let’s elaborate on that statement.

Here’s an example of a unit test that relies on mocks:

[Fact]

public void Order_calculates_shipment_amount()

{

    var rateCalculator = new Mock<IRateCalculator>();

    Order order = new Order(rateCalculator.Object);

 

    order.CalculateShipmentAmount();

 

    rateCalculator.Verify(x => x.Calculate(order));

}

It checks calculation logic by verifying that an Order instance calls the IRateCalculator.Calculate method. We already discussed that such tests are brittle and thus not maintainable. But why is it so?

The reason is that such unit tests check the implementation and not the end result of the system under test (SUT). They go down to how the method is implemented and dictate the right way of doing this. The problem here is that there’s no such thing as the only right way of implementing the calculation logic.

We can do it in many different manners. We could decide to make Order use another calculator or even implement the calculation logic in the Order class itself. All this would result in breaking the test, although the end result could remain exactly the same.

The problem with unit tests using mocks is that they operate on the same level of abstraction as the code of the system under test. That is the reason why they are so fragile and that is the reason why it is hard to build a solid unit test suite with such tests.

The only way to avoid the brittleness is to check the end result of the method, not how this result was achieved. In other words, we need to step away from the actual implementation and rise one step ahead of the code we are unit-testing.

In the example above, we shouldn’t think about how the Order class implements the CalculateShipmentAmount method, all we need to do is verify that the amount matches the expectation.

Not only mocks

Although unit tests using mocks are prone to breaking this rule the most, the rule itself doesn’t refer to the use of mocks specifically. It is possible to write unit tests that don’t rely on mocks and still dictate the system under test how it should implement its behavior.

Here’s an example:

[Fact]

public void GetById_returns_an_order()

{

    var repository = new OrderRepository();

 

    repository.GetById(5);

 

    Assert.Equal(“SELECT * FROM dbo.[Order] WHERE OrderID = 5”, repository.SqlStatement);

}

The unit test above checks that the OrderRepository class prepares a correct SQL statement when fetching an Order instance from the database. The problem with this unit test is the same as with those that use mocks: it verifies the implementation details, not the end result. It insists the SQL statement should be the exact string specified in the test, although there could be myriad different variations of this query leading to the same result.

The solution here is the same as before: the test should operate concepts that reside on a higher level of abstraction. It means we need to verify that the Order instance the repository returns matches our expectations, without paying attention to how the repository does its job.

Shortcomings of the low-level unit tests

Tests that operate on the same level of abstraction as SUT have several drawbacks. Aside from being prone to refactoring, they force programmers do the same work twice, especially when following the test-first approach.

While writing such tests developers are constantly thinking about how the system under test is going to be implemented. They write an expectation of how it should be implemented, then implement it with the exact same code immediately after that.

Low-level tests fully mimic the SUT’s implementation and don’t add any value. While refactoring, you find they break as often as you change any of the implementation details they depend upon, so you need to also apply the same refactoring twice: in the code itself and in the tests covering that code.

Moreover, in many cases, such tests compel you to expose the internal state of the SUT and thus break the encapsulation principles. In the example with OrderRepository, we had to make its SqlStatament property public in order to verify it with the unit test.

It is no coincidence that low-level unit tests lead to poor encapsulation. In order to verify the SUT’s behavior, they need the SUT to expose its internal details to them.

That is actually a good litmus test that can help ensure you follow the “one level of abstraction above” rule while writing unit tests. If the test forces you to expose state that you would keep private otherwise, chances are you are creating a low-level test.

Summary

Low-level tests are the root cause of why TDD can be so painful. The heavy use of such tests is also the underlying reason of why there is such notion as test-induced design damage.

Don’t write tests that operate on the same level of abstraction as the code they cover. It’s hard to overestimate how important that is. This rule is the dividing line between fragile tests and a solid and reliable test suite that helps grow your software project.

If you enjoyed this article, be sure to check out my Pragmatic Unit Testing Pluralsight course too.

Other articles in the series

LinkedInRedditTumblrBufferPocketShare




  • http://www.engineerspock.com Elias Fofanov

    “The only way to avoid the brittleness is to check the end result of the method, not how this result was achieved.”

    When I see tests like you provided in the example, I find that they intended to check the interaction between objects. This kind of tests are pretended to be unit-tests, but they are actually not. They are intregrational tests in essence, and this peculiarity makes them so fragile.
    But what can we do about that in case we really want to check the interaction between objects, not actual results of a particular method?

    • cmllamacho

      I have found it difficulty to break dependencies even when using ports and adapters architecture, especially when creating the CRUD parts of the application that involve almost no domain logic.

      What I can say, is to always test the interaction between your adapter and the domain logic, not directly between domain logic and port, or adapter and port. If your unit tests maintain that level of abstraction you can swap your adapter for a class with the same interface and in theory your unit tests should work.

      For testing adapter and ports, then you can use integration tests for that particular couple of modules.

      You can check this post by Uncle Bob regarding that type of architecture….

      https://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html

    • http://enterprisecraftsmanship.com/ Vladimir Khorikov

      >”But what can we do about that in case we really want to check the interaction between objects, not actual results of a particular method?”

      That’s a good question! I was asked it in the previous post already. The thing is, you as a client of those objects usually don’t care how they interact with each other. What you care about is the side-effects these interactions produce, so it’s a good idea to test them with the client’s point of view, without knowing their implementation details.

      I actually think I will write a separate post on that topic with an example of how to move from testing interactions between classes to verifying the end result of those interactions.

      • Thomas Eyde

        Good point. I never thought of the abstraction levels of my tests, but I always prefer to consider my sut as a black box where I know the interface and nothing else.

        Testing a black box give me two options: Verify the result from the called method or verify the side-effects. Or both.

        Then write the simplest code to do those verifications.

  • Brian Button

    Hi, Vladimir,

    I take some exceptions to a few fundamental points of your post. You cannot generalize test driven and code before testing into the single term TDD. They are fundamentally different. Unit Testing is about proving the correctness of your code for given inputs, while TDD is about driving out the design of your software, either at the leave or the node level, through writing small examples of the intended behavior.

    In this case, testing that a calls b is an entirely proper and necessary type of test to write, because what you’re designing is the set of interactions your method needs to have to perform its job. It is peeking inside the kimono a bit, but that’s the way you have to write mediator-type objects and methods.

    I suggest reading Pryce and Freeman’s original Mock Object paper to understand more about this style of development, and also their book on Developing Object Oriented Software Guided by Tests. Both are really good reads and will explain why this style of development is so important.

    bab

    • http://enterprisecraftsmanship.com/ Vladimir Khorikov

      Hi Brian.

      I read “Growing Object-Oriented Software, Guided by Tests”, it is indeed a good read.

      Regarding your test-first vs code-first point, I agree they are different notions but at the same time I do think that the best practices I’m writing about here are applicable to both of these approaches. I’m going to write a separate post on the Stubs vs Mocks topic, I hope it will cover the concerns you raise in your comment.

  • Guillaume L

    I think “operating on the same level of abstraction as the code of the system under test” is too vague of a heuristic to be correct.

    What about stubs that you use when the SUT talks to an I/O bound object across an application seam (i.e. to an adaptor across a port, in Hexagonal Architecture lingo) ? Your test does have to operate on the same level of abstraction as that stub in order to specify which methods you stub out and what values they return, right ? Or are stub-based tests wrong as well ?

    To me, that kind of blanket statement will only confuse people and scare them away from using any form of fake-based testing, including legitimate ones.

    Regarding your first code sample, I can indeed see a problem with it. But the issue is not what is verified, it’s the mismatch between the test name and the assertion. By naming the test after the production code method (calculates_shipment_amount), you are implicitly saying that this test will check the main behavior of the method. But of course it doesn’t. Of course, the expected main behavior is to change the state of the Order object to store the newly calculated shipment amount. Not to call a calculator. However, the thing is, these are not mutually exclusive. You can have both. Verifying that the Order object was mutated correctly and checking that a calculator-shaped thing was called are both legitimate, as long as they are in different, well-named tests.

    So here’s the fallacy – by introducing that mismatch in your example between the test name and what’s really tested, you’re implicitly saying that every mock-based test will provide a false sense of security with regard to the method’s main functionality. But that doesn’t have to be. Not if you’re a well-thought programmer who names tests after what they’re really testing.

    In the end, if I had to give a most important TDD rule, it would probably be the red-green-refactor cycle itself. Most people tend to forget about it. TDD isn’t a substitute for good design skills, but the Red step is a dedicated space that forces you to think about an external design for your SUT, and the Refactor step is a dedicated space to refine its internal design. Still, TDD has no opinion about the size of the units under test, the stage in the development scenario at which it has to be applied or the usage of mocks, stubs or spies. You can have multiple nested TDD loops. TDD isn’t a squadron-level battle tactics for developing small clusters of objects, but a simple pattern that can be applied at any granularity level in the chain of command.

    • http://enterprisecraftsmanship.com/ Vladimir Khorikov

      Thanks for your comment. I partly agree. I agree with the last paragraph where you talk about the red-green-refactor cycle completely. That’s the problem with this article series. I write “TDD” but actually mean Unit Testing in general.

      I also agree with the part where you say that the test is named incorrectly. That’s indeed not the best name for it.

      However, I think the main point of the post still stands. I didn’t allude to the mismatch between the test name and its content and didn’t conclude the drawbacks of the mockist approach from that. My main point is that this kind of tests, regardless of how they are named, test implementation details. These details change often and that makes the tests signalize false negatives thus diminishing their value significantly.

      I’m finishing my next Pluralsight course at the moment and I’m planning to start writing on the topic of unit tests after that. The review for the GOOS book included.

    • http://enterprisecraftsmanship.com/ Vladimir Khorikov