When to include external systems in testing scope

When it comes to external dependencies (dependencies outside the process that hosts your application, such as a database, a 3rd party system, etc.), there’s no single guideline regarding how to work with them in tests. Sometimes, it makes sense to include them into the testing scope and verify your application together with those dependencies directly. In other situations, the best approach is to substitute them with mocks (or other test doubles). How to determine which way to go?

To do that, you need to look at what exactly the dependency represents.

All external dependencies can be roughly attributed to two categories: those you control, and those you don’t. The first category is about systems that reside "close" to yours. The application database and the file system are good examples here. If your application is the only one working with the database, you have a full control over its underlying structure and data in it. The same is true for the file system.

If, on the other hand, your application shares access to an external dependency, your options in affecting its internal workings are limited. For example, when working with a payment API a bank exposes, you can’t simply change that API. It might be developed by some other team, as well as such a change could affect multiple clients, so backward compatibility must be maintained. And you need to also keep in mind that a sheer interaction with that service may in itself have an effect on other applications integrating with it.

In other words, communications with external dependencies shared across multiple applications are visible to the outside world whereas communications with dependencies you fully control are not.

This distinction is an important one. Remember that the most important rule of unit testing is verifying the end result of the SUT as it’s seen from the outside world. This is the only way you can avoid getting false positives and thus increase the value of your test suite.

Calls to external systems you don’t have control over comply with that definition. They are the end result the SUT produces because they are visible from the outside of your system. On the other hand, dependencies you do control comprise a single cohesive whole with your application. Calls your application issues to them are not visible from the outside.

Communications with external systems outside of your control comprise part of the bounded context’s contract because they are visible from the outside of that context. On the contrary, communications with external systems you do control are internal implementation details.

Here’s how it can be depicted graphically:

Your database, if not shared across multiple applications, is part of the same bounded context. When someone calls your bounded context’s API to, say, create a new product:

they don’t expect to then go directly to the database and work with that product from there. It’s not part of the contract (post-condition) the API method proposes. What it does propose is a promise that after the client creates the product, it can then fetch it using another API method:

and that product will come out as expected, with all the data the client has set up for it.

The underlying storage in this scheme is an implementation detail. It’s not reachable by the clients, they don’t know anything about it. This implementation detail can therefore be changed freely, your tests shouldn’t bind to the way your application communicates with the storage.

You can’t say the same about dependencies you don’t control. The side effects you induce in them are visible from the outside. A client buying a pizza using this API method:

can expect your app to charge some amount of money from their bank account, and they can verify that payment using their bank mobile app. Your application is not the only one working with the bank API.

However, as you don’t have control over what the bank does after you ask it to charge the payment, you can’t declare the statement "the client’s credit card balance increases by 8 dollars" as a post-condition to the /purchase API method. The only thing you can guarantee is that you ask the bank to do that. In other words, call the appropriate API on it. Whether or not the call will be processed correctly is up to the bank itself, you can’t influence that decision in any shape or form.

So, again, the difference between external dependencies you have control over and dependencies you don’t control determines whether to include them into testing scope or not. The best way to work with the former is to exercise them directly. This way you will be able to validate the contract your application promises to fulfill. As for dependencies you don’t control, the only thing you can guarantee is that your application properly communicates its intention to them, and mocks is the best way to verify that guarantee.

This article is an important addition to the post I wrote recently regarding when to use mocks. In it, I stated that the only valid place for them is inter-system communications: communications between your and other applications. But it’s not a sufficient condition. A sufficient condition for the use of mocks would be "inter-system communications with dependencies you don’t have control over". So, the overall picture looks like this:

We can boil this guideline further using DDD terms. Use test doubles only to check communications with other bounded contexts. In that sense, communications with dependencies that are external but still reside inside your bounded context are not a subject for verifying in tests. It is better to check the state those dependencies ended up in. The communication pattern that brought them in to that state doesn’t (and shouldn’t) matter. An application database is a good example here: it is hosted by a different process but is still a part of the same bounded context.

Note that all said above is applicable to integration tests only: tests that verify how your application works with other applications. As for unit tests, I’m assuming they focus on verifying business logic which is properly isolated from external dependencies of any kind, so there’s no need in using test doubles here anyway.

Along with application databases, there’s also the concept of integration database. That is a database which is used by multiple applications and serves as an integration point between them.

An integration database is not part of some single bounded context, it spans across several of them. Because of that, it falls into the same category as other external dependencies you don’t have control over. You can’t change the structure of an integration database without introducing breaking changes and any modification you make to its data is visible to other applications. Therefore, it is preferable to treat them the same way you treat an external bounded context.

There’s a popular practice when it comes to testing how your code works with databases: replacing them with in-memory implementations. For example, SQL Server could be substituted with SQLite which gives a significant performance boost comparing to executing tests against the real database.

While the performance benefit is great, I wouldn’t recommend you employ this practice. The problem with it is that most in-memory databases don’t quite operate the same way as the normal ones. You always have a chance to fall into some edge cases, some scenarios where the in-memory database works differently.

That’s quite annoying, to say the least. Your tests may work in 99% cases and lie in other 1% either providing a false positive or a false negative depending on how the in-memory DB implementation differs from that of the normal one.

And that, to a large degree, defeats the purpose of having integration tests in the first place. Even if you have all your integration tests pass, you can’t really be sure that the integration between your code and the database works entirely correctly, and you will still need to test this integration manually anyway.

Overall, try to avoid those "in-between" solutions. Either mock your database out completely or test your code against the same type and version of the database as you use in production.