Coded UI vs White

September 22, 2015

Today, we’ll look at Coded UI and White automation frameworks in terms of building an automated UI test suite for a WPF application. We are going to compare them from different points of view, such as performance, the ease of getting started and maintainability. You will see which of these two frameworks suits what kind of projects and how to make a decision in your particular circumstances.

Coded UI vs White at first glance

Coded UI from Microsoft and open source White from TestStack are two popular choices for developers starting with automating UI tests for WPF applications.

Coded UI is shipped with Visual Studio starting from version 2010. White is an open source project, you can find it here. It’s also available as a NuGet package.

Both Coded UI and White are based on the low-level UI Automation library from Microsoft. Because of it’s being low-level, the UI Automation library is rather complex, and it’s hard to write UI tests using this library directly. Both Coded UI and White hide its complexity and introduce their own API on top of it.

Coded UI and White depend on UI Automation library

In terms of functionality, the difference between the two frameworks is quite big. While White aims at WPF applications only, Coded UI also allows for testing web applications. It might be helpful if your software consists of web-based and desktop-based parts and you want to verify them within a single test case.

The ease of getting started

Let’s start the comparison off by looking at the ease of getting started.

Unlike White, Coded UI provides a recorder functionality so you can build tests just by walking through your application and recording your interactions with it.

Coded UI recorder

It might be helpful if you want to commission the creation of UI tests to someone without coding experience.

On the other hand, with White, you need to dive into its domain model and write the code manually:

Application application = Application.Launch(@"C:\Application.exe");
Window window = application.GetWindow("Window title");
 
var okButton = window.Get<Button>(SearchCriteria.ByText("OK"));
var aboutTextBox = window.Get<TextBox>(SearchCriteria.Indexed(2));
 
aboutTextBox.Text = "I'm a software developer";
okButton.Click();

Thus, Coded UI is easier to get started with and is a winner in this part of the comparison.

Performance

Performance is another parameter we should take into account when creating UI tests. If your UI test suite contains tens or even hundreds of test cases, their performance might have a substantial impact on how fast you get the feedback from your CI environment.

It turns out that Coded UI loses in this part of the comparison completely. It takes Coded UI 5x more time to complete a typical medium sized test case:

Average results after 10 runs in seconds

White is much closer to the ground and introduces very little overhead comparing to Coded UI which gives it a great advantage in terms of performance.

Generated vs handwritten tests

While the ease of getting started and performance are significant attributes for a test framework, I believe maintainability of the tests written with it is the most important one. The degree to which you can easily change and extend your tests is vital in the long run. In my opinion, it is the single trait that makes a successful UI test suite.

It turns out that with a recorder, it’s really hard or even impossible to create automated UI tests that provide long-lasting value. Let’s look why that is so.

First of all, recorded tests are too brittle. Changing the application UI causes all the tests that rely on that part of the UI to break. When that happens, we have two options: either re-record all the broken tests or go to a UI map and implement the change manually. Re-recording is not the best option as it often requires a lot of work, so most developers end up with fixing such issues in the map files directly.

When we start working with UI maps, other issues come into play. With UI maps, it’s hard to keep up with the teamwork because of the merge problems. As the code of the UI maps is auto-generated, it’s extremely difficult to resolve merge conflicts when more than one team member works on creating automated UI tests. Splitting a single UI map into several smaller maps helps reduce the problem but doesn’t make it go away completely. You still face the same problems when two members start working on the same piece of the UI.

Often, what ends up happening is developers start using the lock mode, so to speak. When one of the developers works with a UI map, they inform other team members so that no one else can work on it at the same time. Only after the changes are committed to the source control, could the other developers start working on that map.

The third point is that it requires a lot of work to make UI maps readable. We have to clean them up after every recording: delete all the elements which are not required for the test to run. If we don’t keep the UI maps clean, they quickly get bloated with unnecessary elements and it becomes hard to understand which element stands for what. Also, in order to be able to reason about the maps, we need to rename the auto-generated fields, which is a tedious work.

It turns out that the process of creating automated UI tests cannot be delegated to a recorder. The code of the UI tests should be treated as production code. We don’t generate our production code, neither should we do that with UI tests. The only way to build a maintainable test suite is to do it manually.

Coded UI vs White: object model comparison

That brings us to the object model comparison. Aside from offering the recording functionality, Coded UI allows for coding against its object model directly. So let’s compare the object models of the Coded UI and White libraries from the maintainability point of view.

Let’s say we need to click on a particular cell in a grid table. Here’s how CodedUI allows us to do that:

public void ClickOnACell()
{
    UIGridTable table = GetGridTable();
 
    var cell = new WpfCell(table);
    cell.SearchProperties[WpfCell.PropertyNames.ColumnIndex] = "0";
    cell.SearchProperties[WpfCell.PropertyNames.RowIndex] = "0";
    Mouse.Click(cell);
}

And this is the version with White:

private void ClickOnACell()
{
    Window window = GetWindow("Northwind");
 
    var listView = window.Get<ListView>();
    listView.Rows[0].Cells[0].Click();
}

With Coded UI we need to set up the properties that are going to be used for searching the cell, whereas with White, we can ask for a particular cell in a particular row and click on it directly.

Alright, let’s take a slightly different example. Now we are going to read data from all cells in the first row of a grid. This is the version that uses Coded UI:

public void GetRowData()
{
    UIGridTable table = GetGridTable();
 
    int count = table.ColumnHeaders.Count;
    var data = new List<string>();
    for (int i = 0; i < count; i++)
    {
        var cell = new WpfCell(table);
        cell.SearchProperties[WpfCell.PropertyNames.ColumnIndex] = i.ToString();
        cell.SearchProperties[WpfCell.PropertyNames.RowIndex] = "0";
        data.Add(cell.Value);
    }
}

White:

private void GetRowData()
{
    ListView listView = GetGridTable();
 
    string[] data = listView.Rows[0].Cells
        .Select(x => x.Text)
        .ToArray();
}

I think you can see a pattern here. Coded UI’s object model is built upon its search functionality which itself is highly optimized for the use in the recorder mode. The object model implies that you know where you need to look for the info on the screen. That makes a lot of sense when you record your actions because indeed the test builder knows exactly where you clicked or what UI element you need to examine.

At the same time, when you try to code against this model directly, it causes many difficulties. For example, you cannot explore the model with IntelliSense. The grid class doesn’t contain any properties that give us a clue we can access its rows. We need to refer to the documentation in order to understand how to retrieve a cell from it.

Also, the Coded UI object model lacks encapsulation; it doesn’t prevent us from errors. We are able to use any string in the SearchProperties collection, and that would compile without a warning. Any mistake in the name of a search property would result in inability to find the required control, which might be quite frustrating. Such behavior extends the feedback loop and slows the overall development process down.

Maintainability comparison results

Both Coded UI and White allow for creating tests that rely on their domain model directly but do that in different ways. The distinctions in their approaches come from the fact they are optimized for different purposes.

The Coded UI object model has to comply with the recorder functionality requirements. That puts a lot of restrictions to its design. At the same time, White is optimized for writing automated UI tests manually. The White framework provides us with a concise and lucid object model that helps build maintainable code. White is a winner in this part of the comparison.

Summary

Coded UI is easier to get started with. At the same time, White is more performant and offers a cleaner object model to code against.

Both of these frameworks are useful in different types of software projects. On small projects or projects where you need to automate only a couple of most important user scenarios, you are better off choosing Coded UI as it doesn’t require a lot of investment and you can start getting real value right away, without spending much time.

On large projects or projects where you want to automate the most of the regression testing, it makes sense to go along with White as it allows for creation of a stable and maintainable test suite which scales over time.

If you want to learn more on how to build maintainable UI tests for WPF application, check out my Pluralsight course: Automating UI Tests for WPF Applications.