Testing is fun again

Using gtest is by far nicer than cppunit. This is a brief overview of it's usage and some of it's features.

Posted on July 11, 2011 C++, gtest, testing .

A long time now I’ve been faithfully clinging on to cppunit, a C++ version of the famous X-Unit-style test frameworks. It has served me well and working around the little glitches did not hurt too much. But it’s a dead project and does not seem to receive love from the developers. Thus I am not really sad to let it go.

But why let it go at all? No more unit-testing? Well… I got kicked in my butt by a colleague that recently started using google test. So I gave it a try and it has been a pleasure to work with so far! For one it is a very active project and in active use at google (where they do a lot of C++). That alone makes it a much safer bet then cppunit.
Much more important though is the fact that it is simply better. A lot of things are made easier or even possible.
Take for example the distinction between fatal and non fatal error that google test offers: this makes it possible to collect non-fatal errors without exiting the test case (using EXPECT) or state clearly that it makes no sense to continue the test case with a fatal error (using ASSERT).

Adding messages to failed test case is a breeze… no more messing around with strings, ostreams etc., simply stream the message into the macro:

EXPECT_EQ(0, transaction.getNumberOfUsedJobs()) << "no jobs added";

Using custom messages will make the resulting error message in case of a test failure much more readable:

[  FAILED  ] TransactionTest.addJob (0 ms)
test/src/TransactionTest.cpp:90: Failure
Value of: transaction.getNumberOfUsedJobs()
  Actual: 1
Expected: 0
no jobs added

Two very useful kinds of test are Death Tests and static assertions.

Death Tests

Death tests let you check for an expected failure like an assert or a bad exit code. They are implemented using a forked sub-process that will be monitored eventually.
The following example shows how to make sure that an assert will happen. Note the regular expression string that can be added to check for the kind of error that has occurred.

#include "gtest/gtest.h"
namespace {
int foo(int x) {
    assert(x < 5);
}
TEST(DeathTest, foo) {
  ASSERT_DEATH( foo(10); , ".*Assertion.*");
}
}  // namespace

Static Assertions

Static asserts come in handy e.g. when you have some compile time configuration or a template meta program that you want to check. (of course boost type traits offer much more here)

#include "gtest/gtest.h"
template <typename H, typename T>
struct typelist
{
    typedef H head;
    typedef T tail;
};
struct null_typelist {};
template <class TList> struct Length;
template <> struct Length<null_typelist>
{
    enum { value = 0 };
};
template <class T, class U>
struct Length< typelist<T, U> >
{
    enum { value = 1 + Length<U>::value };
};
typedef typelist<int,
        typelist<float,
        null_typelist > > testList;
template<int N>
struct IntWrapper {};
void testLength()
{
    ::testing::StaticAssertTypeEq<
        IntWrapper<2>,
        IntWrapper<Length<testList>::value> >();
}

and there is more…

There is an abundance of features to explore (find out more here). Some of my favorites include

  • running tests selectively (using pattern matches on the command line)
  • shuffling the order in which you tests are run (great! Ever wondered if you might have initialization problems? They might show up only if you execute your test cases in a particular order. That happened to me and I was quite astonished to find out about this when I ran shuffled tests…)
  • Type-parameterized tests (let’s you run your templatized code for multiple template instantiations)
  • Predicate Formatters (let’s you tune the generated error messages to make them more meaningful)

As I said, the project is vibrant and it is to be expected that the current features will be kept in good quality while new features will be added.

From Cppunit to gtest

All good news…only that all my tests so far use Cppunit! Thankfully that turned out not to be a major hurdle. Porting tests from Cppunit to google tests is straight forward.
Here an example of a Cppunit-Test:

...
class ClosureTest : public CppUnit::TestFixture
{
public:
    ClosureTestCallee* fpTestCallee;
    void setUp()
    {
        fpTestCallee = new ClosureTestCallee();
    }
    void tearDown()
    {
        delete fpTestCallee;
    }
    void testClosureWithSingleParameter()
    {
        ...
        CPPUNIT_ASSERT_EQUAL(fExpectedCalls, fCallbackCalled);
    }
    ...
    CPPUNIT_TEST_SUITE( ClosureTest );
        CPPUNIT_TEST( testClosureWithSingleParameter );
        ...
    CPPUNIT_TEST_SUITE_END();
};
CPPUNIT_TEST_SUITE_REGISTRATION(ClosureTest);

To get roughly the same tests using google test, only a little structure and some macros have to be changed. A bonus: the arduous work required by Cppunit to declare testsuites and associated tests is no longer necessary.
Every test case is formulated as a macro which turns out to be a function. The two most common test-defining macros are TEST() and TEST_F(). To mimic the structure of the Cppunit test (which rely on a test fixture using the setUp() and tearDown() functions) you can use the TEST_F() macro which will add a fixture.
Of course you will be guaranteed to get a fresh setup for each test so that your tests do not interfere with each other.

...
class ClosureTest : public ::testing::Test
{
public:
    ClosureTestCallee* fpTestCallee;
    virtual void SetUp()
    {
        fpTestCallee = new ClosureTestCallee();
    }
    virtual void TearDown()
    {
        delete fpTestCallee;
    }
};
TEST_F(ClosureTest, closureWithSingleParameter) {
    ...
    EXPECT_EQ(fExpectedCalls, fCallbackCalled);
}
...

And the test output on the command line is quite readable! One glimpse of the output is enough to capture the most important information about the test outcome:

...
[----------] 6 tests from SPriorityQueueTest
[ RUN      ] SPriorityQueueTest.constructor
[       OK ] SPriorityQueueTest.constructor (0 ms)
[ RUN      ] SPriorityQueueTest.size
[       OK ] SPriorityQueueTest.size (0 ms)
[ RUN      ] SPriorityQueueTest.empty
[       OK ] SPriorityQueueTest.empty (0 ms)
[ RUN      ] SPriorityQueueTest.full
[       OK ] SPriorityQueueTest.full (0 ms)
[ RUN      ] SPriorityQueueTest.push
[       OK ] SPriorityQueueTest.push (0 ms)
[ RUN      ] SPriorityQueueTest.pop
[       OK ] SPriorityQueueTest.pop (0 ms)
[----------] 6 tests from SPriorityQueueTest (1 ms total)
[----------] Global test environment tear-down
[==========] 131 tests from 16 test cases ran. (14 ms total)
[  PASSED  ] 131 tests.

In case of a test failure it is also easy to find out about what went wrong. Let’s break the implementation so that some tests fail:

[  FAILED  ] SPriorityQueueTest.push (0 ms)
[ RUN      ] SPriorityQueueTest.pop
test/src/util/SPriorityQueue.cpp:78: Failure
Value of: queue.top()
  Actual: 0
Expected: top[i]
Which is: 3
test/src/util/SPriorityQueue.cpp:78: Failure
Value of: queue.top()
  Actual: 0
Expected: top[i]
Which is: 5
[  FAILED  ] SPriorityQueueTest.pop (0 ms)
[----------] 6 tests from SPriorityQueueTest (2 ms total)
[----------] Global test environment tear-down
[==========] 131 tests from 16 test cases ran. (16 ms total)
[  PASSED  ] 129 tests.
[  FAILED  ] 2 tests, listed below:
[  FAILED  ] SPriorityQueueTest.push
[  FAILED  ] SPriorityQueueTest.pop
 2 FAILED TESTS

Finally there is more fun in writing unit tests for C++ again! Exploring new ways of how to test your code combined with the much easier management of test cases might contribute to getting your unit tests the attention they deserve :)