Short overview of Unit Testing tools in C

The requirements were:

• Easy to integrate into existing code-bases.
• Works on bare-metal systems.
• Can be compiled with standard C compilers.
• Supports mocking functions, including static functions.

Unit Testing frameworks

I ended up investigating the following 5 projects:

• Unity
• CUnit
• Google Test
• Cmockery
• CMocka

Unity

This a nice a project. It is part of the Ceedling build system, but we can use it standalone. The documentation is incredible, especially the information about using the framework on embedded systems.

The only reason we ended up using something else was because some of its features use Ruby (for example test runner generation), which we couldn't use, and we find something with more features.

Tests look a bit like this (source):

void setUp(void)
{
  /* This is run before EACH TEST */
  Counter = 0x5a5a;
}

void tearDown(void)
{
  /* This is run after EACH TEST */
}

void test_FindFunction_WhichIsBroken_ShouldReturnZeroIfItemIsNotInList_WhichWorksEvenInOurBrokenCode(void)
{
  /* All of these should pass */
  TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(78));
  TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(2));
  TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(33));
  TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(999));
  TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(-1));
}

CUnit

This project is quite old and abandoned. Considering the other options available, there isn't much reason to use this. It has an interesting feature where users can add plugins to the framework, but I haven't explored what plugins are out there.

Example test code ([source(https://cunit.sourceforge.net/screenshots.html)):

int init_suite_success(void) { return 0; }
int init_suite_failure(void) { return -1; }
int clean_suite_success(void) { return 0; }
int clean_suite_failure(void) { return -1; }

void test_success(void)
{
   CU_ASSERT(TRUE);
}

int main() {
   CU_pSuite pSuite = NULL;

   /* initialize the CUnit test registry */
   if (CUE_SUCCESS != CU_initialize_registry())
      return CU_get_error();

   /* add a suite to the registry */
   pSuite = CU_add_suite("Suite_success", init_suite_success, clean_suite_success);
   if (NULL == pSuite) {
      CU_cleanup_registry();
      return CU_get_error();
   }

   /* add the tests to the suite */
   if (NULL == CU_add_test(pSuite, "successful_test", test_success)) {
      CU_cleanup_registry();
      return CU_get_error();
   }

   /* Run all tests using the basic interface */
   CU_basic_set_mode(CU_BRM_VERBOSE);
   CU_basic_run_tests();
   printf("\n");
   CU_basic_show_failures(CU_get_failure_list());
   printf("\n\n");

   /* Run all tests using the automated interface */
   CU_automated_run_tests();
   CU_list_tests_to_file();

   /* Clean up registry and return */
   CU_cleanup_registry();
   return CU_get_error();
}

Google Test

Although there are people testing C code using this framework, it is officially a C++ testing framework. With this, we decided to not go with it, because some of our code is related to the Linux Kernel, which can't be compiled with a C++ compiler.

Cmockery

Originally developed by Google, this is the equivalent of Google Test for C. This is superseded by CMocka (see bellow).

CMocka

This is an extremely popular and well tested project. Reportedly used by samba, libssh, coreboot, OpenVPN, and libomemo (so both old and new projects).

In my opinion, the best part of this framework is the integrated support for mocking functions. There's also some memory leak detection capabilities, but nothing like Valgrind. Unfortunately, this isn't a header-only lib, but that's not too hard to deal with.

This is an example of a test using mocks:

#include <limits.h>
#include <setjmp.h>
#include <stdarg.h>
#include <stddef.h>
// IMPORTANT: cmocka needs the imports above
#include <cmocka.h>

#include <stdio.h>

#include "add3.h"

int __wrap_add(int a, int b);

int __wrap_add(int a, int b) {
  printf("------\nHELLO FROM __wrap_add\n------\n");
  check_expected(a);
  check_expected(b);
  return (int)mock();
}

static void test_add3_mock(void **state) {
  // given
  int a = 1, b = 2, c = 4;
  int expected_result = 7;
  int result;

  expect_value(__wrap_add, a, 1);
  expect_value(__wrap_add, b, 2);
  will_return(__wrap_add, 3);
  expect_value(__wrap_add, a, 3);
  expect_value(__wrap_add, b, 4);
  will_return(__wrap_add, 7);

  // when
  result = add3(a, b, c);

  // then
  assert_int_equal(expected_result, result);
}

int main(void) {
  const struct CMUnitTest tests[] = {
      cmocka_unit_test(test_add3_mock),
  };

  return cmocka_run_group_tests(tests, NULL, NULL);
}

Mocks/Stubs

There are also lots of tools for mocking functions in C

I've looked at the following:

• Fake Function Framework (fff)
  • This one is nice and is header-only, so we can add it to our projects really easily. I just found it a bit too verbose, but the documentation is great and full of usage examples.
• CMock
  • Also part of the Ceedling project. Depends on Ruby.
• CMocka
  • Mentioned previously.
• GMock
  • Part of the Google Test suite. C++ only, so we'll ignore it.

None of these libs would report support to magically mock static functions. There are several challenges in doing this, for example, static functions can disappear (be in-lined) during compilation, so there's no symbol to replace. We'll look at this later.

Mocking in CMocka

CMocka's function mocking makes use of the --wrap linker flag. This allows you to replace a function at link time. From ld's manual page:

Use a wrapper function for symbol. Any undefined reference to symbol will be resolved to "__wrap_symbol". Any undefined reference to "__real_symbol" will be resolved to symbol.

This can be used to provide a wrapper for a system function. The wrapper function should be called "__wrap_symbol". If it wishes to call the system function, it should call "__real_symbol".

Note that this might not be available on all linkers.

Mocking static functions

When testing legacy code-bases, it is frequent to find that the code wasn't written in a way favorable for testing. For this reason, we used a fun way to mock static functions:

• Build a mock function (replacement function).
• During testing, re-write the first instruction of the target function to a jump/branch to the replacement function.
• Now everything that would call the original function, would call your replacement function instead.

This process is easier than it seems. Bellow you have an example to do this in ARM (source):

#include <stdint.h>
#include <sys/mman.h>

int hotpatch(void *target, void *replacement)
{
    void *page = (void *)((uintptr_t)target & ~0xfff);
    uint32_t offset = (char *)replacement - (char *)target;
    uint32_t instruction = (0x14 << 24) | ((offset >> 2) & 0x3ffffff);

    if (mprotect(page, 4096, PROT_WRITE | PROT_EXEC)) {
        return -2;
    }

    *(uint64_t *)target = instruction;

    return 0;
}

This code is quite simple. We're assuming that the system's page size is 4096 bytes and that the branching offset is within 32 MB. Still, we're just calculating the offset from the first instruction of the target function to the replacement function and writing an unconditional branch instruction (0x14) with the given offset.

Coverage

Generating coverage reports is easily done with GCovr independently of the tool used. Just compile your code with the following flags with you want to generate test reports: -g -O0 --coverage.

You can get reports on your cli or as HTML reports:

------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: ..
------------------------------------------------------------------------------
File                                       Lines    Exec  Cover   Missing
------------------------------------------------------------------------------
src/add3.c                                     3       3   100%
src/add.c                                      2       2   100%
src/program.c                                  4       0     0%   5-7,9
------------------------------------------------------------------------------
TOTAL                                          9       5    55%
------------------------------------------------------------------------------
lines: 55.6% (5 out of 9)
functions: 75.0% (3 out of 4)
branches: 0.0% (0 out of 0)

Conclusion

We've ended up going with CMocka. It has a decent enough documentation, all the features we need, and is popular. I've published a template project on GitHub to quick start a C project with CMake:

• Builds with CMake.
• Auto downloads and builds CMocka.
• Includes example code and tests.
• Has the funny function to mock static functions for ARM and x86 (including examples).
• Downloads and runs CppCheck and Clang-Tidy on your project.