┌─ root@joaocosta:~# cat posts/12.md ───────────────────────────────────────────

# Short overview of Unit Testing tools in C
published 2024-10-31 · 5 min read · https://joaocosta.dev/blog/12

Recently I've been tasked with choosing a Unit Testing framework for a new
project at work. Having only used proprietary test harnesses in C before, this
was an opportunity to see what's out there. TL;DR we ended up going with CMocka
(https://cmocka.org/).

The requirements were:

→ Easy to integrate into existing code-bases.
→ Works on bare-metal systems.
→ Can be compiled with standard C compilers.
→ Supports mocking functions, including static functions.

## Unit Testing frameworks

I ended up investigating the following 5 projects:

→ Unity (https://github.com/ThrowTheSwitch/Unity)
→ CUnit (https://cunit.sourceforge.net/)
→ Google Test (https://github.com/google/googletest)
→ Cmockery (https://github.com/google/cmockery)
→ CMocka (https://cmocka.org/)

### Unity

This a nice a project. It is part of the Ceedling
(https://www.throwtheswitch.org/ceedling) build system, but we can use it
standalone. The documentation is incredible, especially the information about
using the framework on embedded systems
(https://github.com/ThrowTheSwitch/Unity/blob/master/docs/UnityConfigurationGuide.md).

The only reason we ended up using something else was because some of its
features use Ruby (for example test runner generation), which we couldn't use,
and we find something with more features.

Tests look a bit like this (source
(https://github.com/ThrowTheSwitch/Unity/tree/master/examples/example_1)):

┄─────────────────────────────────────────────────────────────────────────── c 
│ void setUp(void)
│ {
│   /* This is run before EACH TEST */
│   Counter = 0x5a5a;
│ }
│ 
│ void tearDown(void)
│ {
│   /* This is run after EACH TEST */
│ }
│ 
│ void test_FindFunction_WhichIsBroken_ShouldReturnZeroIfItemIsNotInList_WhichWorksEvenInOurBrokenCode(void)
│ {
│   /* All of these should pass */
│   TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(78));
│   TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(2));
│   TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(33));
│   TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(999));
│   TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(-1));
│ }
┄──────────────────────────────────────────────────────────────────────────────

### CUnit

This project is quite old and abandoned. Considering the other options
available, there isn't much reason to use this. It has an interesting feature
where users can add plugins to the framework, but I haven't explored what
plugins are out there.

Example test code ([source(https://cunit.sourceforge.net/screenshots.html)):

┄─────────────────────────────────────────────────────────────────────────── c 
│ int init_suite_success(void) { return 0; }
│ int init_suite_failure(void) { return -1; }
│ int clean_suite_success(void) { return 0; }
│ int clean_suite_failure(void) { return -1; }
│ 
│ void test_success(void)
│ {
│    CU_ASSERT(TRUE);
│ }
│ 
│ int main() {
│    CU_pSuite pSuite = NULL;
│ 
│    /* initialize the CUnit test registry */
│    if (CUE_SUCCESS != CU_initialize_registry())
│       return CU_get_error();
│ 
│    /* add a suite to the registry */
│    pSuite = CU_add_suite("Suite_success", init_suite_success, clean_suite_success);
│    if (NULL == pSuite) {
│       CU_cleanup_registry();
│       return CU_get_error();
│    }
│ 
│    /* add the tests to the suite */
│    if (NULL == CU_add_test(pSuite, "successful_test", test_success)) {
│       CU_cleanup_registry();
│       return CU_get_error();
│    }
│ 
│    /* Run all tests using the basic interface */
│    CU_basic_set_mode(CU_BRM_VERBOSE);
│    CU_basic_run_tests();
│    printf("\n");
│    CU_basic_show_failures(CU_get_failure_list());
│    printf("\n\n");
│ 
│    /* Run all tests using the automated interface */
│    CU_automated_run_tests();
│    CU_list_tests_to_file();
│ 
│    /* Clean up registry and return */
│    CU_cleanup_registry();
│    return CU_get_error();
│ }
┄──────────────────────────────────────────────────────────────────────────────

### Google Test

Although there are people testing C code using this framework, it is officially
a C++ testing framework. With this, we decided to not go with it, because some
of our code is related to the Linux Kernel, which can't be compiled with a C++
compiler.

### Cmockery

Originally developed by Google, this is the equivalent of Google Test for C.
This is superseded by CMocka (see bellow).

### CMocka

This is an extremely popular and well tested project. Reportedly used by samba,
libssh, coreboot, OpenVPN, and libomemo (so both old and new projects).

In my opinion, the best part of this framework is the integrated support for
mocking functions. There's also some memory leak detection capabilities, but
nothing like Valgrind (https://valgrind.org/). Unfortunately, this isn't a
header-only lib, but that's not too hard to deal with.

This is an example of a test using mocks:

┄─────────────────────────────────────────────────────────────────────────── c 
│ #include <limits.h>
│ #include <setjmp.h>
│ #include <stdarg.h>
│ #include <stddef.h>
│ // IMPORTANT: cmocka needs the imports above
│ #include <cmocka.h>
│ 
│ #include <stdio.h>
│ 
│ #include "add3.h"
│ 
│ int __wrap_add(int a, int b);
│ 
│ int __wrap_add(int a, int b) {
│   printf("------\nHELLO FROM __wrap_add\n------\n");
│   check_expected(a);
│   check_expected(b);
│   return (int)mock();
│ }
│ 
│ static void test_add3_mock(void **state) {
│   // given
│   int a = 1, b = 2, c = 4;
│   int expected_result = 7;
│   int result;
│ 
│   expect_value(__wrap_add, a, 1);
│   expect_value(__wrap_add, b, 2);
│   will_return(__wrap_add, 3);
│   expect_value(__wrap_add, a, 3);
│   expect_value(__wrap_add, b, 4);
│   will_return(__wrap_add, 7);
│ 
│   // when
│   result = add3(a, b, c);
│ 
│   // then
│   assert_int_equal(expected_result, result);
│ }
│ 
│ int main(void) {
│   const struct CMUnitTest tests[] = {
│       cmocka_unit_test(test_add3_mock),
│   };
│ 
│   return cmocka_run_group_tests(tests, NULL, NULL);
│ }
┄──────────────────────────────────────────────────────────────────────────────

## Mocks/Stubs

There are also lots of tools for mocking functions in C

I've looked at the following:

→ Fake Function Framework (fff) (https://github.com/meekrosoft/fff)
- This one is nice and is header-only, so we can add it to our projects really
easily. I just found it a bit too verbose, but the documentation is great and
full of usage examples.

→ CMock (https://github.com/ThrowTheSwitch/CMock)
- Also part of the Ceedling (https://www.throwtheswitch.org/ceedling) project.
Depends on Ruby.

→ CMocka (https://cmocka.org/)
- Mentioned previously.

→ GMock (https://google.github.io/googletest/gmock_for_dummies.html)
- Part of the Google Test suite. C++ only, so we'll ignore it.

None of these libs would report support to magically mock static functions.
There are several challenges in doing this, for example, static functions can
disappear (be in-lined) during compilation, so there's no symbol to replace.
We'll look at this later.

### Mocking in CMocka

CMocka's function mocking makes use of the --wrap linker flag. This allows you
to replace a function at link time. From ld's manual page:

┃ Use a wrapper function for symbol. Any undefined reference to symbol will be
  resolved to "__wrap_symbol". Any undefined reference to "__real_symbol" will
  be resolved to symbol. This can be used to provide a wrapper for a system
  function. The wrapper function should be called "__wrap_symbol". If it wishes
  to call the system function, it should call "__real_symbol".

Note that this might not be available on all linkers.

### Mocking static functions

When testing legacy code-bases, it is frequent to find that the code wasn't
written in a way favorable for testing. For this reason, we used a fun way to
mock static functions:

→ Build a mock function (replacement function).
→ During testing, re-write the first instruction of the target function to a
  jump/branch to the replacement function.
→ Now everything that would call the original function, would call your
  replacement function instead.

This process is easier than it seems. Bellow you have an example to do this in
ARM (source
(https://github.com/JoaoCostaIFG/cmake-cmocka-template/blob/main/test/utils.c)):

┄─────────────────────────────────────────────────────────────────────────── c 
│ #include <stdint.h>
│ #include <sys/mman.h>
│ 
│ int hotpatch(void *target, void *replacement)
│ {
│ 	void *page = (void *)((uintptr_t)target & ~0xfff);
│ 	uint32_t offset = (char *)replacement - (char *)target;
│ 	uint32_t instruction = (0x14 << 24) | ((offset >> 2) & 0x3ffffff);
│ 
│ 	if (mprotect(page, 4096, PROT_WRITE | PROT_EXEC)) {
│ 		return -2;
│ 	}
│ 
│ 	*(uint64_t *)target = instruction;
│ 
│ 	return 0;
│ }
┄──────────────────────────────────────────────────────────────────────────────

This code is quite simple. We're assuming that the system's page size is 4096
bytes and that the branching offset is within 32 MB. Still, we're just
calculating the offset from the first instruction of the target function to the
replacement function and writing an unconditional branch instruction
(https://developer.arm.com/documentation/dui0489/i/arm-and-thumb-instructions/b?lang=en)
(0x14) with the given offset.

## Coverage

Generating coverage reports is easily done with GCovr
(https://gcovr.com/en/stable/) independently of the tool used. Just compile your
code with the following flags with you want to generate test reports: -g -O0
--coverage.

You can get reports on your cli or as HTML reports:

┄───────────────────────────────────────────────────────────────────────── txt 
│ ------------------------------------------------------------------------------
│                            GCC Code Coverage Report
│ Directory: ..
│ ------------------------------------------------------------------------------
│ File                                       Lines    Exec  Cover   Missing
│ ------------------------------------------------------------------------------
│ src/add3.c                                     3       3   100%
│ src/add.c                                      2       2   100%
│ src/program.c                                  4       0     0%   5-7,9
│ ------------------------------------------------------------------------------
│ TOTAL                                          9       5    55%
│ ------------------------------------------------------------------------------
│ lines: 55.6% (5 out of 9)
│ functions: 75.0% (3 out of 4)
│ branches: 0.0% (0 out of 0)
┄──────────────────────────────────────────────────────────────────────────────

## Conclusion

We've ended up going with CMocka. It has a decent enough documentation, all the
features we need, and is popular. I've published a template project on GitHub
(https://github.com/JoaoCostaIFG/cmake-cmocka-template/tree/main) to quick start
a C project with CMake:

→ Builds with CMake (https://cmake.org/).
→ Auto downloads and builds CMocka.
→ Includes example code and tests.
→ Has the funny function to mock static functions for ARM and x86 (including
  examples).
→ Downloads and runs CppCheck (https://cppcheck.sourceforge.io/) and Clang-Tidy
  (https://clang.llvm.org/extra/clang-tidy/) on your project.

└─ → view online: https://joaocosta.dev/blog/12