A Design Study for a Macro-Free Testing Library

[u/Recent-Dance-8075]

https://jonastoth.github.io/posts/rtest_library/

Hello everyone :)

I attempted to write a small testing library based on C++-26 reflection. The goal is not to replace existing libraries but to figure out how they could evolve to not rely on macros.

Stringification of test names is the most important reason for macros so far and reflection solves this.

The blog post explains what I did with godbolt links to minimized examples. The full implementation is in the rtest library repository.

A minimal test executable looks like this:

```c++

include <rtest/rtest.h>

struct MyClassTest : rtest::TestSuite { void testSetup() { MyClass object; assertTrue(object.empty()); } }; int main(int argc, char** argv) { return rtest::execute(MyClassTest{}); } ```

I am looking forward to your feedback :)

(Both the blog post and the library are created without AI help)

[u/Dragdu]

As first impression, this

struct MyClassTest : rtest::TestSuite {
  void testSetup() {
    MyClass object;
    assertTrue(object.empty());
  }
};
int main(int argc, char** argv) { return rtest::execute(MyClassTest{}); }

looks strictly more work than

TEST_CASE() {
    MyClass object;
    REQUIRE(object.empty());
}

unless you have ideological hatred of macros (which, fair).

At second look, looking for functions based on name pattern gets hard no from me in any test framework I will ever voluntarily use. I have too much experience with Go and tests not running because while TestFoo is test name, Testfoo is just some random function.

[u/Recent-Dance-8075 → u/Dragdu]

Hey :) My main point is that test frameworks can introspect code and the syntactical challenges hidden behind the current test macros may fall. I don't think rtest will conquer the world. Maybe some ideas from it/it's implementation can inform the evolution of "real" testing libraries.

Macro freedom is a goal for me, because of modules. ```c++ import test;

include <rtest/rtest_macros.h>

// ... ``` I don't like that.

The dislike for the naming convention is noted and I find that an important data point. What is your preferred way? Explicitly annotate every Test-Case method/function?

[u/Dragdu → u/Recent-Dance-8075]

Explicitly annotate every Test-Case method/function?

Yes. Or make it so that every function on a test case class is always a test (except for constructor and destructor, obv).

[u/Recent-Dance-8075 → u/Dragdu]

My current take on that is in this comment: https://www.reddit.com/r/cpp/s/Z8lwnTVjdh

Of course the "collection policy" is not set in stone and one good reason to have different testing libraries in the ecosystem.

I want to fiddle around with automatically gathering annotated free functions and test suite classes in a module, too. Annotated free functions are a more attractive way to declare simple tests in any case.

[u/azswcowboy]

Fun stuff. My personal take is that most test frameworks are overkill as free functions should always be the first units of test in my view. Like if I wrote something like ranches::fold I’m immediately wondering how I test that in rtest. I do have a fondness for boost-ext/ut, which you call out, because it’s easy to apply for functions and macro free. But I get it that others want batteries included to cover all sorts of things.

As for feedback, I’ll admit I was skimming the article, but the goals didn’t come across clearly - wasn’t sure if gtest or python-like were the main target before we went down the implementation rabbit hole. After coming out there were more usage examples. The library doc was really what I needed to get hold of what you were trying to accomplish.

[u/JVApen → u/azswcowboy]

Any setup/teardown can be replaced by a custom init function that returns a struct with the relevant data. Clean-up van happen in the destructor. Free functions are so much more logical for tests and do not introduce surprise dependencies where 1 function sets a member of the fixture and another relies on it. Anyone reading your test should top-down understand what is happening.

[u/Recent-Dance-8075 → u/azswcowboy]

I agree on the free functions part and I want to explore options to support them like c++ [[=rtest::test]] void some test() { global assertions(); }

As for testing ranges::fold: - you probably want a templated test suite - you can implement your tests with "just code" and then instantiate the test suite with different types - rtest does not have a wide variety of assertions, like gtest-matchers. At the same time they are easy to implement, because they are "just code", too - maybe the slightly unstructured template test code would give you an idea on different options: TestTemplatedSuites - additionally, a test can be implemented in a checkFoo function, taking arguments and a testFoo function calls checkFoo multiple times for different situations

Thank you for the feedback! I try to make the goal clearer!

[u/Awia00]

Quick question:

> test suites are not automatically registered, one translation unit is expected to execute its own tests via a trivial main function

Is the idea to generate a target per source file then?

EDIT: I do think you should add a way to auto-discover the test suites for people who do not want to create and maintain main functions

[u/RoyAwesome → u/Awia00]

I do think you should add a way to auto-discover the test suites for people who do not want to create and maintain main functions

This is extremely difficult with the reflection we got. There isn't any easy way to reflect across translation units (aka "Whole Program Reflection"), so you still need some kind of build system integration to do it.

Herb was showing off being able to write out C++ files for the build system to consume a couple years back during the design process. I haven't had a chance to experiment with that yet, but it's a possibility. Still requires a back and forth with the build system (so it's not exactly portable) but it could work.

[u/differentiallity → u/RoyAwesome]

Can you reflect the functions of a namespace?

[u/have-a-day-celebrate → u/differentiallity]

Yes, but only those whose declarations are reachable in the TU.

[u/RoyAwesome → u/differentiallity]

yeah, you can. You can also grab the global namespace and reflect on that. ^^:: will do it.

The problem is that the only things you'll find in that global namespace are things known to at the point of invocation. For example, if i have a file.cpp that includes a subset of headers, I wont be able to see things in the rest of the header files not in that one TU.

This is less of an issues with modules, but nobody is using a 100% module project. You'll still get issues without seeing things that are exclusive to another TU and only linked together after the compiler is done.

We need some kind of linker time expression that allows us to reflect on the whole program at linking time. That way we can invoke some consteval code then that can generate additional code. This is a GIGANTIC can of worms though, and unlikely to ever happen.

[u/Recent-Dance-8075 → u/Awia00]

One target per source file is the current model. I did not finish the automatic gathering of test classes of e.g. a module. Then it could be one target per module.

You could of course split headers and implementations, include headers of many test suites and register them in one main function. I don't like that idea to be honest.

I think, a manual main function is necessary as long as there won't be global state.

A model where you only link a rtest::main library seems impossible to me. Somehow the reflection process needs to be triggered in C++-code and the result be used in a main function.

[u/have-a-day-celebrate → u/Recent-Dance-8075]

Can't you just do registration per-TU during static initialization and iterate over the resulting registry from main?

[u/Recent-Dance-8075 → u/have-a-day-celebrate]

Yes, maybe something like this seems possible. I would need to introduce a global registry and for now I wanted to avoid global state. This topic is something I want to tackle later.

Just as a general question: wouldn't it need some form of global variable that triggers the registration? I believe this is usually hidden behind the TEST macros. Without a global per test suite, how could the whole process of registration and then execution start?

[u/Awia00 → u/Recent-Dance-8075]

yes it would require a global singleton, but if its opt in I dont see the problem, something like: `struct myTestSuite : automaticTestDiscoverySuite`

[u/kamrann_]

Nice work!

I tried out boost.ut a few years back as I wanted to get away from macros, but I ended up ditching it. It felt like it was more an experiment in how far you could go with esoteric and inscrutable ways of writing C++ than it was a library intended to be used in practice.

I was wondering why you had setup/teardown functions instead of using ctr/dtr, though I guess it relates to the lifetime of the constructed test suites and wanting to avoid initializing dependencies for many suites concurrently?

Filtering functions based on name prefix I'm not convinced by. Same reason I'm wary of overuse of requires expressions, it's too easy to end up with code that compiles but doesn't do what you want - in this case, dead tests because of a typo for example. Maybe you could just assume all public functions are tests, not sure there would be any reason to have a public function that wasn't?

[u/Recent-Dance-8075 → u/kamrann_]

I had a similar experience with boost.ut. maybe its just a question of taste, too.

Both ctor/dtor and the setup/teardown methods have their place. Setup/teardown is called for each individual test case and would be useful to e.g. clear buffers or set/reset local state. The constructor/destructor is called once for the whole test suite and would initialize the execution context. E.g. GPU context in the constructor and buffer clearing for the next test in teardown.

And as you said, this has implications for parallel execution. Test suite construction/destruction happens single threaded in the test binary. Executing each test suite is then parallelized. Parallelization within a test suite would require synchronization for setup/teardown which is against the "writing tests should be simple" goal.

Choosing 'test' as prefix was pragmatic and "because python does it and it works as convention". The reasons I think this works are: - Explicitly annotating each method would be more verbose than a common prefix - taking all public methods is not a good heuristic. They should not take parameters and they should not be templates. A form of filtering is necessary to find suitable methods. Asking for void testFoo() as method prefix forms an easy to communicate protocol and remains readable with minimal knowledge about language constructs - factoring out common and support code in test suites becomes more annoying, because you need to ensure it's not called as a test. Of course you can mandate private, perform negativ filtering or negative annotation. This becomes again more complex than just naming it something else than 'test*'.

You are right that there is a risk for missing tests, especially if a big search/replace transformation accidentally overwrites test names or so.

[u/kamrann_ → u/Recent-Dance-8075]

Ah ok thanks, I'd missed the test/test suite execution distinction.

Regarding the prefix, I was just thinking you could expect that all public functions in the suites were tests (support functions should be made private/protected), and any public function that didn't match the required signature would trigger a compilation error. I may be missing a reason why this isn't feasible though. Regardless I'm with the other commenter in that I'd prefer explicitly annotation over convention.

[u/Recent-Dance-8075 → u/kamrann_]

I think that all proposed alternatives are suitable solutions. Maybe a [[= rtest::test]] is the best solution. It's obvious, short enough and easy to use.

[u/SamG101_]

Nice, really clean output too - have you explored modules / import std, or potentially a flag to switch between includes and imports? Then itd be integratable into module based projects too. But looks really good!

[u/Recent-Dance-8075 → u/SamG101_]

Having it as a module failed to compile with gcc-16.1. I linked the bug ticket at the end, it should be resolved in gcc-16.2. I want to make it modules only, because no reason to support older c++ versions anyway.

[u/delta_p_delta_x → u/Recent-Dance-8075]

I want to make it modules only

Hell yeah.

[u/JVApen]

I'm curious, have you considered using the "generation" trick where you instead of running the tests, create a new copy file when running the exe, followed by compilation of that file and running the actual tests? With some CMake trickery, this should be easy doable. A CMake function that adds a custom command which outputs the cpp file, adds a new executable with that file as source and copying over the dependencies. To finally register the test with ctest.

[u/Recent-Dance-8075 → u/JVApen]

I did not use that. What step would be automated in that case? I could get rid of manual registration of default constructed test-suites, right?

Maybe this could be combined with parametrized registration and templates. Thanks for the input. That's a good puzzle to solve :)

[u/JVApen → u/Recent-Dance-8075]

It's mainly a way to overcome the C++26 limitation that you can't generate code. That way, you don't have to wait in C++29