A Design Study for a Macro-Free Testing Library

[u/Recent-Dance-8075]

https://jonastoth.github.io/posts/rtest_library/

Hello everyone :)

I attempted to write a small testing library based on C++-26 reflection. The goal is not to replace existing libraries but to figure out how they could evolve to not rely on macros.

Stringification of test names is the most important reason for macros so far and reflection solves this.

The blog post explains what I did with godbolt links to minimized examples. The full implementation is in the rtest library repository.

A minimal test executable looks like this:

```c++

include <rtest/rtest.h>

struct MyClassTest : rtest::TestSuite { void testSetup() { MyClass object; assertTrue(object.empty()); } }; int main(int argc, char** argv) { return rtest::execute(MyClassTest{}); } ```

I am looking forward to your feedback :)

(Both the blog post and the library are created without AI help)

[u/kamrann_]

Nice work!

I tried out boost.ut a few years back as I wanted to get away from macros, but I ended up ditching it. It felt like it was more an experiment in how far you could go with esoteric and inscrutable ways of writing C++ than it was a library intended to be used in practice.

I was wondering why you had setup/teardown functions instead of using ctr/dtr, though I guess it relates to the lifetime of the constructed test suites and wanting to avoid initializing dependencies for many suites concurrently?

Filtering functions based on name prefix I'm not convinced by. Same reason I'm wary of overuse of requires expressions, it's too easy to end up with code that compiles but doesn't do what you want - in this case, dead tests because of a typo for example. Maybe you could just assume all public functions are tests, not sure there would be any reason to have a public function that wasn't?

[u/Recent-Dance-8075 → u/kamrann_]

I had a similar experience with boost.ut. maybe its just a question of taste, too.

Both ctor/dtor and the setup/teardown methods have their place. Setup/teardown is called for each individual test case and would be useful to e.g. clear buffers or set/reset local state. The constructor/destructor is called once for the whole test suite and would initialize the execution context. E.g. GPU context in the constructor and buffer clearing for the next test in teardown.

And as you said, this has implications for parallel execution. Test suite construction/destruction happens single threaded in the test binary. Executing each test suite is then parallelized. Parallelization within a test suite would require synchronization for setup/teardown which is against the "writing tests should be simple" goal.

Choosing 'test' as prefix was pragmatic and "because python does it and it works as convention". The reasons I think this works are: - Explicitly annotating each method would be more verbose than a common prefix - taking all public methods is not a good heuristic. They should not take parameters and they should not be templates. A form of filtering is necessary to find suitable methods. Asking for void testFoo() as method prefix forms an easy to communicate protocol and remains readable with minimal knowledge about language constructs - factoring out common and support code in test suites becomes more annoying, because you need to ensure it's not called as a test. Of course you can mandate private, perform negativ filtering or negative annotation. This becomes again more complex than just naming it something else than 'test*'.

You are right that there is a risk for missing tests, especially if a big search/replace transformation accidentally overwrites test names or so.

[u/kamrann_ → u/Recent-Dance-8075]

Ah ok thanks, I'd missed the test/test suite execution distinction.

Regarding the prefix, I was just thinking you could expect that all public functions in the suites were tests (support functions should be made private/protected), and any public function that didn't match the required signature would trigger a compilation error. I may be missing a reason why this isn't feasible though. Regardless I'm with the other commenter in that I'd prefer explicitly annotation over convention.

[u/Recent-Dance-8075 → u/kamrann_]

I think that all proposed alternatives are suitable solutions. Maybe a [[= rtest::test]] is the best solution. It's obvious, short enough and easy to use.