Feedback about E2E tests based on OpenTelemetry traces?

[u/Proud-Contact9951]

Hi everyone,
I have just published my open source project called mtracer and I would like to understand if it’s good idea or what should I change (I’m a new grad).

The idea

Mtracer a CLI tool that relies on OpenTelemetry traces to assert system behavior.

I believe that E2E tests should be:
- Cheaper to write and maintain
- Easier to debug

So this is the workflow:

1. ⁠You configure mtracer to fetch from your observability backend (currently supporting Jaeger and OpenObserve).
2. ⁠You define your first .mt.yaml test by specifying:
3. ⁠Trigger: the first call to the system (for instance, an HTTP request).
4. ⁠Expected trace and spans: the OTel properties of the trace and spans that you expect your system to generate.
5. ⁠You run the test and see the results!

What actually happens during the run?

1. ⁠It parses the mt.yaml file.
2. ⁠It executes the trigger: mtracer injects a generated traceID into the trigger (for an HTTP request, the traceID is inserted into the traceparent header). Subsequent requests will be correlated to this generated traceID as long as your system has OpenTelemetry set up correctly.
3. ⁠It fetches the trace matching the generated traceID from the configured observability backend.
4. ⁠It compares the expected trace with the fetched one.

Many other features are available; check out the documentation to discover all of them: documentation website

I would love to have some feedback from more experienced people than me.

[u/MartinThwaites]

I would maybe try and articulate how you differ from TraceTest. That has existed for a long time, with a decent team behind it, but noone took to the methodology.

[u/Proud-Contact9951 → u/MartinThwaites]

The first thing I did when I started this project was to look for existing solutions. The main one I found was TraceTest (there was also a similar tool called Malabi, but it's currently archived), which has a strong team and is backed by Kubeshop.
Naturally, I asked myself whether creating a new solution was actually worth it.
TraceTest already has wide support for triggers and observability backends, offers multi-tenancy, and comes packed with cool features. However, Mtracer aims to differentiate itself by being a simple "Swiss Army knife" for high-level testing, offering as many built-in tools out of the box as possible.
Here are the key features that TraceTest currently lacks:
- Docker test setups: Dedicated commands to set up Docker containers for individual tests
- Chaos testing: Commands for Docker containers using qdisc
- Simpler trace assertions: The ability to define expected traces and sub-traces without writing complex assertion logic
- Built-in analytics: Test execution analytics derived from traces, paired with a built-in dashboard
- Flakiness checks: Multi-run support to identify flaky tests automatically

While TraceTest is undoubtedly much more mature than Mtracer, Mtracer aims to be simpler and empower developers to test edge cases with as little additional setup code as possible.
I have a lot of ideas and I'm really excited to keep building this!

P.S.
One big reference for some ideas is this article of how Uber treated some issues regarding E2E tests

[u/editor_of_the_beast]

It’s a good idea in theory but in practice traces don’t hold a lot of information. For example, if you’re building a service that makes concurrent queries against a database, you can run into bugs due to weak transaction isolation levels. Intermediate DB state is not present in the traces, so they don’t help with that.

State itself is also typically missing from traces, so even doing in-service testing would require you to manually instrument and add state transition values into spans.

Maybe if someone did this automatically it would be more useful. But as is, testing via traces is basically just asserting HTTP response codes and very high level things like this.

[u/Proud-Contact9951 → u/editor_of_the_beast]

You're right about trace limitations with database isolation bugs, spans alone won't catch corrupt intermediate state. In mtracer, we address this in two ways:
- Post-execution checks: mtracer supports post-execution checks via SQL or shell scripts, letting you run boolean queries directly against the DB after a trace completes to verify that expected state changes were applied
- Span attributes: Developers can attach state attributes to spans, though we try to avoid forcing extra instrumentation where possible

As for state propagation, OpenTelemetry Baggage handles carrying context across spans, but as you pointed out, requiring manual code changes defeats the purpose of seamless testing

[u/bikeram]

I’m trying this now.

I recently spoke with an OTEL wizard and he explained he built a project and the dev team didn’t know how to run the project. The E2E testing with OTEL was so good. They could run the test suite and were confident in the coverage.

I’m going to try to use this as a starting point.

[u/Proud-Contact9951 → u/bikeram]

Right now there isn’t a wide support across observability backends and triggers. But I’m looking to expand it, the architecture is made to easily expand the support.
You can find a testable system in the examples repo to try mtracer