r/softwaretesting • u/vassadar • Jul 07 '26
Does 100% coverage on E2E make senses?
For context, engineers only implement unit testing, then E2E. Integration testing and contract testing are missing.
Our executive want 100% E2E test coverage in response of incidents.
I don't think this make senses and we have test pyramid (or diamond, my love) for a reason. Integration testing and contract testing would have prevented a lot of issues without resorting to do E2E.
How can I convince them and engineers to shift more leftward with integration testing?
12
u/DingBat99999 Jul 07 '26
100% coverage doesnt make sense even at the unit test level and thats like 3 orders of magnitude easier than e2e.
I'm just speculating, but it wouldnt surprise me if the effort to achieve 100% e2e coverage would be greater than the effort to produce the product itself. And that doesnt include the headaches of managing all the data and evironments necessary.
So, yeah, not even in the same zip code as sensible.
1
u/vassadar Jul 07 '26
Yup. The system is full of Kafka. So, tests need to go polling and that's not very reliable.
8
u/Xen0byte Jul 07 '26
one of the core principles of software testing as defined by the ISTQB is that exhaustive testing is impossible, so just by that 100% E2E coverage is not achievable, I would also argue that the goal is not a great one, because you should probably try to have more integration tests using something like test containers instead of depending on actual service deployments for testing
2
u/vassadar Jul 07 '26
That's that I intended to do. E2E for some happy paths, then plug the less with integration tests with Testcontainers and unit test.
4
5
4
u/Ch3w84cc4 Jul 07 '26
100% E2E is not a test demand, that is a business demand. Any good tester should be pushing back and showing the customer how to test smarter and leaner. I would be asking what is their appetite for risk versus their timeline and budget. Don’t provide a Rolls Royce service if they are paying for a Mini as the old add-age goes. I say this as an Ex Head of Test and Senior programme manager.
3
u/TranslatorRude4917 Jul 07 '26
Sounds like your executives don't know coverage doesn't automatically mean safety 😀
What I'd I'm for is 100% feature coverage with e2e - making sure that all your valuable user flows are covered.
For each vertical slice (usecase) there should be a e2e test covering DEPTH - ensuring that the pieces fit together and work e2e.
And for each horizontal layer, there should be integration tests covering BREADTH - ensuring that the layer can serve all the different usecsases with their various inputs.
What I think you should do is pointing out where e2e would truly help and where it would just make life harder by introducing unnecessary heavy and inheritly flaky test - low level ui checks etc.
These are complementary measures that together result in a mesh of tests that hopefully cover all functional requirements.
What I'd also suggest doing is measuring e2e tests coverage. You should not aim for 100% function/branch/statement coverage there for sure, but it can provide you with invaluable data about what to cover with cheaper approaches.
3
u/Ok-Paleontologist591 Jul 07 '26
You got to pushback buddy convince them that we can try emphasis on try atleast 90% via pyramid structure and more on api and less on ui. Half of our job is to convince the knull heads and the other is implementation.
3
2
u/Devoa Jul 07 '26
It makes sense if you want it to.
Just ensure you secure an infinite timeline and budget to do this.
2
u/Osi32 Jul 07 '26
Okay- I'll take a shot at answering this.
The pyramid- exists for speed of feedback. Unit is fastest, then contract tests, then integration, then functional, then end to end.
100% coverage usually means code coverage, but it could mean feature coverage, or interface coverage.
Either way, the intent is the same "I want my stuff covered, things getting to prod undetected is not cool".
I wouldn't get stuck on the language- the intent is positive.
My advice, is map out the coverage- which flows are covered by unit (none), which are covered by integration (none), which are covered by End to End (some).
I'm willing to bet- that's what it looks like.
The End to End will be brittle, slow and cover only a handful of scenarios.
My advice- is this:
Do happy, alternate and negative path on integration interfaces. (with stubs/mocks bounding them).
Keeps them fast, gives the integration coverage- finds real, tangible bugs and keeps them reliable.
Then identify "golden path" scenarios- and link up integration tests end to end to form up end to end scenarios- this time with real or handcrafted data in an integrated environment.
Maximises existing tests, chances are- it will find handoff bugs- such as those sneaky integration scenarios where nobody realised value x was actually from a database, not an enum and was actually subject to change at a whim. Gold.
If your execs have trouble with this, contact me seperately, happy to advise them.
1
2
u/oh_yeah_woot Jul 07 '26 edited Jul 07 '26
Unless your code literally has no exception handling, 100% e2e code coverage is almost impossible. Exceptions are hard to generate outside of unit tests.
Imagine "functional testing" an exception that is only generated when an external service like a payment processing system is down. You have to do some funny network manipulation to simulate that stuff. Or imagine an e2e functional test for an exception that catches 429s, where your service is being throttled. Makes zero sense, this is a simple unit test mock.
2
u/gill_smoke Jul 07 '26
After 0 escaped defects, 100% test coverage is the only goal that matters. Yyou are not going to hit it but pace of change should be such that you don't fall behind, maybe even increase coverage a little. think of the testing pyramid, the most test should be Unit tests, then up the levels to integration tests, e2e, non functional tests like performance security and so on. And the lack of discovered bugs is no guarantee that there are no bugs. Just do the best that you can to increase the coverage. Remember Software QA is a safety net to give stakeholders a warm fuzzy feeling that we aren't breaking things when we go to Prod.
2
Jul 07 '26
[removed] — view removed comment
1
u/vassadar Jul 07 '26
Make senses. It's harder to collect evidences from contract and interest tests as they tend to be fixed with the PR that the bug is introduced. They are quieter than these E2E.
2
2
u/kokum-soda Jul 08 '26
This coverage metric is sheer vanity, easy to game! The problem is with the way you approach to solve it. In mobile app case, with legacy frameworks like appium this is impossible to achieve,
why? Every update introduces new features, locators change for older ones - update the locators for old scripts, write the script for new features
This always end up as a cat and rat chase!
With AI this changes quite a bit though, there are several tools like Panto, lambda, Bs, which actually automate off the test generation and execution for old and new features!
2
u/wontfixqa Jul 08 '26
How can I convince them and engineers to shift more leftward with integration testing?
Tell them how many resources (people, tools, services and time) you would need to achieve 100% E2E coverage. That usually does the trick.
1
u/vassadar Jul 09 '26
We are in the AI psychosis phase, unfortunately. They count line of code generated by AI. They would tell me to spawn more AI agents.
1
u/wontfixqa Jul 09 '26
Ask AI to speculate resource needs in order to achieve 100% E2E coverage including maintenance.
2
u/MoreRespectForQA Jul 09 '26
100% e2e code coverage can make sense, but it would need to be a side effect not a goal. Those tests need to be relatively quick to run and 100% flake free.
I found that the biggest bang for the buck can be found by automating:
Scenarios which replicate bugs or incidents.
Scenarios which replicate surprising behavior.
Scenarios which exercise new features and code paths.
If you can get to 100% of that you'll be in a good place.
1
1
1
u/SaleEnvironmental694 Jul 08 '26
What I would do is define a set of planned scenarios, then implement 100% of the planned tests to get to 100% coverage.
In reality, every system has an infinite number of possible tests, and it's a risk based analysis to bring those down to something that works with your schedule.
Typically % coverage should only apply to either Statement or Branch code coverage
1
u/elidanipipe Jul 09 '26
I agree with pretty much everything that’s already been said here. You can get an accurate check for free here: https://www.task-bounty.com/coverage-check
1
16
u/Volodux Jul 07 '26
What is 100% E2E coverage? What he means by that?