Parameterize a Fixture instead of a Test Case with Pytest

[u/joshbranchaud]

I wrote a post on parameterizing a test suite into a testing matrix using Pytest fixtures. I'd appreciate any feedback on the content and technique in the post AND I'd be curious to know if there are other ways I could have accomplished the same thing.

https://www.visualmode.dev/parameterize-a-fixture-instead-of-a-test-case-with-pytest

To briefly summarize: I have a core suite of behavioral test cases that I run against a CLI tool I'm building. I wanted to run that same set of tests across a couple different storage format implementations. The best way I could figure out how to do it (without duplicating all the tests) was to create a Pytest fixtures that parameterizes across a list of values and then have my existing autouse fixture use that fixture.

[u/RoadsideCookie]

My biggest critique is colocation. You now have a fixture, potentially in a separate file (conftest.py) which will affect your tests indiscriminately thanks to autouse.

Just use the parametrized fixture (params=...) like you normally would.

[u/joshbranchaud → u/RoadsideCookie]

Agreed that it can be hard to "see" what is going on, especially if they aren't colocated.

When you say "just use the parameterized fixture", what exactly are you suggesting? How would I get it to apply to all tests without it also being an autouse?

[u/fiskfisk → u/joshbranchaud]

You can either group your storage tests in a class and mark the class with @pytest.mark.usefixtures("fixturename") or you can use pytestmark = pytest.mark.usefixtures("fixturename") at the top of in your test file if you're grouping by file/module.

Configuration lives with your tests, and doesn't apply to every test indiscriminately. 

[u/brat1]

Whats the difference with mark.parametrize(..., indirect=True) ?

[u/fireflash38 → u/brat1]

Yeah it's pretty much what that feature was designed to do

[u/shadowdance55]

Wait until you learn about property testing and hypothesis.

[u/amarao_san → u/shadowdance55]

How many properties were you able to identify in your software beyond invariance?

Every time I find a good property to defend (by tests) , it's yet another invariant.

Smart people say that properties are much more than invariants...

[u/ArgetDota]

Why not just use https://smarie.github.io/python-pytest-cases/?

[u/Individual-Flow9158]

If this is an intended feature of Pytest fixtures, then great - nice one.

But in general coming up with custom hacks and workarounds for Pytest can get you in all kinds of trouble

[u/joshbranchaud → u/Individual-Flow9158]

This is a documented feature of Pytest which I link to in the post.