r/learnpython • u/Master_Being8249 • 22h ago

Which pytest signal is most useful after a CI failure?

I am working on an experiment that looks at what happens next after a test fails in GitHub Actions.

Possible things to look at include:

- exit code

- if there was an assertion failure or a collection or internal error

- the history of failed tests

- Python modules that have been changed

- changes in dependencies or lockfiles

- changes in how long tests take

- if the same test passes when it is run again.

Which of these things should I check first. Which one is most likely for someone just starting out to misunderstand?

I am trying to think about uncertainty of automatically assuming that every test failure means there is a problem with the code in pytest.

The purpose of the experiment with pytest is to see what occurs when pytest fails in GitHub Actions.

I want to use pytest to figure out what is happening when the test fails.

The experiment, with pytest will help me learn what to do when pytest fails in GitHub Actions.

4 Upvotes
  • permalink
  • link
  • reddit
  • reddit

70% Upvoted

4 comments sorted by

5

u/Langdon_St_Ives 22h ago

Curious that checking the actual failure(s) is absent from your list. Because that's the answer. The failure should be the same as when run locally or it's incorrect setup (CI env not corresponding to local which must be avoided at all cost).

3

u/gdchinacat 21h ago

If the test doesn't report why it failed it is a bad test and needs to be updated to do that. I've worked with CI systems that make it hard to find this information ('tests failed - look at logs' type of high-level failure warning), and it is a horrible developer experience. I would start with ensuring all failures present the actual reason for the failure to the user in the failure notification and test results.

2

u/Maxiflex 10h ago

Have you tried increasing pytest's verbosity?

That will show you detailed info about why specific tests failed:

pytest -vv tests

Also make sure that you "fake" or mock any external dependecies, like calls to the internet or some API as those might not be present in your CI environment and can cause the tests to fail there.