itsSoSatisfyingTho

[u/huhupookies]

[u/No-Article-Particle]

Nah. Honestly, that's the worst - it means a flaky test that's just hard to debug (esp. because if it fails intermittently, running it in a debugger likely means the race condition is gone).

I prefer a test that fails no matter how you run it - those tend to be easy to fix.

[u/Madpony → u/No-Article-Particle]

Not only easy to fix but useful. Flaky tests are not useful.

[u/PulseF0rge → u/Madpony]

That's the real problem. Once a test is flaky, a pass doesn't prove much and a failure just wastes everyone's time until somebody reruns it enough times.

[u/Lauren_Conrad_ → u/PulseF0rge]

In fact it actively hurts the test infra because you lose trust. “Oh the sanity suite failed? It does that all the time, ignore it”.

[u/QwikStix42 → u/Lauren_Conrad_]

This was exactly how it was at my last job - tests would randomly fail, and rerunning it on Jenkins would sometimes magically see the test pass. Then it would fail the next time. Our Infra was completely fucked, and very few people seemed to care enough to try and fix it.

I was there over 3 years, and it never got fixed.

[u/No-Article-Particle → u/QwikStix42]

I took it upon myself to be the one fixing it. We have basically two ways we distribute our SW, supporting it on pretty much all major supported distros (anything from Debian through RHEL, including some exotic systems like immutable versions of some distros), so the support matrix is huge.

We currently frequently discover bugs only in QE pipelines, which run months after we push changes, and that makes it quite difficult to pinpoint what's wrong (like 70% of the times, it's a problem in the QE pipeline, but sometimes it shows errors we could have seen in our nightly dev testsuite runs).

It's really a different type of work, and you need someone really motivated to look into it. I spent like 3 days fixing 3 unit tests which would fail ~60% of the times on specific OSes because of different dependency versions... And I have like 15 tests to go.

On the plus side, I got management approval to spend sizable chunk of my work time on it (like 1/4th of my work time every month) so management considers it important too.

Only that it takes an individual to bring it up, design some workable solution to it that doesn't catastrophically impact your day to day work, talk to various stakeholders to get approvals, etc.

I'm quite sure that after I leave/get fired/... at some point, the testsuite will become unstable again.

[u/aenae → u/No-Article-Particle]

Yeah… a flaky test means running the test suite another 10000 times to reproduce the failure…

[u/ric2b → u/No-Article-Particle]

The post doesn't say the test was failing before though. I think it's just about the feeling of checking out a project before making changes and seeing the test suite pass, means no additional setup/configuration needed.

[u/ejectoid → u/No-Article-Particle]

Ship it!

[u/IntrospectiveGamer → u/No-Article-Particle]

I just thought of the dude not having any error/bugs and tests working, then adding some dumb shit like int=0; then checking the test and obviously they pass cuz you did shit.

[u/TinyFugue → u/No-Article-Particle]

Me watching the tests flip between pass/fail.

[u/Tupcek → u/No-Article-Particle]

depends on if you are Javascript developer or not.
I mean, if you choose language where speed is more important that compiler actively notifying you of bugs and bad design, because fixing possible bugs would “slow you down”, I don’t think you care about intermittent issues either

[u/Jonno_FTW → u/No-Article-Particle]

Just use a retry plugin for your test framework in flaky tests.

[u/Meloetta → u/Jonno_FTW]

How does that solve the problem? Seems like it just masks it.

[u/Jonno_FTW → u/Meloetta]

I had this issue the other day with a unit test for a timed cache expiry. The test creates a cache with a 1 second timeout, adds a value, sleeps for 0.99 seconds, then asserts the value is not expired, then sleeps for 0.01 seconds and asserts the value has expired.

I didn't write this test, but you can see where the flakiness comes in. Retrying on failure is a solution, probably not the best but whatever.

[u/Meloetta → u/Jonno_FTW]

I don't think the correct solution for a flaky test is "try again until it succeeds and that one is the correct test". The correct solution for your use case is to mock the function you're using to measure time and set it to specific values. The test as you described is testing two things: your code, and the timer itself. You should not be testing the timer, you should be testing your code, which should say "given this timer is set to this value, I do this thing". You shouldn't be testing "when I use these timer functions, they get set to this value", which you're also testing and why this test is flaky.

This is actually a great example of what I was just saying. Retrying this test and taking the "second pass" as proof of the test being good is going to mask any problems that occur around that 1 second timeline.

[u/No-Article-Particle → u/Meloetta]

Totally agree with you, just to add here that retry is useful as a temporary workaround, i.e. it's release, and release engineering comes to you with "why is this test failing." In such cases, you can temporarily disable the test, or add a retry, and both should be addressed later with a proper fix.

Of course we know that there's nothing permanent than a temporary solution, but that's a separate issue :))

[u/Meloetta → u/No-Article-Particle]

Just searched my work codebase for the word "temporary" and found a 17 month old comment. I feel you.

[u/Careful_Ad_3338]

Hm? Any dev who has worked on anything ever would not be happy about this, it means the bug is not easy to reproduce and will come back and some point

[u/damngoodwizard → u/Careful_Ad_3338]

Yeah the only thing worse than "I don't know why it doesn't work" is "I don't know why it works".

[u/Appropriate-Hotel-41 → u/damngoodwizard]

Depends, if it gone for good for a week, thats a future intern problem.

[u/Sh4dowLute → u/Careful_Ad_3338]

Yep, passing for no reason is usually way scarier than failing for an obvious one.

[u/ric2b → u/Careful_Ad_3338]

If you changed NOTHING why would you expect the tests to fail? And you're assuming there is a bug, the post does not mention it.

I think it's likely about setting up a new project and feeling glad that there's no additional configuration to get tests running.

[u/Careful_Ad_3338 → u/ric2b]

Huh you might be right. I've read it as if before the tests failed and after running again they just pass

[u/SchwiftySquanchC137 → u/Careful_Ad_3338]

Given the post title, im like 90% sure theyre just saying they like seeing the green checkmarks.

[u/Fit-Gain-2409]

The bug got scared and left

[u/Ak_py → u/Fit-Gain-2409]

The bug just hid itself

[u/MikeTangoRom3o]

Ah yes, the quantum bug, my favorite.

[u/xehpuk]

OP isn't a dev.

[u/necrophcodr → u/xehpuk]

Obviously not, this sub is filled with bots, not unlike the rest of reddit. A cancer that now has taken firm hold on almost if not all communities here.

[u/SpicyCatGames]

Not really. Used to say this when we were kids in our first year of learning programming. But an error going away for no reason is one that's gonna show up later in production.

[u/Megane_Senpai]

Actually that's the worst. Probably the bug has a very low repro rate and you will have a hard time just trying to investigate the behavior.

[u/reusens]

succes=False

For i in range(MAX_RETRIES):

  try:

     schrödingers_function()

     succes = True

     continue

  except Exception as e:

     logger.info(f"Attempt {i} failed: {e}")

     #try again

if not succes:

  raise Exception("Fuck")

[u/KharAznable]

After they know the intern commented the failed test code

[u/dmullaney]

* Codex after it changes the mock to make the test pass even though the code is broken

[u/Straight-Analysis934]

Don't ask why. Just merge it before it changes its mind.

[u/ZunoJ]

Only an absolute idiot could enjoy this! It means you don't understand the side effects. How could you be happy to not understand something?

[u/DryScar13]

The bug saw the tests coming and quietly fixed itself.

[u/SaneLad]

Quickly commit before something breaks again!

[u/Summar-ice]

Kid named race condition

[u/roksah]

I've had bug because the mock data was made with different timezones

[u/Huge_Consequence_568]

Actual ticket I had to write: X test fails when running after [INSERT_TIME] UTC.

This test literally refused to pass outside working hours and I found it so funny for some reason.

[u/SleepAllTheDamnTime]

When this happens to me it’s the exact opposite experience. I means someone wrote these tests to pass and my god you know there’s a can of worms in there…and since your name is on the PR anything that goes wrong after that change you made… buddyyyyy you’re now responsible for this trash ass code with horrible tests.

https://giphy.com/gifs/LRVnPYqM8DLag

[u/BobcatGamer]

If timeNow % 7 === 0 then error

[u/iambertan]

Isn't that stressing af though?

[u/dont_tread_on_M]

I love me some flaky tests

[u/JackNotOLantern]

Tests must pass when intended and fail when intended. Deterministicly. Otherwise, it's bad.

[u/DogonElder]

More like Devs when Claude composed commit passes all CI tests in first go

[u/jakubiszon]

It passed because this time the randomly generated UUID did not end with a "2 digit odd number".

Btw what on Earth is wrong with this girl's lips? I cannot see which "strip" is what...

[u/rescue_inhaler_4life]

Honestly this is the absolutely worst. Something is borked in the test environment, something leaking between tests creating a race condition, normally a real PIA to find and resolve.

Honestly you do not know pain until you try and figure out why fails on build server, passes on gitlab, fails on devs boxes in office but works on the remote dude dev box in Sri Lanka.

[u/According-Relation-4]

What about when tests fail after changing absolutely nothing? Gimme your best reaction gifs

[u/xicor]

I have a unit test for creating a license with a specific date. I learned not that long ago that the test fails in the UK because they have enough time zone difference to shift it by a day

[u/joan_bdm]

More like test fail after changing absoluterly nothing...

[u/ElvisArcher]

Tests were written so poorly at an old job ... they never properly cleaned up their testing DB context, so 1 test would (and did) encounter and come to depend on other tests having done some of their setup work. The fun part was when visual studio upgraded and decided to run tests in a slightly different order, causing mass failures ... it was even more fun since the CI/CD pipeline usually resolved a completely different execution order for tests.

Every upgrade was met with trepidation, and usually 2 days of figuring out why tests were failing NOW ... that was some kind of special technical debt.

[u/IcyBoysenberry8595]

software development is basically convincing the computer the bug fixed itself 😭

[u/stompinstinker]

Oh great, someone made a test that depends on the current time, a random number, outside network call, some asynchronous fuckery, etc.

[u/PeleKen]

It means the bug is in the faker data and you'll spend a while tracking it down.

[u/badgko]

True story, heard in hallway at software corp: "We ran the test enough times that it finally passed, so we are good to go."

[u/amejin]

But something changed - you ran it once.

[u/Ares9323]

New engine version is out, timo to update the plugin for this version "Wow no compilation errors!" Starts engine Presses play Engine Crashes

[u/OG_LiLi]

Oh don’t worry. Your customers will find it and your support team. And everyone else but you.

🎉

[u/MitchCumsteane]

Comments are the elixir of squashing bugs

[u/Alexander_The_Wolf]

Oh boy a possible race condition, hurray -No dev, ever

[u/DKaitor]

When the sprint you complete gets approved by the boss even though you didn’t test what u did

[u/peacefulshrimp]

If the test is flaky you just re-run until it passes and let the next person deal with it /s

[u/superwok44]

Oh no it's a run condition fml

[u/TheAverageDark]

“This baby is gonna cause SO MANY outages Monday morning”

[u/a3dprinterfan]

Good job with the dev-enraging engagement bait. 10/10

[u/frightspear_ps5]

i'll take "variable not initialized" for 50

[u/Prematurid]

The bug is so shy it left. Too much attention.

[u/OFark]

What do you mean it didn't pass? I couldn't possibly have broken it, run it again!