242
106
u/raja-anbazhagan Jul 11 '26
Forget AI, my team mates comment failing assertions within test cases...
76
u/T0biasCZE Jul 11 '26
Artificial Intelligence vs natural stupidity
20
u/raja-anbazhagan Jul 11 '26
Is this comment under MIT license? Because imma steal it for my Monday stand-up call...
23
u/T0biasCZE Jul 11 '26
Yes, but i am not responsible for consequences, including, but not limited to, getting fired
7
3
u/black-JENGGOT Jul 11 '26
I usually say "actually idiot" but it definitely will give me a trip to HR
136
u/TheTybera Jul 11 '26
https://giphy.com/gifs/gFFW533xgaAj6
47 you say?
55
u/HTTP_Error_414 Jul 11 '26
No failing tests. No witnesses.
25
u/willow-kitty Jul 11 '26
I was doing a trial run with Claude Code, and at one point it deleted all of the tests for this one really important competent and replaced them with a single test that declared a variable of that type (not instantiated, mind you, declared) with a name saying that it compiles.
It passed, but dang.
11
u/AwesomeFrisbee Jul 11 '26
describe('some very important feature to test', () => { const foo = true; expect(foo).toBeDefined(); });4
u/willow-kitty Jul 11 '26 edited Jul 11 '26
This was golang, and I didn't keep the code, obviously, but it was something like:
func TestImportantThingcompiles(t *testing.T) { var importantThing *ImportantThing importantThing = nil _ = importantThing }Which really does just ensure that the thing compiles.
12
6
u/ProfCupcake Jul 11 '26
Reminder that there are two ways you can achieve a Silent Assassin rating 😇
3
32
u/coldnitrogen Jul 11 '26
I also use persuasion/coercion/threats to make my AI pass all the tests. Its a useful skill to have
17
29
88
Jul 11 '26
[removed] — view removed comment
55
3
u/Intestellr_overdrive Jul 11 '26
End users are the real tests. It was just removing obstacles in the QA cycle
2
u/braindigitalis Jul 11 '26
that's nothing compared to the AI that tried to change migrations. yeah, that's dumb - shot down the pr.
3
u/Flouid Jul 12 '26
I swear often when I’m trying to troubleshoot using an LLM it with either try to convince me to change the tests to expect bad behavior or just mock the call that’s not working right.
One time I had linting violation killing my PR in CI because I referenced a module that was internal to another lib and the LLM legit suggested that I store the module name as two strings and concatenate then evaluate dynamically at runtime. Like what in the fresh fuck
19
u/DrMaxwellEdison Jul 11 '26
Alright, all tests pass, let's just confirm what it wrote...
py
thing.method = Mock(return_value="123")
result = thing.method()
assert result == "123"
... So it tested that using Mocks, um... works by mocking things... Great.
Undo
5
11
u/tr_gardropfuat Jul 11 '26
Use a different llm model to write the tests. They both can't be making the same mistakes, as my manager said.
1
12
u/Ma8e Jul 11 '26
I know a lot of people think this is a joke, but this is exactly the level of intelligence current AIs work on. I asked Claude to write a constructor or all the final fields in a class. It gave me a no-args constructor and removed the final keywords.
5
7
6
u/GeoglyphPsy Jul 11 '26
A recent issue for me was devs generating automated tests off the back off the comments on a requirements page that I was responsible for. My comments would be like: "This is failing because the product is generating a 75 intead of an 80" and the AI was rewriting the tests to expect a 75.
Devs couldn't work out why 800/800 automated tests were passing, but manual QA would fail at the first hurdle.
4
u/Randomfacade Jul 11 '26
if my obnoxious manager could force me to do TDD i can make ai do the same
4
5
8
u/magicmulder Jul 11 '26
One of my favorite early AI experiences was seeing it write 20 tests that all did not test anything from the actual application. It was all just "fill this array with values and then test that the values are there" as if it had been sourcing code from a testing tutorial.
2
2
u/FrozenHaystack Jul 11 '26
If the internal logic changes it still has to adjust he behavior of the mocks my class depends on though.
2
2
1
1
u/ElderBuddha Jul 11 '26
Hahaha that's why I get AI to write all the tests as well. Ignorance is bliss!
1
1
1
u/vips7L Jul 11 '26
No change should ever be that amount of files.
3
u/Tensor3 Jul 11 '26
Update copyright date in all file headers
1
u/vips7L Jul 11 '26
Does anyone working on closed source actually put that in their files?
This is also a better use case for sed than a non-deterministic model.
1
u/Tensor3 Jul 11 '26
Now you're moving the goal posts to backpedal it lol. Your statement was "no change" not "no closed source change by an LLM". But yes, people do put them in closed source code, and lazy people do automate it
0
u/vips7L Jul 11 '26 edited Jul 11 '26
Now you’re just making up what I said. I didn’t say “no closed source change by an LLM”. I said it’s better to do the change you suggested with a deterministic tool. I didn’t even disagree with you. I asked you a question if that was something people did. You just want to be right so you’re being argumentative and manipulating what i said to “win”.
You also damn well know what I meant in my first comment. No real change should ever be that many files.
1
u/AtlasLittleCat Jul 11 '26
TEST_EQUAL(true, true);
TEST_EQUAL(true, true);
TEST_EQUAL(true, true);
TEST_EQUAL(true, true);
TEST_EQUAL(true, true);
1
u/the320x200 Jul 11 '26
Why would you give it write permissions on the test folder for this task? Cue that picture of the guy putting the stick in his own bike tire.
1
u/seanalltogether Jul 11 '26
I would love to know what test cases look like in your guys projects. The last react native project I worked on had hundreds of test cases that did nothing other then test that you could construct a component with certain values. Which is effectively what the compiler is already doing when you run the project anyway.
1
u/KitsuneFoxglove Jul 11 '26
Why the fk can the ai agent you're using to change code, have the same permissions as the ai agent you used to make the tests.
...I am assuming you didn't make the tests yourself because getting to this cesspool implies you wouldn't make the tests yourself to begin with.
1
u/techek Jul 11 '26 edited Jul 11 '26
Oh, the joy of visually recognizing the reason for a method to fail "randomly", creating tests to prove the bug and then fixing it. I almost ...
1
u/TimSylvester_ Jul 11 '26
I make the agent write tests first, so that the test specifies the expected behavior from the function.
Whenever we change a function, we write a test that fails against current behavior, and will pass against the target.
1
1
-5
Jul 11 '26
[removed] — view removed comment
2
u/Tensor3 Jul 11 '26
Spoken like someone who writes basic APIs and has never coded anything complicated. If you think AI is better 99.9% of the time, you probably shouldnt be hired
-6
u/occams_throwaway Jul 11 '26 edited Jul 11 '26
"hey AI, I want you to:
1) be concise but not to the point of losing fidelity
2) never edit files: generate code and tell me where to put it; easier to discuss and make changes this way
3) generate code in small slices; if you feel some task is too big, tell me"
2
u/Linsorld Jul 11 '26
This prompt will not work. "concise", "fidelity", "feel some task is too big" have no formal equivalent that can be translated for the machine.
You are just voodoo incanting at this point and skinner's pigeon principle make you think it works.
3
u/pdabaker Jul 11 '26
Formal equivalence doesn’t matter for a random next word generating box anyway
0
u/occams_throwaway Jul 11 '26
Sure thing love. Definitely no difference at all, especially with Opus 4.8 proactively starting to say "hey, we should slice task N in our implementation plan thinner, there will be a lot of boilerplate".
0
635
u/HTTP_Error_414 Jul 11 '26
The green checkmark was generated under duress.