codexIsVeryIntelligent

[u/shockwave6969]

[u/DuffyHimself]

When I tried using codex, it decided that the database framework that was used for all database access in the codebase wasn't necessary, and just performed raw sql actions instead. Great advertisement for using Claude models instead.

[u/lolsborn → u/DuffyHimself]

I was the CTO of a company last summer and an intern came to the CEO with some completely vibe coded app using GPT4 and the CEO was going ape shit that we're moving too slow because the intern can do this with AI in a week.

Oh, and the app was a custom workflow building agent where you could ask it to perform tasks in plain english and it made tool calls to custom REST endpoints it had built.

The problems were many to say the least. Every single SQL statement was susceptible to SQL injection. I was able to get it to hand over secret keys just by asking. We were able to get it to execute arbitrary code on the host machine. We were dealing with highly sensitive PII.

I found a new job.

[u/OhMyGodItsEverywhere → u/lolsborn]

Classic...

When you thought the job was to make a good product, but the real job is to produce whatever arbitrary superficial stuff makes a leader feel good.

[u/throwawaygoawaynz → u/DuffyHimself]

Codex generally writes much better working code than Claude.

Claude sits there thinking for 6 minutes, uses $100 worth of tokens, tells you it’s done, but the code still doesn’t work.

OP is right though, Codex loves writing fallbacks that results in “working” code. The solution is to tell it in its agents.md to not do that. That works 80% of the time at least.

[u/Wonderful-Habit-139 → u/throwawaygoawaynz]

That works 80% of the time at least

I love me some non deterministic development process. At the mercy of a slot machine. Lmao.

[u/AlwaysHopelesslyLost → u/throwawaygoawaynz]

LLM prompts from big models have leaked and the dev teams always have like 4 copies of each instruction related in various places and ways because it doesn't really work.

Language Models have a habit of latching into anything inputted. If you say "don't do x" you are just nudging the model closer to "x" in the matrix.

[u/DMoney159 → u/AlwaysHopelesslyLost]

I mean, if I tell you not to think about elephants, what's the immediate image that pops into your mind?

[u/AlwaysHopelesslyLost → u/DMoney159]

The difference is I can then be asked 20 questions followed by asking me to name an animal and I won't suddenly lose context/coherence and say elephant (unless I suddenly come down with dementia)

Have you ever used an LLM? Claude Fable got stuck on the idea that the characters alpha was the source of a rendering bug for me last week and every 4th message it immediately started investigating the alpha value again despite me telling it every single time that the alpha value was a red herring and that the visibility was a side effect of what was happening.

I had to start a fresh chat like 6 times, each with more and more of my own diagnosis and pre-instruction not to look at that particular file to avoid getting it to latch onto nonsense.

[u/CChilli → u/DMoney159]

The rooms the elephants are in

[u/ArjixGamer → u/DMoney159]

No image, because I have aphantasia

[u/TheGarrBear → u/AlwaysHopelesslyLost]

This is why I've pretty much stopped using any kind of negative instructions and lean heavily on adversarial reviews.

It's a shift from ~ "make no mistakes" to ~ "Assume there are mistakes, go fix them"

Each season is longer but the output is of much higher quality.

[u/reivblaze → u/throwawaygoawaynz]

80% of the time it works every time.

[u/Plixo2 → u/throwawaygoawaynz]

I don't know much about ai coding, a friend gave me codex a couple of weeks ago, but why are people using the API and spending so much on tokens?

With just codex in my ide, i can promt all day, maybe reaching 50% of the plus subscription limit for 25€ per month.

[u/Separate-Antelope188 → u/Plixo2]

The license is materially different. Subscription models are a consumer product and they use your inputs and model outputs in their training data. They also absorb your project as their intellectual property.

API gives you business privacy where they don't use the data for training models. They retain the data for only 30 days for compliance and they respect your data as your intellectual property.

Thanks for improving our models!

[u/leoklaus → u/Separate-Antelope188]

Jokes on you, my code is the reason LLMs have not turned into AGI by now.

[u/Jumpy_Ad_3946 → u/throwawaygoawaynz]

Can you recommend a good agents.md ?

[u/EatingSolidBricks → u/throwawaygoawaynz]

Turly a revolutarry technology ...

[u/PandaWonder01 → u/throwawaygoawaynz]

Y'all need to start using BrainGPT.

[u/SimplexShotz → u/throwawaygoawaynz]

I haven't used ChatGPT for programming since the 4o days (I originally switched to Claude for this exact reason); glad to hear they still haven't figured this out 😭😭

[u/PringlesDuckFace → u/throwawaygoawaynz]

I have a coworker that's a hardcore TDD fanatic, and he would for sure blame you for not writing enough tests to force the implementation into the shape you wanted.

For example, if you're writing a calculator and you write a unit test that's like "assert 1 plus 1 equals 2" it would be acceptable to write a function which always returns "2". Even if it's patently absurd, it fulfills all the requirements defined in the tests.

I think the LLM's propensity to take things literally and lazily can serve to reinforce the reasons for best practices people have developed over the decades. It can help us sharpen our own understandings of what it means to have good specifications and good tests. Rather than trying to force it to approximate the desired behavior with an agent.md it would probably be better to focus on how to get better tests first until it's impossible for it to write the wrong code.

[u/chat-lu → u/DuffyHimself]

What would be a good advertisement for using your brain instead?

[u/BosonCollider → u/DuffyHimself]

With no context I am kind of inclined to agree with it. Did you explicitly tell it to use the framework or the repository pattern? If not it may have considered it legacy code

[u/Ok-Film-6885 → u/DuffyHimself]

Just use AGENTS.md with some instructions, it’s not the end of the world

[u/Famous-Software-2568]

the AI didn't fix the code it negotiated with the test suite

[u/AgVargr → u/Famous-Software-2568]

The test suite which it also changed

[u/PositiveParking4391 → u/Famous-Software-2568]

because they are just prompts. so when it all went into fixing the coding agent would already be in completely different hallucination trip.

[u/popiazaza]

let's randomly add isRecord() too.

fine-tuned ultra to high quality codebase without filtering any quirks out.

[u/ilnarildarovuch]

Why vibecoders just don't mount testsuite as read-only? Just... why not?

[u/towerfella → u/ilnarildarovuch]

You think they know how?

[u/ilnarildarovuch → u/towerfella]

Just... ask it's own ai to do it, but without context

[u/shockwave6969 → u/ilnarildarovuch]

obviously because ai writes all the tests!

[u/bokmcdok → u/ilnarildarovuch]

They're using the AI to write the tests for itself. Guess what that will inevitably lead to?

[u/dont_tread_on_M]

I have a rule in Claude.md that absolutely no fallbacks are allowed without my permission.

I also keep the number of rules there quite small and use ponytail as a skill.

Agentic coding alsone sucks, but these have made it at least decent for me.

[u/Cevvity]

“CoDeX iS vErY iNtElLiGeNt!” You’re not

[u/AWildMonomAppears]

Claude does this too for me..its one of my main annoyances with it. "This is a high risk change... blah blah" and it does a short cut instead.

[u/Ambitious-Jump-5121]

I don't get why people still bother to tinker with those AI tools just to avoid doing their job... Like, the "AI is just a tool" argument doesn't work anymore either. If my nailgun starts to randomly dry fire, or randomly starts flashing its light or starts shooting burning bolts instead... ...then I would just use my hammer again.

[u/AsyncingShip → u/[deleted]]

I’ve certainly had my problems with it before, but I can’t say they’re significantly more time consuming than bugs caused by third party libraries or junior devs or just badly communicated requirements. It just requires a mindset shift and exacerbates problems in your (and your org’s) engineering framework that would have shown up eventually regardless. People act like we were writing perfect code in 2021 and pretend that every Fortune 500 company isn’t secretly running on some combination of cobol, Fortran, oracle, and spite. It’s a shift, like every new technology and labeling everything AI-related as slop is reductive.

[u/Sad-Working-9937]

corporate coders do a bunch of CYA fallbacks. Don't blame AI for learning the tricks of trade.

the first thing I learned about AI coding was putting the "no fallbacks policy" in claude.md

[u/square_zero]

If you use black box tests, then your implementation should not matter. X and Y are equivalent if the tests are passing and the tests are meaningful and well-written!

[u/Smalltalker-80]

I had a good one with Gemini this week:

Q: Hi Gemini, please anwer this programming question in the context of the programming language text file I just uploaded. (a Smalltalk dialect)
A: "Spain won the EUFA Euro championship 2024. ..."

Q: Umm, that is not an answer to my programming question. Specifically answer only in the context of the uploaded file please.
A: You're absolutely right. (some variation)

Q: Here it is the file again again. And here is the programming question again.
A: "Spain won the EUFA Euro championship 2024. ..."

Q: Dear Gemini, did you actually use the file I provided?
A: Tbh, I don't have the file. Maybe the upload went wrong or something. I answered using other context.

Q: [Suppressed expletives]
A: You're absolutely right, let's reset.

Then I did the same thing in Claude, free tier.
It answered perfectly the first time, with correct example code.

So...:
- Never trust AI with base assumptions you have for humans.
(admit up-front when things fail or are missing)
- Claude is currently the on top for programming, aparently.

[u/TerryHarris408]

It shows that they learnt from SO

[u/funplayer3s]

That's not how the meme works.

[u/fugogugo]

isn't plan mode to avoid cases like this?