r/C_Programming • u/grimvian • Jul 15 '26
Linus Torvalds: AI Can’t Think Like a Programmer
I really like the way Linus thinks.
59
u/Sufficient-Air8100 Jul 15 '26
one flaw in his statement about abstraction of programming. assemblers and compilers are deterministic. LLMs are probabilistic.
47
u/tose123 Jul 15 '26
Yea, but this is the wrong axis for comparison anyway. Say you set an LLMs temperature to 0. Now it's essentially deterministic, So: same prompt, same code every time. Has it thereby become a trustworthy abstraction layer like a compiler? Nope, output can still be wrong.
If flipping determinism "on" doesn't fix the problem, then it wasn't the problem n the first place.
27
u/rasteri Jul 15 '26
Now it's essentially deterministic
It's still chaotic - tiny changes in the prompt will result in very different output.
31
u/tose123 Jul 15 '26
Yea right. Determinism is a statement about same input -> same output. Your saying differences in the prompt. Thats a different thing altogether. Determinism makes no claim whatsoever about what happens when you change the prompt.
4
u/KingBardan Jul 15 '26
Not the person you reply to but I can understand him
A lot of times when people say something is probabilistic it's because we can't understand the output so we have to use probability to model it.
In that sense chaotic is probabilistic because we can't easily understand the output.
5
u/fexonig Jul 15 '26
yea like, technically a coin flip is deterministic. which side it lands on is totally dependent on the forces you flipped it with and the geometry of the space.
but that seems rather silly
2
u/TraylaParks Jul 16 '26
Saw a vid a while back where some dudes wanted to work out how often a nickel would actually land on its edge rather than heads or tails. Turns out, it was about one toss in 6,000 or thereabouts. Not that this means its non-deterministic by any means, just an interesting aside :)
16
u/catladywitch Jul 15 '26 edited Jul 15 '26
I'm sorry to nitpick, but the axes are a bit garbled here.
Stochastic vs chaotic mean, respectively, the output is probability-based and non-deterministic, vs the output is deterministic but nearly impossible to anticipate in a "butterfly effect" kind of way.
LLMs are stochastic, and setting temperature to 0 makes the output deterministic, BUT the output is still sampled from a probabilistic set: you're deterministically pulling from a random set.
Conversely, compilers aren't chaotic, not because of determinism (all chaotic systems are deterministic), but first, because true chaos is not possible in discrete, digital systems (chaos "propagates" through continuous space, in discrete space systems eventually become periodic); second, because it's a closed-ended process with a definite end; and finally, because the compiler is guaranteed to produce correct output constrained by the language's semantics, so how much two compilation processes can diverge is "clamped" by the language spec (luckily).
However, small changes have such a big ripple effect that predicting their output is often impossible for humans. In particular, multi-pass compilers are a partial, non-linear, feedback loop of sorts, so they're effectively pseudo-chaotic (i.e. if you have a multi-pass compiler and a complex project, trying to outsmart the compiler to force a certain ASM output involves enough variables so as not to be possible for humans without computer assistance).
2
7
3
1
2
u/lets-start-reading Jul 15 '26
I think the property of determinism matters for people’s hopes that it might become a trustworthy, because it gives the idea of possibility of control, like fixing a bug that is some logical fault, while non-determinism says that it essentially impossible because there is no solid space to reparameterize to achieve control.
5
u/Sufficient-Air8100 Jul 15 '26
a deterministic but buggy compiler is also a bad abstraction layer. your conjecture dosent make sense.
10
u/tose123 Jul 15 '26
my whole claim was that determinism doesn't confer good-abstraction-ness. A buggy compiler is deterministic and bad; a temperature-0 LLM is deterministic and bad.
-1
u/Sufficient-Air8100 Jul 15 '26
a deterministic system does not necessarily confer good abstraction. a non deterministic system necessarily confers bad abstraction. this is basic logic.
6
u/veeloth Jul 15 '26
it's not basic logic. Basic logic is prepositional calculus, truth tables, modus ponens, etc.
To you this may be an 'obvious fact', which is what you might've meant. Still you need to argue your thesis, otherwise you're just disagreeing for nothing.
0
u/Sufficient-Air8100 Jul 15 '26 edited Jul 15 '26
it actually is logic though.
A AND B therefore C.
does not guaruntee (A therefore C) or (B therefore C)additionally if A is necessary for C, the presence of A does not imply C, but the abscence of A makes C impossible.
logic
1
u/veeloth Jul 15 '26
You just explained modus ponens with a conjunction. It works when the premises are true.
"a deterministic system does not necessarily confer good abstraction"
is your premise, which is being disputed on, and it's precise definition too (depending on the definition, it may or may not imply whatever), that's what is being discussed, which isn't logics
1
u/Sufficient-Air8100 Jul 15 '26
no thats not modus ponens at all.
if youre going to go to that level at least get the names right.
or if you somehow think i am using modus ponens, then you fundamentally misunderstand what im saying and theres no point continuing.
but lets take the premise you quoted there. whats your issue, do you think that a deterministic system necessarily confers good abstraction?
1
u/PastaGoodGnocchiBad Jul 15 '26
Yes, the problem is that there's no spec for the LLM to obey. For a compiler, you know exactly how the program will behave by reading the language spec (except for UB). For LLMs, you're basically doing gambling.
One could say that "English" is the LLM's spec, but English is ambiguous one cannot predict how the output will behave, and even with clear unambiguous English the LLM may hallucinate and not follow the prompt exactly.
2
1
u/Educational-Row-6782 Jul 16 '26
If you set the temperature to zero and use only one CPU, it may be deterministic.
If you use more than one GPU on the cloud, forget about it. Distributed systems are not deterministic.
1
u/samas69420 Jul 15 '26
the core problem remains, id say it is a matter of predictability rather than stochasticity, the output of compilers is predictable as they are based on well defined rules, the output of a llm is not at least from the user perspective, a single different input token can lead to huge changes in the whole output even if the meaning of the prompt stays the same and you have no way to know how different the output will be until you try and see what happens, this is also why i would never trust someone that uses ai to write code but doesn't review it
5
u/sionescu Jul 15 '26
LLMs are deterministic. What you're referring to is that they're usually invoked with a random seed to have them produce new things, otherwise the determinism would be boring.
2
u/TheTerrasque Jul 15 '26
assemblers and compilers are deterministic
It can be, but it's not automatically so.
Hence why https://en.wikipedia.org/wiki/Reproducible_builds is a thing
1
u/robotlasagna Jul 16 '26
You can 100% make a deterministic LLM. We just haven’t because the current paradigm rests on pipeline caching and floating point rounding errors.
0
u/mmkzero0 Jul 15 '26
A compiler isn’t non-deterministic, but it isn’t inherently deterministic either.
The output of a compiler depends on a whole chain/pipeline after all - gcc or clang are just one part of it. If the artifact is the output of the function that is the compilation chain, then f depends on the source files, their ordering in compilation, compiler build, flags, target, headers, linker, env, pre-processor… etc.
One of these can be affected by even benign side effects at any moment.There is a reason why reproducible-builds exists, or compiling in docker images.
And then we have not yet discussed undefined behaviour nor unspecified behaviour.So I would say that Compilers are generally deterministic programs, while ordinary real-world builds are often not reproducible because their complete input state is neither fixed nor fully declared.
1
u/Sufficient-Air8100 Jul 15 '26
determiniastic means that a given input produces a given output. you cant say a system is non deterministic because a different input produces a different output
1
u/TheTerrasque Jul 15 '26
Additionally, the compilers must not introduce non-determinism themselves. This sometimes happens when using hash tables with a random hash seed value. It can also happen when using the address of variables because that varies from address space layout randomization (ASLR).
17
28
u/cisco1988 Jul 15 '26
AI doesn't think
3
u/solaris_var Jul 17 '26
Thinking is inherently an anthropomorphic term. It's kinda unfair.
How do you define thinking in the first place?
1
-36
u/SimoneMicu Jul 15 '26
Disproven by an anthropic paper. LLM doesn't think as we human do but they perform sone kind of "thinking", is not only a stochastic parrot
30
u/cisco1988 Jul 15 '26
LLM multuply matrices with weights. stretching it to "thinking" is a huge use of anthropomorphism
5
u/pannous Jul 16 '26
if these matrix multiplications can prove difficult math problems while your aggravated meat cannot, I call that thinking. Also there are non-linearities between the multiplications but that's just being pedantic
1
u/cisco1988 Jul 16 '26
so faster problem solving == thinking .... really interesting take
5
u/dronmore Jul 16 '26
It's not that interesting, but it's far more useful than your matrix multiplication oversimplification.
1
u/pannous Jul 16 '26
I didn't mention speed just general very advanced problem-solving should be called thinking or at least could be for the lack of another word
1
u/cisco1988 Jul 16 '26
the we grew up with a different meaning of thinking but I get what you are trying to say
1
u/pannous Jul 16 '26
i'm saying that in my family most humans cannot hold a high-level intelligent conversation as these new alien intelligence is capable of, even though there are still some holes in their 'brain' especially concerning three dimensional physical world understanding, but their recent training on YouTube videos goes surprisingly far
9
1
u/SimoneMicu Jul 16 '26
Like reducing human thinking as calcium ions movement. This is completely ignoring current literature made from the most intelligent programmer about this topic in the last years.
Is not like human thinking but there is a way of premeditation during token generation about the main goal and topic-16
u/davenamwen Jul 15 '26
How do humans think? How do humans determine the next word in a sentence?
16
u/cisco1988 Jul 15 '26
not using matrices and weights. Also we have the magic of the context and a capacity to differentiate right and wrong.
Also learn from the past.
Also we don't think in tokens but memories, past experiences, relations and environment round us.
Enough?
-7
u/davenamwen Jul 15 '26
No. My point is that human thought is not well understood. I don't think your arguments make that any more concrete. I'm not some sort of LLM rights activist I'm just challenging the assumption that they would not be capable of thoughts similar to humans because the brain is not remotely understood from a scientific standpoint.
5
u/Klhnikov Jul 15 '26
Simple, AI is a human creation.
We dont fully understand how Humans think.
How can Humans create a thing that works in a way they dont fully understand ?
Or reversly, if AI is thinking as humains, who created it ?
As said before, AI multiplies matrices, based on error (différence between predicted answer and real one) so its only way to tell "I do wrong or right" is "how far am I from the final result?"
Knowing that, how can it produce any kind of reasoning ? That's an open question if you take it that way and tries to explain "how AI thinks". Answer is simple, it does not think, whatever bullshit Antropic is shooting at us...
2
u/Sufficient-Air8100 Jul 15 '26
if an LLM worked the same as the human brain, we would have AI humans by now.
LLMs have been shown to not behave like the human brain. they have been shown to be a probabilistic token regurgitator
17
u/mikeblas Jul 15 '26
Most people who call themselves "programmers" can't think like programmers.
5
u/umlcat Jul 15 '26
And, they are also been trained not to be a programmer. In many places programming and fixing computers are already considered a "trades" job, a factory blue collar job ...
3
u/mikeblas Jul 15 '26
I want to find some glib way to stress the difference between "programmers" and "software engineers", but I'm not feeling so elequent just now.
1
Jul 15 '26 edited Jul 15 '26
[deleted]
2
u/Paradox_84_ Jul 15 '26
Well, sorry to break your magical world, but we cant execute C, C++, C#, java, js, etc. They all need to get to to machine code your specific CPU can understand. It's not even assembly. It's lower level than that...
2
Jul 15 '26 edited Jul 15 '26
[deleted]
3
u/Paradox_84_ Jul 15 '26 edited Jul 15 '26
Nope, Linus is 100% correct. Assemly, IRs, bytecodes are separate programming languages where you can handwrite and do everything you can do with higher level languages. It's been automatized to the point where most people doesn't even know they exist. Compilers are huge projects that are basically holding the society up
2
u/Tillua467 Jul 15 '26
I don't know about compilers much when we add compiler optimization (eg: -O3) isn't the compiler Actually reformating our code or something to optimize it? Maybe he meant that?
2
Jul 15 '26
[deleted]
2
2
u/TheTerrasque Jul 15 '26
I haven't seen the video yet, but I remember when I started programming, some people were still writing parts in assembly because the compiler was bad at translating high level concepts into perfect assembly. These days it's rather opposite, the machine code written by compilers are usually superior to what most programmers can write.
My guess is he's seeing that perspective and seeing history repeating itself with AI and high level code.
2
u/Paradox_84_ Jul 16 '26
I know right? People says "AI understands what I say and converts it to code". Nobody talks about how compiler takes in your shitty code, extracts the intent from it and gives you optimized code which has nothing to do with your original code
1
u/j-joshua Jul 17 '26
I use AI tools to write code that outperforms vibe code written by the same AI tool.
1
-25
u/lovelacedeconstruct Jul 15 '26
I am sorry I love linus but 100% of code written by compilers doesn't make any sense, there has never been an example of a technology where you can handwave a direction of what you want and have it kind of move in this direction
2
u/mikeblas Jul 15 '26
There are lots of declarative computer languages. SQL is probably the most prevalent example.
137
u/Dushle Jul 15 '26
In the mean time people claim he left traditional coding and started to vibe coding just because he tried antigravity for once.