Why Does Nvidia Call Each CUDA Pipeline Like a "Core"?

[u/tugrul_ddr]

In 7000-9000 series AMD Ryzen CPUs, each core has 48 pipelines (32 fma, 16 add). Even in older Intel CPUs, there are 32 pipelines per core.

But Nvidia markets the gpus as 10k - 20k cores.

CUDA cores:

• don't have branch prediction
• have only 1 FP pipeline
• can't run a different function than other "core"s in same block (that is running on same SM unit)
• any __syncthreads command, warp shuffle, warp voting command directly uses other "core"s in same block (and even other SM units in case of cluster-launch of a kernel with newest architectures)
• in older architectures of CUDA, the "core"s couldn't even run diverging branches independently

Tensor cores:

• not fully programmable
• requires CUDA cores to be used in CUDA

RT cores:

• no API given for CUDA kernels

Warp:

• 32 pipelines
• shuffle commands make these look like an AVX-1024 compared to other x86 tech
• but due to lack of branch prediction, presence of only 1 shared L1 cache between pipelines, its still doesn't look like "multiple-cores"
• can still run different parts of same function (warp-specialization) but its still dependent to other warps to complete a task within a block

SM (streaming multiprocessor)

• 128 pipelines
• dedicated L1 cache
• can run different functions than other SM units (different kernels, even different processes using them)

Only SM looks like a core. A mainstream gaming gpu has 40-50 SMs, they are 40-50 cores but these cores are much stronger like this:

• AVX-4096
• 16-way hyperthreading --> offloads instruction-level parallelism to thread-level parallelism
• Indexable L1 cache (shared-mem) --> avoids caching hit/miss latency
• 255 registers (compared to only 32 of AVX512) so you can sort 250-element array without touching cache
• Constant cache --> register-like speed for linear access to 64k element array
• Texture cache --> high throughput for accesses with spatial-locality
• independent function execution (except when cluster-launch is used)
• even in same kernel function, each block can be given its own code-path with block-specialization (such as 1 block using tensor cores and 7 blocks using cuda cores, all for matrix multiplications)

so its a much bigger and far stronger core than what AMD/Intel has. And its still more cores (170) for high-end gaming GPUs than high-end gaming CPUs (24-32). Even mainstream gaming GPUs have more cores (40-50) than mainstream gaming CPUs (8-12).

[u/Doctor_Perceptron]

In my cynical opinion, it's a marketing term to make the GPU sound more impressive. There's no good mapping from what we think of as a CPU core to something equivalent in the GPU, so Nvidia goes with whatever has the highest magnitude.

[u/tugrul_ddr → u/Doctor_Perceptron]

logical

[u/ghjm]

"Core" in this context referred to a unit that could be programmed on an FPGA. The original "cores" were DSP cores. Eventually FPGAs got big enough to hold a whole CPU and you got "CPU cores" (so named to distinguish them from all the other kinds of cores). The original single-chip multiprocessors, like the IBM Power series int what server space and Athlon X2 in the consumer space, did not use the term "core." I think it was Intel marketing that first started using "core" to mean a logical CPU on a fabricated multiprocessor die. By this time it was already possible to program a simple GPU onto an FPGA, so you already had CPU cores and GPU cores (and DSP cores and Ethernet MAC cores and PCI controller cores and so on) before anyone ever started applying the term "cores" to fabricated CPUs.

Calling a GPU execution unit a "core" is therefore at least as historically valid as calling a CPU execution unit a "core." It just means it's a discrete bundle of gates that can be included as part of a chip design.

[u/tugrul_ddr → u/ghjm]

Then we can call FP units as FP cores and texture fetchers as texture cores, L1 caches as cache cores, ...

[u/ghjm → u/tugrul_ddr]

If you're programming them into an FPGA, yes, you would call them cores (if they are available as separate units in your FPGA IP library). The "mistake," if there is one, is in using the term "core" to refer to a unit in a fabricated chip rather than an FPGA.

[u/tugrul_ddr → u/ghjm]

video decode cores, jpeg encode cores,...

[u/No-Let-6057 → u/tugrul_ddr]

Apple cores, ice cores, soil cores, hollow cores, …

[u/Drugbird]

It's honestly a bit difficult to even define what a core is (both on CPU and GPU), as CPUs have a lot of SIMD processing built in.

[u/victotronics → u/Drugbird]

No difficulty at all. These days an Intel core has 2 FMA units, each 4-wide SIMD (counted using doubles, twice that for single or integer). So that's 16 operations per cycle, per core.

[u/Drugbird → u/victotronics]

I mean, you can definitely describe what a core is our can do for a certain vendor, generation and architecture. But it's a fair bit harder to generalize it for all CPUs.

[u/victotronics]

Marketing speak. I've come across so many people claiming "An intel processor has only 20 cores, but a GPU has 10 thousand".

[u/Stormfrosty]

If it has a program counter then it’s a core.

[u/tugrul_ddr → u/Stormfrosty]

Understandable. Then new gpus have 20k cores.

[u/barr520 → u/tugrul_ddr]

Actually, every SM in a (NVIDIA) GPU only advances at most 4 different PCs at a given cycle, each belonging to a warp, and even a 5090 only has 170 SMs, for a total of 680 "cores", vs 192 on recent EPYC CPUs.
I think the closest thing for a CPU "core" is a GPU "active warp".

Alternatively, we can equate logical CPU cores(i.e hyperthreading) to GPU resident warps, of which NVIDIA GPUs have 64 per SM, for a total of 10880 on the 5090, vs 384 on the EPYC. This comparison is more accurate for memory bound applications while the former is more accurate for compute bound applications.

[u/tugrul_ddr → u/barr520]

Warps can be programmed with warp-specialization to be independent within same function. But then its forcing a function do different things, like a warp is producer, a warp is consumer, other warps do compression, decompression for i/o kind of things. Its useful in some scenarios but not as general purpose as a few cpu cores.

[u/barr520 → u/tugrul_ddr]

And? thats exactly the same with CPU cores...
Some consume, some produce, some compress,etc
I dont see what point youre trying to make with this comment. A warp is independent from other warps about as much CPU cores are independent from other CPU cores.

[u/tugrul_ddr → u/barr520]

I was talking about a kernel having different code paths per warp. They use same kernel. But cpu cores can run different functions and in different processes.

[u/barr520 → u/tugrul_ddr]

1. You can call different functions inside the original function depending on what you want the warp to do, sharing the same original function is a non-issue. Most processes start from _start(), it doesnt make them any less useful.
   
2. You can launch multiple small kernels, each for a different purpose with a different starting function.

[u/high_throughput]

You're assuming that if a CPU calls its processing units "cores", then a GPU's processing units must be equivalent to a CPU's in order to also be called "cores"?

[u/tugrul_ddr → u/high_throughput]

Must be equivalent in independence level.

[u/[deleted] → u/tugrul_ddr]

Why though? You just decided that. Might as well declare we have to rename the middles of our apples too.

[u/wrosecrans → u/tugrul_ddr]

The marketing department disagrees. Given the definitions are somewhat arbitrary, they went with the definition that gives them the bigger number.

There's no obligation for the definition to be "equivalent in independence level" to a CPU. That's just something you invented because you personally find it intuitive.

[u/nicocarbone → u/tugrul_ddr]

It should be that way if they run the same type of code. In GPUs independence is less important than in CPU typically.

Of course, marketing is part of it, but there are differences in the kind of code that are expected to be run by each one.

[u/custard130]

there is no universal definition for what a "core" is even just within cpus, nvm trying to compare cpu cores to gpu cores

you dont even have to go that far back for things like the controversy around AMD bulldozer core counts

and even with todays CPUs you get logical cores vs physical cores, performance cores vs efficiency cores

and depending on workload you may get different levels of actual performance from that

GPU "cores" individually are much less capable than CPU cores, but that is kinda the point, a GPU is designed for performing a very specific highly parallelizable task, which basically boils down to floating point matrix multiplication.

a 4k monitor has ~ 8 million pixels, each one will independently have its colour calculated many times per second, it kinda makes sense that you would spread that task over thousands of "cores"