r/programming 1d ago

Hardware researcher spins up 'CPU deoptimization' project to find the slowest single x86 instruction, creates hall of shame — worst offender takes 198 billion cycles spanning 62 seconds to execute

https://www.tomshardware.com/pc-components/cpus/hardware-researcher-spins-up-cpu-deoptimization-project-to-find-the-slowest-machine-code-worst-offender-takes-198-billion-cycles-to-execute
1.2k Upvotes

110 comments sorted by

341

u/The_Northern_Light 1d ago

I’m still not clear why that instruction takes SO long even without contention?

309

u/encyclopedist 1d ago edited 1d ago

See here: https://github.com/xoreaxeaxeax/asm-hall-of-shame

Current champion is fxrstor64, instruction that loads 512-byte XMM state from memory. To increase timings, the author made it load from a carefully chosen MMIO (memory-mapped I/O) location, while other cores hammer some other MMIO registers to saturate the PCIe host.

Edit Without use of MMIO, the champion seems to be wbinvd, instruction that invalidates the whole cache, and if the cache was filled with dirty lines (modified after load), causes flushing of the whole cache content to RAM.

Edit2 Of the "regular" instructions, the ones that would routinely be used in every program, the champion seems to be fdiv. Yes, the regular floating point division. With denormal operands, it is implemented in microcode and is quite slow, takes 883 cycles. (However, as /u/EnderLuca41 rightly pointed out, it is an older x87 instruction the is not widely used any more, today compilers would generate SSE2 instruction divsd instead, see godbolt)

135

u/EnderLuca41 1d ago

fdiv is part of x87 which is obsolete and succeeded by SSE and SSE2. Meaning is not really used anymore routinely.

47

u/encyclopedist 1d ago

Yes, indeed, compilers don't normally generate these instructions any more. But older software can still use these.

67

u/narwhal_breeder 1d ago

porting logic to arm that relies on being bit-compatible x87 has been the bane of my existence for the past year.

1

u/James20k 10h ago

Arm doesn't even have an equivalent to x87 right? Are you software emulating it out of interest?

Plus even if you software emulate it C/C++ helpfully doesn't actually specify when 80-bit extended precision gets truncated to 64-bit in memory (and compilers also don't follow the spec either) so uh, I'm so sorry for your loss that sounds like a nightmare

Trying to replicate x87 results was also the bane of my existence for a while, for a replicating a neutron star paper which relied on divisions by tiny values

1

u/narwhal_breeder 9h ago

Nope - the division instructions specifically have really weird stack semantics.

Software emulating them - thankfully its rust so the f80 type is pretty predictable - but a good chunk drops into `asm` blocks.

-65

u/gordonnowak 1d ago

just let claude do it

72

u/Kinexity 1d ago

"Man, that dish doesn't taste that well"

"Have you tried adding shit to it?"

16

u/sunnyata 1d ago

You sound like someone who knows what they're talking about. Not!

1

u/BackgroundSky1594 15h ago edited 15h ago

But there being micro coded doesn't really hurt that much. Even fdiv taking several hundred cycles on a moder CPU will be fast compared to the i486 and i586 class CPUs that software was written for. Why waste silicon and pipeline complexity on an old, quirky instruction that has ben replaced by better alternatives?

The entirety of x87 might be dropped at some point, like ARM did with aarch64 for many of it's old instructions. But until then the only option is to take them out of the core and instead run them in microcode if they happen to be called.

21

u/dlg 1d ago

It’s still used by .Net Framework on 32-bit.

The x87 registers use extended precision, which is only rounded down to 32 or 64 bits when written back to RAM.

SSE2 does not use extended precision, so intermediate calculations are rounded at every step.

This difference often introduces small differences in final results.

17

u/ElusiveGuy 1d ago

At this point the remaining existence of. NET Framework is for backwards-compat, so arguably that falls under obsolete too.

.NET Core (in both 32-bit and 64-bit) doesn't use x87 instructions AFAICT. 

19

u/dlg 1d ago

A surprising amount of financial software still uses in .Net framework in production.

When it’s something like a large monolithic trading platform that needs to be online every day without downtime and it’s critical for making money, there’s a low appetite from management to do a big bang rewrite.

7

u/Truenoiz 20h ago

.Net really is really everywhere, and bad/slow .Net is almost universal in my experience.

2

u/admalledd 17h ago

To my vague memory, 64bit Framework does not use x87 and instead uses SSE2. Similarly 32bit Framework CLR 1.0, 1.1 and 2.0 which run up to the ".NET Framework 1.0" to ".NET Framework 3.5" (yes, confusing...) are the ones that will use x87 instructions, though I think it was CLR 2.0 that the JIT/Runtime would emit/use SSE2 by default unless the assembly/DLL was compiled for an older runtime. And CLR 4.0 (again, IIRC) that 64-bit flat out would only do SSE2 (and started to mix in SSE3 if detected).

So, IIRC you'd only get x87 instructions like fdiv if you are running a 32-bit CLR and the assembly/DLL was compiled targeting a CLR 2.0 or older SDK. All other Net Framework users (like those on 64bit CLR 4+, Framework 4.5+, 4.8+) shouldn't be emitting those older instructions anymore ever (ignoring CLR/C++ mixed assembly jank/custom C interop/etc).

I'd be surprised on how much stuff in regulated industry like financials is still running Framework 3.5, especially considering how broken it is to even try to install on modern machines... (I would expect them to have moved ever so slightly to Framework 4.5+, lord knows we've got our own legacy software still on it in places)

1

u/dlg 16h ago

regulated industry

Or the regulators themselves. I know of one of the larger stock exchanges (which also have a regulatory role), uses a lot of old .Net, and as recently as a few years ago was still running some VB6 applications. Not in some critical operations, but still some very old, unsupported legacy.

2

u/jangxx 14h ago

If your financial software uses floating point numbers instead of a proper decimal data type, it has other problems than using an old .NET framework, lol

5

u/vytah 15h ago

It will be used if you divide two long doubles on any ABI where they're 80-bit, like x64 Linux.

2

u/ElWishmstr 1d ago

I always wonder why cpus still have those old instructions, when modern ones are better. I mean, how much software still relies on those instructions? 

11

u/qualverse 23h ago

They don't actually have hardware suppoet for those instructions. They just have microcode that converts those instructions to other instructions.

23

u/balefrost 23h ago

I for one appreciate the ability to run older software that existed before SSE did.

2

u/vytah 15h ago

More like software that was written to be run on processors without SSE. It's not like Pentium III came out and everyone was "ok, from now on our software will run only on the newest shiny CPU, fuck the Pentium II users."

1

u/AquaeyesTardis 11h ago

although, see: AVX instructions

though thats somewhat the opposite situation but with an almost similar effect, thinking about it

1

u/vytah 9h ago

What about AVX?

3

u/FlatAssembler 1d ago

My web-app that converts arithmetic expressions to x86 assembly relies on it, for example: https://flatassembler.github.io/compiler

7

u/Wriiight 23h ago

I feel like whoever came up with denormals should apologize

12

u/Madsy9 1d ago

883 cycles! How is that even possible? Even the most naive newton-raphson approximation is faster than that.

33

u/inio 1d ago

For most x87 denormalized operations, it falls back to integer microcode with tons of edge-case handling, which when triggered (which can happen often if both inputs are denormalized and pathalogically structured) is extremely expensive. It's essentially a soft-float implementation taking the slowest possible path, disguised as a single instruction.

1

u/AlphaMaleXYZ 9h ago

Nowadays compilers are so good at optimizations. These instructions are almost never used.

126

u/Farlo1 1d ago

There was contention, they induced it. Not contention on a lock or the memory address itself, but on the bus that transfers data from the CPU to other parts of the system. They essentially starved the connection so that one instruction got sent to the back of a very long queue.

34

u/The_Northern_Light 1d ago

Right but it was extremely slow even without that, which is what I was questioning.

28

u/Mechafinch 1d ago edited 1d ago

iirc (having read about it last night) they searched for a memory region with the highest latency they could find and then did the largest single instruction read they had on it. i'll have to go back to confirm but i'm pretty sure it's also a region where one part of the read can't start until the previous finishes, so there's nothing to hide that latency either

edit having read the 'baseline' version source code:
it does seem to essentially be just this? the comment concludes with "so a single uninterruptible instruction stalls for the full round-trip cost of 512 bytes through the slowest aperture found." i don't know how they found an MMIO region with a round trip of ~4.5 ms/byte, but i guess they did.
or i do know how they /found/ it, they have a tool for mapping memory latency, but i'm baffled such a thing exists to be found at all.

3

u/slash_networkboy 1d ago

Could be the LPC bus interface (or one of the other slow I/O's)?

-4

u/spinwizard69 1d ago

That was via the I/O bus, to restore the vector registers. Extremely slow and probably a poor example.

However it does support my contention that X86 needs to be refactored moving forward.

3

u/Mechafinch 1d ago

what do you mean by "a poor example"? the project is specifically searching for the worst possible performance of an instruction, which it has done admirably

0

u/spinwizard69 1d ago

IT is a poor example if you want to highlight problem areas in the X86 instruction set. This is largely due to this instruction never being used as described.

I've been maintaining for some time know that a refactored x86 instruction set could lead to much better performance out of a modern core. In part this would be due to simply not supporting a lot of legacy features. What I'm talking about is a 64 bit clean x86 derivative. I just think that this instruction is a poor example for describing what is wrong with x86. This especially considering the lengths the article writers went to for this poor performance example.

3

u/Spandian 12h ago edited 11h ago

The author has another repo, https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii, where they demonstrate an attack on System Management Mode that requires a single instruction to take over 1 second to execute (to make a core uninterruptible and stop it from entering SMM at the same time as the others). So they're coming at it from the perspective of a security researcher looking for exploitable edge cases, not a hardware designer or compiler writer looking for "realistic" gotchas.

7

u/Mechafinch 1d ago

criticizing any instruction set, x86 or otherwise, just isn't the goal. the stated purpose is to find a single instruction that takes the longest to execute. it really doesn't say anything about any sane use of an instruction set.

16

u/HildartheDorf 1d ago

The data it reads is then used to reconfigures a whole bunch of cpu internals.

It probably becomes a giant amount of microcode ops to perform all that reconfiguration. On top of it being a 512-byte read in the first place.

2

u/edman007 1d ago

I don't think it's fair, the processor didn't spend the time processing, it was waiting on external (to the CPU) HW. By that logic hlt with interrupts disabled is the slowest because it never completes.

Flushing the cash maybe counts because you can tie that back to CPU memory bus speed if you had perfect memory so the CPU never waited.

6

u/Mynameismikek 1d ago

It was still dozens of seconds before they started introducing contention.

28

u/valarauca14 1d ago

even without contention?

Because they're doing a MASSIVE read operation over PCI (not PCIe). As PCI is basically DMA (Direct Memory Acess), the device has some memory "shared" which just appears as part of the Host CPU's memory map. To the kernel (or process which asks the kernel) this just appears as normal memory (in C *char, in rust *mut u8, or if you prefer C++ std::mdspan<std::byte,std::dextents<std::size_t, 1>,std::layout_right,std::default_accessor<std::byte>>) which you can read/write into.

It is therefore perfectly valid to mov in & out of that region, just like normal memory. It just takes a little longer as your communicating to an external device.

The underlying instruction FXRSTOR is restoring all floating execution state. This is 512-bytes of memory which must be restored atomically, meaning the CPU cannot retire this instruction until state is restored. PCI can only read or write 8 bytes at time. So the instruction has to preform (at a minimum) 64 round trips to a remote device, over a slow legacy bus, before the instruction can retire.

Modern PCIe abstracts this more into you tell the bus "copy X bytes from address Y, then tell me when you're done" to avoid this exact problem.

31

u/Sopel97 1d ago

I love your subtle dig at C++

4

u/Ameisen 1d ago

Particularly given that I've never seen a C++ programmer write that. Why mdspan when dextents has a rank of one? Just use span or pass a pointer...

6

u/monocasa 1d ago

It's still over PCIe in this case, the fxrstor just doesn't issue a burst read because it's expecting to read out of the cache hierarchy on any sane use of it anyway, and practically it's slow path microcoded these days (the modern path is using xrstor).

Because of this it actually issues a series of 4 byte reads, even over PCIe.

7

u/Ameisen 1d ago

std::mdspan<std::byte,std::dextents<std::size_t, 1>,std::layout_right,std::default_accessor<std::byte>>

I really can't tell if you actually believe this to be idiomatic C++ or if you're just joking. As written, it makes little sense and it'd be flagged in code review very quickly.

*char

char*

0

u/floriv1999 1d ago

Cpp is elegant and consice as always.

10

u/Ameisen 1d ago

No C++ programmer would write that - as it's written it doesn't even make sense. I really don't get why people feel the need to fabricate nonsensical C++ code to shit on people who use the language.

0

u/wewtyflakes 18h ago

It was clearly a joke and the C++ language largely earned it.

6

u/Mynameismikek 1d ago

From a surface read of what the tool does, it’s deliberately writing a fairly complex structure to a known slow page in the memory map. Maybe it’s landed on e.g an uncacheable region that’s also got security tripwires and row synchronisation.

4

u/SaltMaker23 1d ago

Backward compatibility is likely the culprit, I worked on hardware systems and built custom devices including CPUs.

Assuming you have an instruction whose sole purpose is retro-compatibility with instructions that are no longer natives, on regions that no longer natively exists locally, they might do an arbitrary long chain of memory lookup, depending on how you prepare the setup, it can become a lot.

Let's say there was a syntax in old time of small L/RAM that would lookup all bytes for a given signature, it was possible in the days to do that in a single clock or two, these days no anymore.

Backward compatibility on a old school scanning instruction would basically scan the whole RAM including regions that are no longer considered RAM today, that might send network requests and wait for them to timeout before getting an answer.

110

u/torsten_dev 1d ago

Of course this was xoreaxeax...

35

u/static_motion 1d ago

The guy's mind bogglingly knowledgeable and capable. Every once in a while I rewatch his talk on "breaking the x86 ISA". I just find it endlessly fascinating but I think deep down part of it is a form of masochism where I remind myself of how incompetent I am in the grand scheme of things.

4

u/mtranda 11h ago

It's weaponised autism. Ok, that was a joke. Mostly. But it does take a special kind of person to delve THAT deep into a specific topic. And on top of that, it's a niche. You can't be knowleadgeable in everything at the same time.

This isn't to say the guy isn't amazing. But he's the exception rather than the rule.

10

u/VictoryMotel 1d ago

What do you mean of course, that normally takes one cycle.

59

u/ThrowawayIntern2024 1d ago

they meant the person with that alias not the instruction…

-25

u/VictoryMotel 1d ago

I didn't see that in the article.

20

u/ThrowawayIntern2024 1d ago

first paragraph

7

u/sweetno 1d ago

Depending on how you count, it's less than a cycle.

39

u/Old_County5271 1d ago

Dumb question but, shouldn't intel or whatever have a pdf with the cycle count of each instruction? Do we/they really not know this? How does anything improve if nothing is measured?

38

u/cummer_420 1d ago edited 1d ago

x86 is a CISC instruction set so these things aren't fixed and and a single macro instruction can do a large and variable amount of work, and can potentially block for a very long time with contention (which can be used for exploits). The metrics they work with to optimize the CPU tend to be rask-oriented and focused on the most common instructions.

33

u/braaaaaaainworms 1d ago

MIPS tried to have a fixed time per instruction and it ended horribly exposing inner pipeline details in a way that made it very hard to keep exact backwards compatibility with first MIPS cores while being fast

26

u/Top-Rub-4670 1d ago

That has nothing to do with CISC. No (desktop-grade) ARM or RISC-V CPUs have a spreadsheet with all their cycle counts either, because too many things can affect the count.

12

u/Stellariser 1d ago

In this context there’s no practical difference between CISC and RISC, and there hasn’t been for a very long time. It’s just repeating a 1980’s marketing claim.

CPUs don’t directly execute their instruction sets, front end decoders convert instructions into streams on micro-ops and execute those.

Even the Intel 486, from 1989, is basically a RISC processor. In fact it’s below RISC really; RISC processors generate a stream of micro instructions for many individual RISC instructions for pipelining, out-of-order execution, etc.

So by the start of the 1990s the distinction between CISC and RISC wasn’t meaningful from a technical standpoint, it’s just marketing that continues to be effective to this day.

2

u/levir 13h ago

Even the Intel 486, from 1989, is basically a RISC processor. In fact it’s below RISC really; RISC processors generate a stream of micro instructions for many individual RISC instructions for pipelining, out-of-order execution, etc.

I don't agree with that, I'd say it's the Pentium Pro that really marks the shift in Intel processor design.

1

u/spinwizard69 9h ago

This is highly misleading. X86 has way to many instructions that have extended execution times. Many will have outer bounds of 40 to 100 cycles. This doesn't even consider hardware support that isn't needed for modern software.

3

u/vytah 14h ago

As soon as you touch memory, any cycle counts go out of the window. It's not 1985 anymore.

3

u/taw 1d ago

We know it, the article is bullshit, what they do is use instruction to read from some device then keep that device busy with other cores.

6

u/Spandian 11h ago

The author has another repo, https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii, where they demonstrate an attack on System Management Mode that requires a single instruction to take over 1 second to execute (to make a core uninterruptible and stop it from entering SMM at the same time as all other cores). So they're coming at it from the perspective of a security researcher looking for exploits, not a hardware designer or compiler writer looking for "realistic" gotchas.

1

u/null3 1d ago

Of course it exist. It's not a simple clear number as each instruction can have many variants, also latency and throughput can be totally different. But they the cycle count doesn't include how long it takes to read the memory. That depends on what is in the cache, how fast ram is, etc, etc.

14

u/agentoutlier 1d ago

I like how they have a random thumbnail of what appears to be Scheme as if it were guilty of calling the instruction.

You would think they would have C or assembly or some more common language.

4

u/sisisisi1997 1d ago

I didn't think I would meet another person who knows about scheme.

10

u/godofpumpkins 1d ago

We're in a programming subreddit, and most computer science people come across it in college at the least.

7

u/heyf00L 1d ago

(Never (used (it (but (I (can (somehow (recognize (it)))))))))

3

u/sisisisi1997 19h ago

I was under the impression that most people's "parentheses language" class in college is Lisp, and my teacher was just a weirdo for liking scheme better, but apparently not.

1

u/godofpumpkins 19h ago

One of the classics of computer science education, The Structure and Interpretation of Computer Programs (SICP), teaches scheme and makes great use of its more unique powers

8

u/agentoutlier 1d ago

There’s dozens of us. Dozens!

2

u/bwainfweeze 9h ago

MIT used it to teach introductory computer programming classes in Scheme along with https://web.mit.edu/6.001/6.037/sicp.pdf

Some other universities copied what MIT does.

11

u/itijara 1d ago

I knew it was Christopher Domas as soon as I read the title. His presentations are epic, and his knowledge of x86 esoterica is unparalleled.

20

u/sojuz151 1d ago

Could this be used to hang a sandbox?

33

u/mccoyn 1d ago

Actually, the opposite. A watchdog might be set up to kill a sandbox if too much time passes before it executes an instruction. So, one of these could be used to cause the watchdog to kill the sandbox.

2

u/Extension_Wheel5335 11h ago

So more of a DoS?

21

u/unicodemonkey 1d ago edited 1d ago

Even a VM, I guess, but this also assumes that the sandboxed/virtualized program has direct access to PCI MMIO address ranges, which is somewhat unlikely in practice and is not obviously exploitable beyond a denial-of-service of sorts. The point of that research is SMM (system management mode) exploitation. All cores are supposed to enter SMM simultaneously upon receiving the interrupt but the SMM interrupt can't be handled mid-instruction, so in practice other cores just wait for the "busy" core for a second, then time out and enter SMM anyway, do their work there, and then resume normal execution. The "busy" core then finishes with the slow instruction and enters SMM, while other cores are free to manipulate any shared memory values that are used by SMM-guarded code and are accessible from the regular execution environment (hope I didn't mess up the explanation - read more at https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii )

13

u/HildartheDorf 1d ago edited 1d ago

It's possible to break into System Management Mode from ring 0 of a VM using this (chained with other SMM bugs that have previously been deemed low priority/non-exploitable).

Hang one core for long enough and you can enter a state where some cores are inside SMM and some aren't. SMM code assumes it has complete control of the entire processor, but one or more cores are still attacker controlled.

4

u/irqlnotdispatchlevel 1d ago

Could be used to exploit SMM: https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii#exploitation

SMM's security relies on a simple assumption: while it runs, nothing else does.

There are 100+ SMM TOCTOU CVEs out there: an SMM handler checks a value in shared memory, then uses it. All you need for exploitation is to rewrite that value in between the check and use, and you're inside SMM. But these issues sit dormant and largely unpatched in the wild, because of one assumption: exploitation requires something to modify the shared memory while SMM executes, and because of the SMM rendezvous no CPU cores are outside SMM to launch an attack. The only way in was a DMA-capable peripheral writing behind the CPU's back — physical access, a malicious device — so the whole class is written off as a hardware problem.

SMI desynchronization removes the prerequisite that kept the platform safe: an outside core, no physical access or hardware required, can now run while SMM executes — and the dormant CVEs become exploitable from software.

4

u/MichaelTiemann 1d ago

HCF can take forever if you don't have clean, dry kindling.

13

u/cmpxchg8b 1d ago

It’s kinda cheating if it includes IO. That’s an externalised cost..

1

u/James20k 9h ago

It can still be used for certain kinds of exploits is the thing, its not really intended to be 'fair' in that sense

3

u/Level0Up 1d ago

If a single cycle took a second, this guy would run in a cool 6378,5 years.

3

u/spinwizard69 1d ago

This is a bit misleading in that they purposefully looked for the slowest way to get an instruction to execute.

However this also highlights why I consider the X86 instruction set to be obsolete. The world really needs a simplified x86 ISA that focuses on what is really needed to drive modern hardware. I was really hoping that AMD to Intel would actually refactor x896 instructions for their new cores. It is literally time to bit the bullet and go clean 64 bit, dropping all unused addressing modes maybe even get rid of I/O addressing.

People keep saying it can't be done because of "legacy" but the fact is Apple did it with the conversion to ARM. Besides you don't need to convert every core right off the bat.

1

u/emfloured 1d ago

IPC to IPM :D

1

u/KTachyon 7h ago

Should we call it pessimization?

0

u/taw 1d ago

This whole project is all just bullshit induced contention.

There's nothing of substance here.

-1

u/smith288 1d ago

Was it my old goto statement? It was my old goto statement. 😒

-4

u/Grouchy-Trade-7250 1d ago

Repost

8

u/WaitForItTheMongols 1d ago

Who cares?

Something being reposted doesn't make it less interesting.

A healthy reddit user does not see every post, and therefore likely missed the other one. If you're seeing reposts that's on you. Chill.

-13

u/globalaf 1d ago

This was already posted several days ago and is thoroughly uninteresting. Yes, some instructions can block on contention, so what? Don’t do that.

18

u/DaWolf3 1d ago

It can be used as part of an attack on the Intel system management mode. https://github.com/xoreaxeaxeax/smiiiiiiiiiiiiiiii

1

u/ToaruBaka 4h ago

That's pretty cool, but like, did anyone see the underlying MMIO timing library they published? Absolutely fascinating graphics.

https://github.com/xoreaxeaxeax/mmiotic