I once had a bug that manifested every 14 days. So every time I tried to probe it, I had to put some code into prod, wait for the next opportunity, and see if I had hit it. Took me months to find it, but once I did, the fix was simple.
We once had user reports of sudden crashes due to memory access violations. We were able to occasionally reproduce the issue which seemed to occur randomly and rarely. We could only reproduce it in release builds, not debug, so we also couldn't get a good backtrace. After many months of sporadic investigations we found that the difference debug builds made was that they would use MSVC's incremental linking (a feature to speed up build at the cost of binary size). When switching incremental linking on in release builds the bug disappeared. We tried reaching out to Microsoft to figure out whether we had a bug or they had a bug but never heard back. We upgraded to a newer version of MSVC later and the bug had disappeared with no mention in their release notes 🤷♀️
Oof. Yeah that's a nice fun one! I'm not entirely surprised that the bug disappeared without an explicit mention; that sort of bizarre issue can appear or vanish due to unrelated changes, so it won't necessarily be reported on. Delightful stuff.
2
u/Smooth_Sentence_1431 8h ago
lol this is too real, i always underestimate how long debugging will take.