OpenAI traces ChatGPT search crashes to a bad Azure host and an 18-year-old libunwind race

Two crashes wearing the same costume

The subject is Rockset, the cloud-native search and analytics system OpenAI acquired in 2024 and now uses to keep an up-to-date index of a workspace's knowledge base for ChatGPT. Its execution layer is written in C++, and a few months before the post it began throwing crashes that looked physically impossible: a normal function would appear to finish and then return to a bogus address, sometimes a NULL return-address slot, sometimes with the stack pointer %rsp off by exactly 8 bytes.

Neither symptom fits ordinary application code. A stray write that lands only on a saved return address is extremely unlikely, and misaligning %rsp by 8 without inline assembly, setcontext, or longjmp — none of which Rockset uses — is stranger still, because compiled code only touches that register in a function's prologue and epilogue. Every hypothesis the team or ChatGPT proposed had strong evidence against it. The reason, it turned out, was that they were staring at two unrelated bugs at once: silent hardware corruption on a single Azure host, and an 18-year-old race condition in GNU libunwind.

Doctor versus epidemiologist

The team's first instinct was clinical: inspect a handful of core dumps closely, form hypotheses, rule them out one by one. Most crashes landed in a single heavily-inlined method, DocumentTree::updateDocument, which left an overwhelming set of candidate functions. Log queries produced both false positives and false negatives because stack-corruption bugs corrupt the very stack traces used to classify them. Manual inspection didn't scale. Worse, the team had already ruled out a hardware cause because crashes appeared across multiple regions and hardware types — a conclusion that was exactly backward for one of the two populations.

The turning point was a shift in method. Instead of diagnosing one patient, they treated the whole fleet as a population. They had ChatGPT write a script that downloaded a prefix of each core file, extracted registers, filtered known false positives against the logs, and labeled each crash as return-to-null, misaligned-stack, or other. Running it across every production Rockset core dump from the previous year made the correlations appear instantly: return-to-null crashes were spread across many clusters with no clean start date, while misaligned-stack crashes came from one region, had a clear start date, and never occurred on long-running nodes — the signature of one physical machine with bad hardware.

A hundred-picosecond window, and the Fermi estimate that made it credible

The bad host was denylisted, and its misaligned-stack crashes disappeared even though the team could never reproduce the register corruption in weeks of stress testing. With that cluster removed, the remaining return-to-null cores were all happening during C++ exception unwinding. GNU libunwind synthesizes a ucontext_t on the stack and hands it to an internal routine, _Ux86_64_setcontext, whose final instructions move %rsp to the new stack bottom before a later instruction reads the destination instruction pointer from that same struct. Once %rsp moves, the struct is no longer protected by the red zone, so a signal delivered in the gap lets the kernel build its signal frame over that memory and zero the restored instruction pointer.

The vulnerable window is a single instruction — roughly a hundred picoseconds on a modern out-of-order CPU — which seemed far too narrow to explain more than a dozen return-to-null crashes per day. A Fermi estimate closed the gap: a 10⁻¹⁰-second window against a SIGUSR2 arriving every 10⁻² seconds of CPU time gives each cleanup handler about a 10⁻⁸ chance of losing the race; a host throwing 10⁴ exceptions per second for ingest backpressure then fails on the order of once every few hours, which at fleet scale is plenty.

Rockset was unusual on all three axes that drive the crash rate. It throws exceptions frequently as overload control, it delivers SIGUSR2 unusually often via its coarse_thread_cputime_clock, and earlier in the year it had added a timer_getoverrun call that made the signal handler consume more stack — enough to reach and clobber the stale ucontext_t. Before that change, the crashes did not appear at all. The 18-year-old bug had always been present; the product of exception rate, signal rate, and handler stack usage had only just crossed the threshold where it became visible.

Instrumentation as the actual fix

The immediate mitigation was to switch from GNU libunwind to libgcc's unwinder — independently a good trade, since libgcc has had more work to reduce lock contention on large VMs — and OpenAI upstreamed a self-contained reproducer and fix to GNU libunwind. For the hardware failure, the team improved the fatal signal handler to log register state so recurrence can be detected from logs alone without a core dump, and changed the control plane to reuse VMs rather than recycle them, which makes bad-node detection easier.

The load-bearing lesson is not any of the ABI, DWARF, or exception-machinery detail the investigation uncovered. It is that the confusion persisted only because two phenomena were mixed into one story, and it dissolved the moment the data set became clean and complete. That is the point worth carrying into other infrastructure teams: when a bug looks impossible, the missing piece is often accurate population data, not a smarter single-case diagnosis.

Reliability is not just about fixing bugs after they happen—it's about building the data, workflows, and skills that turn impossible problems into diagnosable and solvable ones.Montana Labs