Mysterious memory corruption derails builds — is it hardware, the kernel, or something stranger?

The failures

It has been reported that a developer who upgraded to GCC's master branch began seeing a string of intermittent build crashes across unrelated projects — LLVM, protobuf, OpenJDK, Qt WebEngine and others. The crashes shared a pattern: SIGSEGVs emerging from GCC's ggc garbage-collector routines while walking C++ AST nodes. Retry the build and—poof—sometimes it works. Compiler bug? Library mismatch? Hardware gremlin? Welcome to the debugging funhouse.

The hunt

He tried the obvious tricks: reduce inputs, force more frequent garbage collection with ggc params, rebuild many packages to gather more failure traces. No clear reproducible compiler trigger showed up. It has been reported that long runs of memtest86+ and memtest86 found nothing — hours of testing, zero errors. So the trail went cold. Or did it?

The breakthrough (and the plot twist)

Then memtester — a user-space memory tester — allegedly detected a single-bit flip (0x72 → 0x52). That small, lonely change reignited hope: maybe it wasn't a fresh GCC regression after all. Subsequent memtester runs were clean, which was maddening. So the developer flipped the kernel's page-poisoning and page-owner checks on (page_poison=on, page_owner=on) to catch freelist misuse. It has been reported that this produced a reliable reproduction — the same single-bit corruption showing up at a specific physical page frame number (pfn 0x75d9b4) and the same byte pattern. Suddenly the bug wasn't a ghost.

What it means — and what's next

If true, this points away from a neat compiler bug and more toward memory corruption that manifests at the kernel or hardware level. Single-bit flips can be the signature of flaky RAM, bad memory controllers, or subtle kernel/driver use-after-free behavior. So what now? Isolate the DIMM, swap hardware, run long-term memtester runs and kernel debug builds, and gather traces for upstream bug reports. It has been reported that the developer is continuing to chase the root cause — because when software and silicon start playing hide-and-seek, the only cure is patience, persistence, and a healthy appetite for sleuthing.

Sources: trofi.github.io, Lobsters