I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.
Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.
>Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault shows the backing is the kernel's zero page. A captured core dump confirms the crash site with matched virtual addresses. The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction, and a git comparison across v6.19/v7.0/v7.1/mainline identifies the v7.0-introduced change on the munmap-teardown side that widens it.
"a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function."
-> ??? What's a "backing" of a page? Freshly-faulted? Fresh fries?
"~10 instructions later"?
"A pagemap read at the instant of the fault shows the backing is the kernel's zero page."
-> Backing?
"The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown."
-> How can a mechanism "localize"?
And more.
This is... words strung together. Nothing more. I wonder how people read and make sense of this.
I'm becoming increasingly allergic to the performative writing of LLMs.
The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to "reason" and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.
fwiw "backing" is standard jargon in this context, and "freshly faulted" is phrasing you'll see in Linux source comments. The writeup as a whole is indeed slop, though.
The overflow would still overflow; the use-after-free would still use after free; a musl mask race would still race.
Hilarious. Apart from the computer poetry, the conclusion seems to be “it’s something in Linux 7.0 + musl 1.2.5”, although the only reproduction is still on the same physical Threadripper CPU and only sometimes when heavily exercised, so it hasn’t really ruled out a hardware issue.
The worst thing is that I know there's something interesting in there but I'm too arsed to take the time and decode what data the author must have used to generate that text.
Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be that somehow a zero-page PTE was present when it shouldn’t have been.
This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs.
Also, I don’t think the zero page has pfn zero.
If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.
The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)).
I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent.
Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
After more digging, I bet it's a paging-structure-cache flush bug. I've debugged these before, and they're nasty, hardware dependent, hard-to-reproduce issues. Looks like the code that the AI flagged might actually be wrong, but not for the reason that the AI thought.
> Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
Which raises the question of whether any HN readers have successfully reproduced this?
I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
I have never, in my entire personal history of kernel development, successfully reproduced a paging structure cache bug. I have stared at these sorts of bugs and puzzled them out.
With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
Yeah but the thing is that someone tries to waste time of humans.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
In the history of Linux that I'm aware of (and I've been nose-deep in this stuff on and off for quite a while), these issues are almost never debugged because they're reproduced -- they're debugged because someone who actually understands the messy interactions of the code and hardware in question thinks about them.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned as such around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints.
You got none of that here. It’s just realms of text.
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
> From my point of view as long as the information is accurate
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
What? it is mostly literal nonsense. Moreover the last paragraph where they say it only reproduced on one machine just does not justify the pseudo-deep analysis in the whole document.
This is a bit sad. Here we have an example of rust and musl struggling whereas C/C++ and glibc does not. That's an oversimplification, but we also had not long ago the rewrite of ... I believe it was coreutils, in rust (or was it utillinux ... but I think it was coreutils), also have issues. Rust needs to toughen up here when it really wants to replace all of C in the linux ecosystem. Alternatively it could admit failure, then people can say that C will be - and remain - the forever king.
This bug is almost certainly not a bug in ripgrep or Rust or musl or any user code at all. Maybe one could complain that musl is using an inferior allocator that is freeing then immediately reallocating the same address.
In addition to this likely not being a bug in the code you're pointing at, but the kernel (written in C). musl is literally C code, and that most rust users compile against glibc just like most C users do.
I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
> I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient).
ripgrep actually sets jemalloc as global allocator when built for 64b musl: https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab...
It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.
Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.
mallocng should not be used at all.
The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis.
I tried reading and gave up at the "Headline"...
Quoting from the bug analysis:
>Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault shows the backing is the kernel's zero page. A captured core dump confirms the crash site with matched virtual addresses. The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction, and a git comparison across v6.19/v7.0/v7.1/mainline identifies the v7.0-introduced change on the munmap-teardown side that widens it.
"a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function." -> ??? What's a "backing" of a page? Freshly-faulted? Fresh fries? "~10 instructions later"?
"A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?
"The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown." -> How can a mechanism "localize"?
And more.
This is... words strung together. Nothing more. I wonder how people read and make sense of this.
In case people have forgotten what real technical writing looks like, here's a sample (I am not the author): https://yifan.lu/2019/01/11/the-first-f00d-exploit/
I'm becoming increasingly allergic to the performative writing of LLMs.
The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to "reason" and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.
Agreed, the article you linked is a great example of clear, understandable, detailed technical writing.
fwiw "backing" is standard jargon in this context, and "freshly faulted" is phrasing you'll see in Linux source comments. The writeup as a whole is indeed slop, though.
Reading an AI-generated bug report is awful.
Stopped after
> Headline. The crash is real and reproducible.
I really hated that kind of language, feels like speaking to a motivator
anyway it's slop, can sense it even before i started reading
The worst thing is that I know there's something interesting in there but I'm too arsed to take the time and decode what data the author must have used to generate that text.
Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be that somehow a zero-page PTE was present when it shouldn’t have been.
This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs.
Also, I don’t think the zero page has pfn zero.
If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.
The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)).
I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent.
Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
After more digging, I bet it's a paging-structure-cache flush bug. I've debugged these before, and they're nasty, hardware dependent, hard-to-reproduce issues. Looks like the code that the AI flagged might actually be wrong, but not for the reason that the AI thought.
https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
> Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
Which raises the question of whether any HN readers have successfully reproduced this?
I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
I have never, in my entire personal history of kernel development, successfully reproduced a paging structure cache bug. I have stared at these sorts of bugs and puzzled them out.
With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
The explanation is obviously written by an AI agent.
soulless unreadable AI slop
I can read it but not sure if worth spending energy on it.
It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
> It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
Great way to summarize cultural "economics"
Couldn't put the words on this pattern but sometimes all I care about is that someone cared about.
energy might be the wrong metric, plenty of energy was wasted spinning up that LLM.
Yeah but the thing is that someone tries to waste time of humans.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
PoC||GTFO, if it reproduces, it's valuable
Then post whatever notes were fed into the AI instead. The verbosity and self-congratulating add negative value.
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
That I agree with (not instead, in addition)
In the history of Linux that I'm aware of (and I've been nose-deep in this stuff on and off for quite a while), these issues are almost never debugged because they're reproduced -- they're debugged because someone who actually understands the messy interactions of the code and hardware in question thinks about them.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
They have a PoC, we're arguing about the LLM-generated slop explanation of the PoC.
Is it actually reproducible though and do I want to chase and reproduce something that had no human touch? Do you?
Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned as such around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
[1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints.
You got none of that here. It’s just realms of text.
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
> Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it?
Then maybe that person should not do it? At least until they find the time?
> From my point of view as long as the information is accurate
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
Then where’s the lie in “soulless AI slop”?
I had the opposite reaction. Clear, detailed, well-organized. Pretty close to the ideal writeup.
Could you explain it in your own words? (I tried myself, and quickly discovered that it is incoherent)
Perhaps you are talking about the ripgrep issue itself, which is fine. The others are talking about the linked analysis, which is not.
Hi Claude! :wave:
Really? It's chock full of poorly written and irrelevant chain-of-thought rambling.
What? it is mostly literal nonsense. Moreover the last paragraph where they say it only reproduced on one machine just does not justify the pseudo-deep analysis in the whole document.
So, why the bug triggers only with muslc and not other libc ?
No wonder search in codex is so a$$
wow, everything is broken lol
This is a bit sad. Here we have an example of rust and musl struggling whereas C/C++ and glibc does not. That's an oversimplification, but we also had not long ago the rewrite of ... I believe it was coreutils, in rust (or was it utillinux ... but I think it was coreutils), also have issues. Rust needs to toughen up here when it really wants to replace all of C in the linux ecosystem. Alternatively it could admit failure, then people can say that C will be - and remain - the forever king.
This bug is almost certainly not a bug in ripgrep or Rust or musl or any user code at all. Maybe one could complain that musl is using an inferior allocator that is freeing then immediately reallocating the same address.
Thank you for writing virtually all of the intelligent competent comments on this page.
In addition to this likely not being a bug in the code you're pointing at, but the kernel (written in C). musl is literally C code, and that most rust users compile against glibc just like most C users do.
A kernel bug and you’re criticizing Rust. Weird take