Maybe you can have a look at RadDbg [0], as I understand the project has been bought by Epic Games recently. The goal is to make a performant debugger, from what I understand its target audience is mostly game developers, but you can use it for other programs obviously. You can see a talk of the core developer and his vision here: https://www.youtube.com/watch?v=_9_bK_WjuYY
Sadly it's windows only yet, but they have plans to port it to other platforms.
The readme talks about plans for Linux support, but I'm guessing that's no longer on the table after the Epic acquisition? Sweeney is the single most publicly anti-Linux CEO I'm aware of.
Yes actually: plenty of companies don't care where the money comes from, they're happy as long as there's money. Unity, the other big ad- and IAP-peddling game engine company, has pretty good Linux support.
It's weird for a company to explicitly say, "if you use this one operating system you can go F yourself, we don't want your money". (Note: this is not the same as saying "we only officially support Windows at this time, sorry". There's seething hatred in Sweeney's words.)
My background is in teaching C programming at the university level and DDD was very helpful there, although not very comfortable to use. For years, I've looked for a replacement and finally found Seer and was very happy with it.
Interactive debugging is definitely useful when teaching but obviously teaching is a different context. But Seer is not an educational tool and I believe it will hold up in other cases as well.
Yes, that’s a good example — thanks for the link.
Tools like this seem very strong at visualizing and exploring state, but they still tend to stay fairly close to the traditional “pause and inspect” model.
What I keep struggling with is understanding how a particular state came to be — especially with concurrency or events that happened much earlier.
That gap between state visualization and causality feels hard to bridge, and I’m not sure what the right abstraction should be yet.
This doesn't sound like a particularly difficult problem for some scenarios.
It's definitely convoluted as it comes to memory obtained from the stack, but for heap allocations, a debugger could trace the returns of the allocator APIs, use that as a beginning point of some data's lifetime, and then trace any access to that address, and then gather the high-level info on the address of the reader/writer.
Global variables should also be trivial (fairly so) as you'll just need to track memory accesses to their address.
(Of course, further work is required to actually apply this.)
For variables on the stack, or registers, though, you'll possibly need heuristics which account for reusage of memory/variables, and maybe maintain a strong association with the thread this is happening in (for both the thread's allocated stack and the thread context), etc.
Sounds like you want a time travel debugger, eg. rr.
Sophisticated live debuggers are great when you can use them but you have to be able to reproduce the bug under the debugger. Particularly in distributed systems, the hardest bugs aren't reproducible at all and there are multiple levels of difficulty below that before you get to ones that can be reliably reproduced under a live debugger, which are usually relatively easy. Not being able to use your most powerful tools on your hardest problems rather reduces their value. (Time travel debuggers do record/replay, which expands the set of problems you can use them on, but you still need to get the behaviour to happen while it's being recorded.)
Sounds like you want time travel debugging [1]. Then you can just run forwards and backwards as needed and look at the full evolution of state and causality. You usually want to use a integrated history visualization tool to make the most of that since the amount of state you are looking at is truly immense; identifying the single wrong store 17 billion instructions ago can be a pain without it.
It's for embedded systems though, which is where I come from. In embedded we have this concept called instruction trace where every instruction executed with the target gets sent over to the host. The host can reconstruct part of what's been going on in the target system. But there's usually so much data, I've always assumed a live view is kind of impractical and only used it for offline debugging. But maybe that's not a correct assumption. I would love to see better observability in embedded systems.
Maybe you can have a look at RadDbg [0], as I understand the project has been bought by Epic Games recently. The goal is to make a performant debugger, from what I understand its target audience is mostly game developers, but you can use it for other programs obviously. You can see a talk of the core developer and his vision here: https://www.youtube.com/watch?v=_9_bK_WjuYY
Sadly it's windows only yet, but they have plans to port it to other platforms.
- [0]: https://github.com/EpicGamesExt/raddebugger
The readme talks about plans for Linux support, but I'm guessing that's no longer on the table after the Epic acquisition? Sweeney is the single most publicly anti-Linux CEO I'm aware of.
Do you expect an IAP peddler to support free software?
Yes actually: plenty of companies don't care where the money comes from, they're happy as long as there's money. Unity, the other big ad- and IAP-peddling game engine company, has pretty good Linux support.
It's weird for a company to explicitly say, "if you use this one operating system you can go F yourself, we don't want your money". (Note: this is not the same as saying "we only officially support Windows at this time, sorry". There's seething hatred in Sweeney's words.)
I suppose Apple bribed him to support Macs. Who's going to bribe him to support Linux, RedHat? Canonical?
May I recommend this episode of The Stand Up podcast with Ryan Fleury as guest, who is the driving force behind the rad debugger.
https://www.youtube.com/watch?v=O-3gEsfEm0g
Casey also makes a good point here on why printf-debugging is still extremely popular.
Pernosco
Blows everything else out of the water.
https://pernos.co/ ( I’m not affiliated to them in any way, just a happy customer)
Seems like IDA Pro does the same but 10x better
What are you using as a Rust debugger?
printf("Got here, x=%u"\n", x);
“The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.”
- Brian Kernighan
Although that was true at the time, it was before the creation of modern omniscient debuggers like Pernosco (<https://pernos.co/>).
My background is in teaching C programming at the university level and DDD was very helpful there, although not very comfortable to use. For years, I've looked for a replacement and finally found Seer and was very happy with it.
https://github.com/epasveer/seer
Interactive debugging is definitely useful when teaching but obviously teaching is a different context. But Seer is not an educational tool and I believe it will hold up in other cases as well.
Have you also tried KDbg, and if so, what's the reason for picking seer over KDbg?
I haven’t touched in a while, but Visual Studio’s (standalone, not code) debugger was very cool.
Also rr is impressive in theory, although it never worked on codebases that I worked on.
Something like this maybe:
https://whitebox.systems/
Doesn't seem to meet all your desired features though.
Yes, that’s a good example — thanks for the link. Tools like this seem very strong at visualizing and exploring state, but they still tend to stay fairly close to the traditional “pause and inspect” model. What I keep struggling with is understanding how a particular state came to be — especially with concurrency or events that happened much earlier. That gap between state visualization and causality feels hard to bridge, and I’m not sure what the right abstraction should be yet.
This doesn't sound like a particularly difficult problem for some scenarios.
It's definitely convoluted as it comes to memory obtained from the stack, but for heap allocations, a debugger could trace the returns of the allocator APIs, use that as a beginning point of some data's lifetime, and then trace any access to that address, and then gather the high-level info on the address of the reader/writer.
Global variables should also be trivial (fairly so) as you'll just need to track memory accesses to their address.
(Of course, further work is required to actually apply this.)
For variables on the stack, or registers, though, you'll possibly need heuristics which account for reusage of memory/variables, and maybe maintain a strong association with the thread this is happening in (for both the thread's allocated stack and the thread context), etc.
Sounds like you want a time travel debugger, eg. rr.
Sophisticated live debuggers are great when you can use them but you have to be able to reproduce the bug under the debugger. Particularly in distributed systems, the hardest bugs aren't reproducible at all and there are multiple levels of difficulty below that before you get to ones that can be reliably reproduced under a live debugger, which are usually relatively easy. Not being able to use your most powerful tools on your hardest problems rather reduces their value. (Time travel debuggers do record/replay, which expands the set of problems you can use them on, but you still need to get the behaviour to happen while it's being recorded.)
Sounds like you want time travel debugging [1]. Then you can just run forwards and backwards as needed and look at the full evolution of state and causality. You usually want to use a integrated history visualization tool to make the most of that since the amount of state you are looking at is truly immense; identifying the single wrong store 17 billion instructions ago can be a pain without it.
[1] https://en.wikipedia.org/wiki/Time_travel_debugging
Here's another one
https://scrutinydebugger.com/
It's for embedded systems though, which is where I come from. In embedded we have this concept called instruction trace where every instruction executed with the target gets sent over to the host. The host can reconstruct part of what's been going on in the target system. But there's usually so much data, I've always assumed a live view is kind of impractical and only used it for offline debugging. But maybe that's not a correct assumption. I would love to see better observability in embedded systems.
Modern equivalent, working on Linux with remote connection from Visual Studio Code is LLDB.
Takes some effort to configure it but beats "printf" (i.e. logging) in the end.
Check out Binary Ninja if you haven’t. Especially if you have large binaries!
Plan 9's acid.
Linaro (need Allinea) DDT?
Radare and friends maybe. For sure it has some graph-like options as DDD had where you could graphically see everything.
Most RE tools today will integrate a debugger (or talk to gdb).
I am pretty happy with the debugger from Visual Studio.
Domain driven design?
I assume is this one https://www.gnu.org/software/ddd/
I used it back in Uni, in 98, and it really helped me to understand debuggers. After it, even using gdb made sense.
This is a legitimate question but this was clearly generated using an LLM.
To add something constructive, this demo represents an amazing ideal of what debugging could be: https://www.youtube.com/watch?v=72y2EC5fkcE