That is extremely interesting. I always thought that the debugging tools we have are cool... but we could do _so much more_ to observe the systems we develop. And the scripting languages in most of the embedded debuggers I had worked with sucked.
To get us to the state of the art debugging research in the 90s:
- FOL scripting trap on every variable, function, etc. essentially you need this "monitor" or supervisor behind the scenes observing the total state of the system _as it exists in source code, not in hex_. - reversible programs that can go forward and backward in program state & history. Not sure if the 90s research had a full "tree" of possible executions you could walk or if it was linear. - if you attach something like a Lamport state matrix with the a transport layer you can move to a parallel/concurrent debugger
OTel and friends are essentially trying to build points 2 and 3 there, but its very mushy, since you're not actually capturing a full system state as part of the system, but whatever the coders chose to put in the system capture.
I remember in 2012, 2013 or so, Green River came out with a single core reversible debugger. gdb had one too in '09, or so.
Some cool information in an old SO thread: https://stackoverflow.com/q/1470434/26227
Isn’t this at odds with highly optimized AOT code? I suspect that’s the ultimate blocker but certainly you have this today for any kind of JIT/Interpreted language.
EBF does kind of let you get some of it back, but ultimately it turns out that writing code to debug other code is really complex and you experience rapidly diminishing returns for most bugs (except the super gnarly ones) because a little bit of state and system’s understanding proves sufficient most of the time.
I’m sure you’re aware but GDB has reversible debugging via rr. It’s still hard to get all the way back to source if you’re debugging an optimized build of your code. And polyglot environments can be extra challenging unless special care is taken. I agree it’s a mess, but I think that’s because it’s actually extremely hard to meet competing goals. Do you see the reason why it’s so bad differently with your greater experience in the space?
It's locked behind the really expensive (like 10k or so? For I guess a seat a year?) Visual studio license and IIUC windows-only.
My experience was that C++/Java debuggers beat interpreted language debuggers shockingly well. So much so I still don't touch interpreted language debuggers today. I should check in sometime on Python soon.
Debug code has to, essentially, include debug symbols and tracking, or, alternatively, there's that supervisor monitoring all source-code relevant locations. Things will be slower ... on a conventional x86 chip.
I would suggest that its plausible today that a separate _core_ could be developed for internal system monitoring (think on-SoC JTAG and you get the idea) and that core could be dedicated to more OS functionality otherwise. Notice that this involves a different ISA.
The core tooling advancement problem I believe reduces to this:
> because a little bit of state and system’s understanding proves sufficient most of the time
After a decade being an industry professional, I would say most is 99.5% or greater. 5 out of 1000 bugs seems about right.
So the genuinely advanced tools only start to come into play in very difficult situations - with the implication that only groups with difficult situations will _pay_ for the advanced tooling - with the implication that there's virtually no commercial market, and open source developers have to do this on their own time. Which in turn seems to be generally decreasing IMO - I believe the problem is approximately (say, 90% of the way) isomorphic to "cost of housing market" discussions essentially (how do I spend 1-2 years being someone who can understand graduate-level CS/SWE papers and write high quality code, and then write this state-of-the-art tool that no one will pay for, and still have housing, food, make debt payments etc).
Polyglot environments have to have an element of a debuggable protocol baked in at a pretty deep level to really make it work at, say, the level of Visual Studio in 1999 or something. There's all sorts of issues there around non-aligned interests that shouldn't need to exist. For example, a simple polyglot problem is "how do I send structured data from one process to another in a standardized way". The standard Unix way is "line structured", which works until it doesn't, and then you're staring at writing your own marshal/unmarshal system, which is, definitionally, only your standard. If we don't have _that_ solved, then the social pressure to have a interop debugger between, e.g., Python and Ruby is ... well, take your own guess there.
I think there's absolutely space for a hardware-software codesign to occur which builds out a hardware system, a compiler qua LLVM, a standardized interpreter layer, interop protocols, etc, and to produce a developer system that is a quantum leap beyond what we have today. But that would be a decade long project and have a very long term payoff.
the tl;dr then, imo, is that profound advancement in the field only directly benefits a few, and then only sometimes; so the incentive problem is the basic issue.
I do have to admit though, that my first read of your post was wrong tough, I read it as "debuggers used in embedded" like JTAG stuff, OpenOCD, etc. but you mean it as, if I read this right "debuggers embedded in the runtime/program" stuff like Python, or GDB server running on system right?
Think something like an Arduino.