Debuggers Suck
robert.ocallahan.org
robert.ocallahan.org
It really is that much better. Once you start using it, it's really hard to go back to a situation where you don't have it. It's like driving a car that can only turn left -- sure, if you need to go right you can always 270 degrees to the left (by rerunning over and over again to get closer to what you want to examine, assuming your problem is reproducible enough), but it feels burdensome and ridiculous.
If AMD fixed their performance counters so that rr could work, I would switch immediately.
I am also a fan of Pernosco. It is especially useful when hooked up to a continuous integration system that can hand you a ready-to-go recording of all failures. You can get assigned a bug with a recording to go along with it. Even if it doesn't look important or tractable to look into the ordinary way, having a recording makes it easy enough to take a glance and check it out that you'll actually do it. At shops that do a lot of system-level programming (as opposed to scripting), the productivity boost ought to be worth a serious amount to the bean counters.
The other demotivator is noise. I work from home, and the other family members who share my office aren't keen on the sound of a fully spun up build server. I could run ethernet cable under the house to another room, maybe. (Relying on wifi bandwidth hasn't worked out very well. If only debuginfo weren't so enormous...)
10 gigabit is pretty cheap nowadays. Just in case your problem could simply be solved with higher bandwidth...
I would argue find bugs at compilation would be a separate solution from this and even earlier in the dev cycle.
I reread my comment to make sure, but I am being painfully explicit that good debug tools are still useful. But I prefer good tests over even good debug tools. I very rarely even printf debug as a result of automated testing.
This isn’t really just lip service; I have a project right now where I literally can’t attach a debugger or run locally. Of course, that’s a flaw and unreasonable. But, with good test hygiene, you can still be pretty productive in the face of this. Not only that, pushing code feels safe, even without the ability to actually run the code locally.
Do I think debugging is bad? No. I think it’s worse than testing, which when done right can save you from a hell of a lot of manual debugging, even in the face of complex software with networked dependencies.
I get ire on HN every single time I mention the virtues of testing instead of debugging. I suspect people literally don’t believe you can exchange quantities of one for the other, but where I’m standing it’s obviously possible. Same thing comes up with strong typing frequently, where you have people making the claim that typings don’t prevent bugs - but anyone who’s transitioned a medium sized code base to TypeScript can attest to the improvements it can bring.
In my opinion, automated tests are almost literally just debugging done by a script instead of a human.
edit: It's also being pointed out to me that rr is being used to debug failing tests, which is not something I thought of and my ignorance may be contributing to some of the reason my comment was so poorly received here rather than the testing vs debugging comparison. (In many cases, for unit tests, I had never thought of using a debugger to try to find the problem, because the actual problem is often made obvious enough by the failed assertion output.)
If you've ever used GDB in command-line mode then it's evident to you that it's the same thing. You can write tests of any level.
With a bit less funky GDB it's completely viable to solve a hard task of writing unit tests for the old untestable code without refactoring for example.
Also, one surprisingly useful feature of rr is that just about everything is debuggable without needing to plumb through debugger invocations through your whole system. It will record an entire process tree, so you can go back after the fact and select which process to replay debug. It's incredibly useful. I've been so happy to ditch my weird wrapper scripts to run cc1plus under a debugger while compiling with a buggy static analysis plugin I'm working on. Now I just run `rr record make` and wait for it to crash. It's magical.
We have a system when a test fails you get a rr replay of it to debug. It's the primary use for it.
I think this is an awesome use case for rewind debugging, that I had never thought of, and look forward to hopefully seeing it run on AMD Ryzen processors in the future, since I've bought into that ecosystem too much to go back already.
You can do reverse debugging in gdb.
Here's how:
Run: gdb --args /usr/bin/foo --bar baz
Type "start", then "record", then "continue"
Now can try a number of reverse-debugging commands, for example:
* reverse-stepi
* reverse-next
* etcLately, I've used Pernosco to debug my changes to a multi-process-relevant decade-old part of two-decade-old codebase, which was single-process when written, only later became multi-process-relevant, and recently became even multier-process. (And there are test cases, but there's a huge mass of them and they depend on decade-old comments not actually being true the interesting situations.)
Pernosco is very useful for debugging multi-process interaction in cases that would be very hard to debug even with rr.
edit: Implied is that image-based runtimes by definition produce reproducible snapshots of the error, with the full power of the Lisp or Smalltalk introspection tools. Edit-continue is the icing.
The CI or even the client can attaches the live image of the error, the programmer opens up the image when he opens the ticket and has the full venerable introspection tools of CL or Smalltalk. This directly addresses the reproducability issue you raised in said paragraph. There are indeed occasions where you need a proper rr-like trace, but what fraction of bugs fall into that category.
For illustration purposes: Grammarly has an example of fixing an intermittent network bug on a production server under load on the much-HN-linked blog post: https://tech.grammarly.com/blog/running-lisp-in-production
Good question. I guess it depends on what you mean by "need".
If you mean "for what fraction of bugs will a developer be unable to figure out the bug, given only the state where the error surfaced, but given unlimited time", I don't know. Most developers I've worked with either have rr recordings available or they do the work to reproduce the bug locally so they can work backwards to the root cause by rerunning the test, so their experiences don't reflect those constraints.
If you mean "for what fraction of bugs is it valuable to have an rr recording", I think the answer is very clearly "almost all". Many rr users, some on this very HN thread, will testify that debugging with rr is almost always better than debugging with gdb alone, whether or not they are able to reproduce the bug on-demand for debugging with gdb.
On the other hand, rr gives you a different set of features. You can rewind the full state of the program to any previous point in time. I've used rr a lot, and this capability is extremely useful. It makes debugging heap corruption, buffer overflows, use-after-free, and many other complicated bugs much, much easier. You just set a watchpoint on the data that was overwritten, then run the program in reverse (using the reverse-continue command) until it gets back to the point where the write happened. It really is like having a super power, but it's a different kind of super power than the Common Lisp repl.
Pernosco is another level above that. Instead of presenting you with the state of the program at a single moment in time and letting you run it backwards and forwards, it lets you query the full state of the program. Where before you would set a watchpoint and run backwards until you got to the previous write, with Pernosco you click on the value and it presents you with a list of _all_ writes to _all_ locations in the chain that leads to that value. For example, you can see that the value was written to the memory location from a register, and that before that it was read from some other memory location into the register, etc. It's really very cool.
There's no reason why a Common Lisp or Smalltalk runtime couldn't have these features in addition to a proper repl.
I'd like to try Pernosco, but we don't use it at work. There's no way I could afford it for my open-source projects, and I can't lobby for it without first trying it.
The real expense is in integrating with lots of different environments; Roc mentioned that integrations with Github and Travis were already working, but not Gitlab yet. (On the other hand you can do without any integration at all. Take a look at https://github.com/Pernosco/pernosco-submit/; it will upload the source from your computer to your debugging session if it's not cloned from a recognized git/hg host.)
We definitely are not aiming for "If you have to ask, you can't afford it".
I could pay a bit for personal use, and hopefully it'd still be useful without integrations etc; I don't think I could ask anyone else I'm working with (open-source wise) to do that.
> You can rewind the full state of the program to any previous point in time
Where does all this saved state go? I'm running a game that runs at 60fps, every frame 10s of megs of data change. It won't take more than a few second to run out of memory to save all the state. Further, if the game doesn't run at 60fps (or 90fps on VR) it's not usable, if saving this state slows the game down to 20fps or 15fps I can't tell if my code is working since the entire goal is to run at speed and check it all feels good.
Is there some technique these kinds of tools use to make it possible to do keep all changed state in this type of app or do they only work for some other type of app?
On the other hand, outputs are not really recorded. During the replay your program will generate exactly the same output, so saving it to disk isn't necessary.
See http://rr-project.org/ for all the details.
There's no such thing as a free lunch, so it'll certainly slow your game down a bit. I recommend saving the trace to the fastest disk you can find. Also, you would want to run your game during development both with and without rr, precisely so that you know that you're not exceeding your time budget.
https://morepypy.blogspot.com/2016/07/reverse-debugging-for-...
Never got popular enough to be ported to cpython, let alone python 3.
It's no surprise though: my experience is that most python devs don't know how to use a debugger. In fact, the huge majority of tutorials to learn python don't include one, nor they give explanation about python -m, venv, pip, the python path and other very important basis.
Even the people using pdb don't know very important commands like until, jump, up or tbreak.
Those who desperately need a debugger to learn are the first ones to not use it because of no GUI.
I saw Fernando Pérez use it in a YouTube video. Shame I cannot find it anymore. He was saying he debugs his code only by placing 1/0 in the part of the code he wants to inspect. After the exception is thrown %debug allows you to step into the stack and explore it via a repl. I found that crazy and tried it it. It is life changing. That's how I debug my python code for years now.
All the bugs I had in python were easier to reproduce, probably because it provides less rope than C or C++ to hang yourself. Usually, stopping a bit before the exception, exploring the variable states was enough to spot the bug.
Isn't there something about garbage-collected, dynamic-types, GIL-constrained languages that makes their typical bug easier to spot with a more limited debugger?
"gdb rr" - tried it with demo helloworld. That works. But when tried it with a more complex program (FBOSS - Faceboss' Switch/Router open source program), gdb rr function core dump immediately.
Also there are certain categories of bugs what are very difficult to use gdb with:
* multi threads / race condition bugs - gdb break point in affect the flow of the program and cause different behaviors.
* Bugs can not be reliably reproduced.
* Server programs or embedded system programs where the code can not be stopped.
* This problem get much worst when combine with performance related issue - I work on one system design requirement that would switch over to backup system when system missing a heart beat for 50 milliseconds. Any gdb breakpoint in that app would automatically trigger a hard system fail over to backup.
They're not as powerful as a debugger, but you can use them when nothing else will work. They're totally safe to use in production, because they can't modify memory, and they have a very small performance impact when in use.
There's little control flow to speak of in there, so if they had a problem, they could just dump locals() to get a good view of everything that happened.
When I went back to doing compilers work, I missed my debugger dearly. I also realized why I needed a debugger and my friends didn't: debuggers like this are particularly useful for work with compiler-like characteristics, of which there is unfortunately not enough.
Often big CRUD apps with a lot of business logic tend to have higher level, "logic flaws" that generate bugs. In those cases having something like `rr` to trace the origin of certain values would be a godsend.
I think these apps tend to be very wide in breadth, where instrumenting the entire system can be hard. Languages like Python (with `pdb` being a single-line in the middle of a program) are great in these scenarios because it allows for a pretty exploratory strategy when debugging.
Unfortunately there's a lot of people who do "print statement debugging" because the tooling really isn't practical in most places, but it would definitely be useful.
In a way they are often right, but in part because it's a self-fulfilling prophecy. If you don't use the debugger much, then excessive amount of indirection in many codebases is academic to you, the way (unfortunately) people think of color blindness, accessibility, hell internationalization for some. It's abstract, not something you have to deal with all the time. It doesn't affect you personally so you just ignore it (in other words, complete lack of empathy).
Once the codebase is full of shared mutable state, lasagna code, lavaflow antipatterns, callbacks, handlers, etc, then it really does seem like it's easier to pepper some print statements around and pray.
It's coherent, but wrong. Like a teenager's justification for how they deal with their messy room. Your life really would be easier and healthier if you cleaned some of this shit up, man.
Yes a debugger is necessary for debugging a system that has been written by someone else.
As someone who gets paid to clean up the cluster fuck that can be created by people building "crud" apps let me tell you better debugging tools would be a godsend.
In the old days with more monolithic apps you at least had one pile everyone shit in, and it was possible to get a handle on how it "functioned" and what its quirks were. Between micro-services and docker it isn't one big shit pile its a bunch of small ones, everywhere. And because of docker on occasion I now get to dig down to the underlying OS, because someone who had no right touching a production server, ever, built and tweaked a container and it "runs" so it went out. Did you know that you can run diff over ssh to compare and contrast the code on two containers? I can think of two rather heathy pay days in the last year where some Jr. on site engineer is the one who found the problem by going bottom up while I went top down.
The article states that "developers suck" and on a bad day I would say that the author is giving them too much credit. On a good day I realize that the issue is much deeper and that we hire technological magpies who want to chase shiny new objects (the javascript world is probably the worst at this). On a really bad day I realize that our entire industry is "wrong headed" starting from hiring. We test people on "can they write code" but no one ever asks the people walking through the door to read 50 pages of it. To find the one misplaced semicolon just to see if they picked up on any of the logic traps that were littered all over the other 49 pages.
So to you sir (or madam), I raise a glass for at least trying to build tools for me, the guy who fixes the "gnarly bugs" in crud apps. I wish you would go back and finish your work, the 6 of us who really appreciate you would happily buy you a beer.
Good luck doing that with Windows containers and diffing the HKEY_CURRENT_USER hive.
Anyways, rollback the container and start over.
The situation of the "live" (container) works and we can't figure out how to build a new one comes up more often than you think. Knowledge leaves the building. People run off in the weeds and then quit so they don't have to clean up the mess they just made and no one is willing to touch it. Some times an executive shits a kitten and your policy changes from "vendor everything" to "always use the latest and greatest version cause "security" . Sometimes in house staff are stupid, or incompetent, or lazy, or is smart enough to know that if they touch it they own it forever and choose to play one of the aforementioned roles. Sometimes the staff does know and no one in management listens to that guy or gal. Other times it is a minefield that every one dodges on their way to getting what ever else they need done, and no one addressed the elephant in the room till it causes a panic.
I think there are more versions and variations of the "we don't know how any more" tale that I haven't listed, and I am sure that my list is no where near complete however I'm sure you get the picture. The call aways starts the same, I always listen and nod and say not a problem, and use soothing tones to tell whoever is on the other end of the phone that the world isn't going to end. Some of them freak out when I ask for an NDA, and a development environment - many give in but some are unwilling to let me peek under the kimono. Most of the time I can tell them that "price" might get better (or worse) but it will be more accurate if they do so. It is good to get into a system, to talk to someone who can "solve your setup problems" because most of the time they will point you to a closet marked "dead body(es)" and in many cases your half way there. Sometimes it is simple and large (back to that elegant) other times there are a few corpses or an orderly grave yard, other times it is an Alderan level apocalypse and you shut the door and slowly back away.
At that point I can give a rate, and a guess at a time line for a fix (and yes I estimate like scotty from Star Trek). Most of the time the person on the other end of the phone is desperate enough to say yes. With that magic little word the hunt begins - and many systems are shockingly resilient to being ripped apart and abused. Often times things that look important can be ripped out, swapped out or shut off and things still happily churn along up to a point. From a global view you can quickly get to a city or block sized area and then the fun begins. The solution in more than half the cases is fairly simple, I'm changing code because requirements, directions and documentation artifacts have become more sparse as time goes on. Sometimes I'm quashing good ideas for the sake of consistency but most of the time its some sort of "cute" piece of code that is getting ripped out for the betterment of all involved.
Rare is it that I tell someone to really "start over", to burn the whole thing to the ground and not look back. Those closets with Alderan in them do exist and are frighting and fun when you find them. I do love a green field but I am wary of the company that offers me one because most of them make a whole separate set of mistakes on V2.
No one every reads any of the 475 billion lines of crap they are updating, who really knows if you are making things worse or better, or where any of it comes from. Most of us wouldn't eat a ham sandwich handed to us on the street by a stranger but will happily run code from 100's of other people on our servers with out even looking at it.
I have three philosophical perspectives that help me better frame the software industry.
Number one: People aren't stupid. Rather, things are hard.
Number two: Writing software is like doing a trillion piece jigsaw puzzle.
Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.
People who could do better are pushed to ship a feature tomorrow because hey it's simple and it can't take more than a day or so, right? And then we have the next feature..
I haven't seen anyone pull off a large code base without ever going back and reiterating on the design and refactoring things as needed in an attempt to try and maintain the qualities we (developers) would want in software. Unfortunately there are too many places in the commercial world where the time for that work is never given.
Then suddenly you learn to do the right thing ("right" means the one that is not stressing you out) faster. And you're a happy person.
Or customers, for that matter. Sucks when there's weeks' worth of reworking guts, refactoring, and testing to be done and you need to bill someone for the time.
What I've experienced is that in open source projects, patches get rejected and rejected until everyone agrees they're good enough, and sometimes that takes months or years of iteration. Not good enough? Spend more time on it. That's how things improve.
Wanna write a test suite? Need time.
Wanna run a battery of static analysis tools and inspect their output? Need time.
Wanna run a fuzzer? Need time.
Wanna document things? Need time.
Wanna rework the buggy guts of your application? Need time.
Want to figure out the heisenbug that rears its head every once in a while? Need time.
Realize the API the guy before you wrote is crap and needs to be rewritten, and all components that depend on it need to be updated to match the new API? Need time.
Want to formally verify that the core mechanisms of your critical application are correct? Need time.
Need to figure out and rewrite the part of the program that slows everything down to a miserable crawl? Need time.
On the projects I'm on right now, I could think of months of stuff to do to get them to a state where I'm somewhat happy with them.
Every goddamn thing just takes time and not much else. And IME the commercial world presents the same problem over and over again, far too often: you don't have time, you can't bill your customer for all the work that should be done (if good software were a concern), and you can't block the thing they want just because you first want to fix three other things that'd take a while to do. Crap just accumulates, because people build in a hurry, on a bad foundation.
Even open source projects where deadlines and "we sold this as an item that takes three days of work" aren't a thing, time is always a problem because there's only so many people putting hours in.
There is that old adage "perfect is the enemy of the good", well I would argue that "working is the enemy of the good" as well. When your "average" engineer spends 2 to 3 years at a company how often do they have to pick up the messes they make? Rarely? Never? It doesn't pay to be a good citizen in that case - it pays to be (as another poster pointed out) fast.
> The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in.
If software is hard, and average people are not ~willing~ to put in the work to make it less so, doesn't that make them lazy? Willfully and willingly lazy at that.
> Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.
Look around you,how many engineers do you know who's code you would be willing to bet your life on? The problem is that every engineer thinks they are a surgeon (and a good one at that) when many of them wouldn't even make good nurses.
No, it simply makes good systems brutally improbable.
Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.
If I walk the 1/4 of a mile flat loop at the park down the street from my house every day and call my self a hiker I would be a liar.
Really were talking about Dunning–Kruger, "If you're incompetent, you can't know you're incompetent". I go back to "stupid" as the clear winner, and add in that they are too lazy to get out of that rut (put in the work) and that the title "engineer" becomes a willful or delusional lie.
> Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.
To your earlier point, we give everyone with a pre med degree a scalpel and a patient with a traumatic brain injury and say "go" were going to have a lot of bodies. The case is clear that we are far too tolerant of bad code and bad engineers, and we don't have roles like "podiatry" or "nurse" to demote people down to where they can contribute and we can limit their harm.
It's not just that, but the typical company is too busy chasing new features and fighting fires with understaffed engineering teams on constant death marches to unrealistic deadlines. Nobody has time to do things right, though many of us want to. Instead the systems accrete endless layers of bandaids and half-assed solutions which everybody hates but no one has the luxury to fix or solve properly the first time.
This I think is the reason most software nowadays is garbage, there is a lot of low skilled people creating it. I blame bootcamps and dumbed down CS courses for this. It's not necessarily the developers fault, they can't improve on the things they don't know they are bad at.
Seriously. I do this for living. For relaxing, I made a language in the side, because is MORE relaxing.
---
In the case of debuggers:
1- Yes, most debuggers suck not only suck, are near useless. What exactly can I do with "0Xfff23dfblabla" when I try to poke at a value?
This simple thing fail almost everything (.NET, swift, rust, java, ...)
Ok-ish: python, foxpro, js, probably other scripting langs?
2- Skip over thirdy-party and/or standard libraries and focus on MY code. Too much stupid "step in/out" when get in cross calls to code I'm not interested. This also mean better filtering of modules? Libraries?
3- Be INTEGRATED with my logging or tracing calls! print debugging is so useful, but is disconnected from step debugging!
4- Have search capabilities: Let me search in the debugger for vars, functions, etc to step in.
5- Have all the breakpoints on view! I wish I can enable/disable breakpoints with simple clicks without jumping around code.
6- Yes, I wanna look at that array/map with a MILLON values there, Imagine if computers could search stuff like that. Also, how nice if looking at/scrolling(remember, the debugger have not search capabilities) a array of few hundreds not kill my 32 GB machine..
This is also key. I fear A LOT to debug if it have too much variables or if some of it hold large values, like above 20...
Also: Allow to put simple invariants: Stop when this function is called N times, or when this var become Y, etc
This is some of the minimal stuff. I have not use rr or similar but if I can't see simple values, no opaque, can't filter code out, etc is the same pain but with more sophistication, I fear...
Some other stuff, nice but maybe is ask too much:
- Diff of values. Most debugging is to see what have changed (also nice: only stop IF changed)
- The debugging overlay the editor. Put the values above the vars, color/mark the touched lines, tell me how many times X have changed...
- Pull in a SINGLE view all the code on the stack, ie: Instead of jump around dozen files, put the dozen call in a single view and jump into that instead.
- Find a solution if thing are around threads. What if I have 100 threads?
- Have a command window (alike terminal) as useful as FoxPro 2.6 DOS. Something close today is Jupiter or the console in the JS debug tools. But this requiere integration that is available on FoxPro and maybe not cross easy to other langs
--
A step debugger is just a part. Also:
- Structured logging is good... but where is the log viewer for that? How collapse all the noise from logging that is only a repeat of the same stuff? How filter to the section where ERRs/WARns happened? What can I do with a 1 GB of logs to truly find the issue?
- How put tracing that can I enable/disable at runtime?
- How put tracing at runtime, if I don't have put the code for tracing before? Could I say "Trace before/after the code enter this function call?
Visual Studio team started doing VS productivity talks, because they noticed that many feature requests were about features that VS already supports for ages.
There's some progress - these days it's mostly confined to the data science crowd, and web devs have mostly caught up, due to the rapidly increasing popularity of PyCharm and VS Code over the past few years.
In general, it feels like there's some weird gulf when it comes to tooling. On one side, you have the hardcore C gurus (often doing embedded), who do heavy scripting in gdb to solve very complex problems. On the other side, you have IDEs that grew out of 1990s RAD - Eclipse, Visual Studio etc - in ecosystems where the language and the tooling are usually developed and packaged together (e.g. whenever a C# compiler ships a new language feature in stable, there's also a new stable version of VS that supports it in the editor, the debugger etc - and UX for those is explicitly considered when designing features). And in the middle, you have all the high-level "scripting" languages, which have both low-level debuggers and powerful high-level IDEs - but their respective dev cultures largely ignore both for historical reasons.
I have lost track of the number of times that I kill myself debugging something with gdb only to later explain it to some group and somebody says: "LOL, n00b. Why didn't you just use <magic gdb command>?"
Why? Because I didn't know about the magic gdb command and there was no documentation anywhere that would have pointed me to the fact as to that command being exactly what I was busting my head over.
Debuggers are like build systems. You don't diddle with them every day and when you are something is wrong and you're on a deadline.
If people want programmers to appreciate better debugging, we need some good videos of people debugging real problems in real time with debuggers.
One other way developers suck (myself too often included) is that they don't RTFM. And I don't mean "skim the manpage" RTFM, but actually read the included manual. If you're going to use something day in, day out, it's really worth it to spend the couple of hours required and read the included manual cover-to-cover. It saves a lot of time and frustration in the long run.
(I don't use GDB myself all that much, but the last time I had to, I got tired of flailing around; I read the info page cover to cover, and the time investment repaid itself on that single task.)
We explicitly tried hard to make Pernosco discoverable, partly by creating simple but powerful and composable features, and partly using traditional GUI techniques. We definitely haven't fully succeeded, but I think we moved the needle. If you dig through https://pernos.co/about/ you can see what we tried, and if you try the demo https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html you can see how well we succeeded.
> static.pernos.co has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.
> The issue is most likely with the website, and there is nothing you can do to resolve it.
I never had any such problem with Visual Studio. Isn't there a GUI for gdb? (I really don't know)
The way I want to write code is directly as it is executing. I think the distinction between writing code in an editor and debugging is arbitrary - I want to edit programs as they are running and see the data as it flows through beside the code. I want to be able to browse the object graph as if it were a filesystem like I could in Objective-C (I think Smalltalk invented this?).
This does require a different code architecture to be effective: more pure functions and less tangled state, but those are usually considered to be good things. Improving debugging might have a positive effect on code quality.
I do something like that every day, especially when I'm dealing with weird and arbitrary APIs (which is most of them). I write as much of a function as I know how to do, set a breakpoint on the next line, and run what I have until it hits that breakpoint.
Then I can write the next few lines of code while looking at actual data.
If it happens to be Python code and I'm using IntelliJ, I can also open a Python console - a REPL running in the context of my breakpoint.
This helps me understand the libraries and APIs I'm working with in a way that just editing static non-running code doesn't.
I would like if existing languages allowed you to "level up" your code.
examples that have worked for me: "use strict" in perl, "tainting" in perl, assertions in C (runtime and compile time), -Wall -Werror in shared codebases, and similar.
the idea would be to incrementally make everybody's code a little better in manageable increments.
One problem I found was it doesn't work with generic methods though, not sure if MS is working on making that work. To me "Edit and continue" is a great tool.
I am writing C++ in both IDEs and VS is far more useful to me.
On the other hand, running a debugger in Clojure is a bit of a mess. Generally I just don't and end up creating "watchdog" atoms that hold bits of state that I think are relevant to the situation at hand and evaluate those with a live REPL connection. See e.g. https://eli.thegreenplace.net/2017/notes-on-debugging-clojur.... Although the article ends on an optimistic tone, I've found using a debugger with Java or Scala more rewarding than using one with Clojure.
Does anyone knows something similar in other languages or build systems? Does any of LISP environments provide such debugging functionality?
I think there are a couple of reasons for this. One is that for many if not most developer tools, there are free options that are at least competitive with paid tools. Another is that developers can, at least in theory, read source code to better understand a tool they use, or even improve it, and therefore want access to the source code. Which means open source tools have an advantage over proprietary tools in addition to price.
It's a bit like Github, which also isn't free software, but so many free software projects use it, in part because people think they could easily switch away from it if they wanted to. (I think in practice switching away from Github would far harder than switching debuggers.)
Agree.
A major problem that needs to be solved is creating a user-funded, donation-based model of free software development. The old argument was that copies of free software can be given a price and sold on tapes, thus the distinctions between price and speech - you could potentially support the development of free software by selling copies. But this practice has largely disappeared in the Internet age, and it seems the only business model of free software today is value-added services.
One could argue that free software, by its definition, allows unrestricted, free redistribution (both speech and beer), others can still distribute it for free, so it's meaningless to sell copies. But I would say that many users may still want to pay to support its development if there's a system in place to encourage it. Currently, the visibility of donation in free software remains low.
For example, there's nothing to prevent us from asking for a donation to download on the official website or an "app store", or integrating a payment system on a code hosting platform, or a micropayment system that encourages you to pay $1 for reading some documentation. You can even integrating a micropayment system to a package manager that pays while you are installing something, or even refuse to provide documentation from the official website or support from a package manager unless you've paid, forcing people to either pay or find a 3rd-party's copy, it's ethically questionable but remains valid under the principle of free software), the possibilities are endless.
The only obstacle I see is the lack of a good micropayment system, but it can be solved in principle.
* Most developers start to learn at a time where they don't have the money for tools.
* If the free solution is good enough, there is little motivation to invest in higher grade tools.
* If there are "good enough" free tools, the market is very small for a paid solution so the price has to be high.
Developers tending to read code to understand a tool or even improve it, is a very rare case in my experience, which is limited to mostly computer science students and some researchers though.
There are many reasons why I don't do it myself, e.g., code of many projects is hard to read and takes a lot of time, implementation only losely helps me understand the tool besides edge cases where it doesn't do what I want, I am interested in solving a problem with the tool instead of understanding the tool.
There is even more to this point. Most programmers still like to work on some personal projects here and there, and they don't like to have a completely different workflow at work and at home. Plus they usually have multiple devices (desktop PC, laptop) or even multiple VMs and in the past licensing wasn't really compatible with these kinds of scenarios, and you'd often have to purchase the software 5 times.
Not to mention it is annoying when you sit in front of a new device and you can't just install your tools and instead have to think about how you're going to transfer or apply your license.
That is only because of greed. During the 80s and 90s, Borland for example had a "no-nonsense license agreement" that told you to treat the software as a book: make as many copies as you want, have as many users as you want and even resell it as long as - like a book - only one person at a time is using it.
They did change it at some point in the 2000s, probably around the time when they stopped selling Delphi for $99 as the cheapest option, introduced online DRM and prices skyrocketed to the thousands.
In principle proprietary tools could also provide the source.
It is not very surprising given that it is a SaaS which doesn't even list pricing on its page.
>I want everyone to know about Pernosco, not just to attract the customers we need for sustainable debugging investment, but so that developers everywhere wake up to the awful state of debugging and rise up to demand an end to it.
This is rather conflicting statement as Pernosco business model seems to be solely targeted at bigcorps paying big money, and not developers promoting Pernosco through using it in own hobby projects. (I don't know if this is actually true, but email-us-to-just-get-a-quote is huge discouragement)
I don't want to pay an undisclosed amount of money for the opportunity to spend an unknown amount of time to get started with a new tool that may very well suck.
It is true, nobody wants to pay for engineering tools. I wish the author luck.
I remember many years ago trying out purify, and it revolutionized my debugging. The problem seemed to be that the program I wrote could be debugged by everyone - if you just got everyone a per-user license.
It would have been hard enough to justify a license for me (a person at the bottom with no power), it was impossible to justify a license for everybody.
"Floating" licenses are better here. In my experience, companies are reluctant to buy licenses for an entire team for something that's used only occasionally. It's easier to make them buy a few floating licenses to be shared by the team.
Unfortunately there's another way to do this nowadays. Offer convenient tools with analytics, etc... (and coincidentally spy on everyone)
IDA Pro has been doing alright: https://www.hex-rays.com/products/ida/order.shtml
But it's definitely hard to compete with open-source alternatives, piracy, general skepticism, etc.
I found an easier path for dev tooling is focus on tools for non-traditional/non-fulltime coders, e.g., analysts, devops, secops, ... , for whom being able to code introduces a superpower, not an increment on status quo within a crowded space. It's a shame devs destroy our own potential.
About a dozen years ago my workplace built something like that in-house. I see you've made progress. You have the reverse dataflow. You can't debug an arbitrary firmware or bootloader or OS however, and you can't properly debug a bunch of interacting systems.
Simics also seems to have most of that. I last checked about half a dozen years ago, so the other features might now also be supported. It's not cheap, but you could model all the computers of a factory or aircraft all at once. You could then step forwards and backwards as desired, coherently examining the RAM and other hardware of all the computers as you please.
I wouldn't be surprised if dozens of similar tools have been built.
Computing today looks different compared to 2009, AWS came out in 2006, the original iPhone came out in 2007. Why, then, would the market for debuggers and other developer tools still look the same as it did 10 years ago?
Coding bootcamps have sprung up and regardless of what you think of them, there are more developers today than before. The more developers out there, the larger the market to sell developer tools to. Auto-mechanics and other tradesmen have a couple thousand dollars worth of tools that they bought themselves. Why then are software developers so uniquely hard to sell to?
JetBrains, makers and sellers of Python-specific IDE PyCharm, seems to be doing okay.
With companies now tracking "developer velocity" as a metric (somehow), a tool that improves velocity seems like it would be an easy sell. If I can't rewind code execution in the debugger, I might as well just be printing variables to the screen. (Which is a time honored debugging method that will never die. But maybe it should.)
There's a B2C vs B2B question in there, but maybe the market is changing in the software developer tools market.
This recent article and related thread on HN offers some additional insight: https://news.ycombinator.com/item?id=21511411 .
That was a TECO like editor instead of a line editor.
It provides more than that - the "Edit and Continue" feature is invaluable, allowing you to pause execution at a breakpoint and change the code, such that the new code runs when execution resumes.
As an aside, Rider also has this feature, and I find it to be much faster and more reliable than VS.
Visual Studio also recently got "Time Travel Debugging"[0], which lets you record execution then replay in the debugger. I haven't actually tried this yet.
[0] https://devblogs.microsoft.com/visualstudio/introducing-time...
I think that not many people have used TTD yet so when people talk about how great the VS debugger is, they're not really talking about TTD.
So I dug down into the bowels to see why it's so lame......
Oh my.
The hideous complexity imposed by (multiple) CPU designs, OS interfaces, the vast mismatch between the code as written and the code as compiled. The complexity of the DWARF standard....
Sigh! Sorry gdb dev's... I'm sorry I ever thought bad of you guys...
You guys are doing way better than I'd ever do.
https://developer.apple.com/documentation/metal/shader_autho...
But if the debugger doesn’t come with source code that I can inspect and modify, that’s a big negative that will have to compete with whatever advantages the debugger has. (It’s not just a hypothetical use case. I’ve grepped through the source code of GDB and LLDB on several occasions to figure out why they weren’t working properly.)
And if the debugger is SaaS, meaning it will almost certainly disappear in a few years, after I’ve gotten used to it… it would have to be some super magic unicorn stuff for me to consider touching it, even if it were free.
Pernosco is both closed-source and SaaS. Pass.
Then you'll be in a far better position to evaluate the additional benefits Pernosco might provide you. I think your reasons for being leery of it are valid, but there are very legitimate scenarios where it makes sense IMO. (Recording CI jobs automatically, collaborative debugging, situations where the critical dataflow is hard to follow...) And they integrate, too - you can upload rr recordings to Pernosco when you need the extra functionality.
I guess it also depends on the pricing model, which I know nothing about.
And yes, Pernosco is super magic unicorn stuff. Do read about it.
If there is a silver lining, it must be that my lack of debugger skills can only mean I spend a lot of time writing safe, bug free code. Or more likely, no code at all.
Fixing bugs with a debugger is a great skill. If you're good at it, you'll be able to figure out many problems far faster than people who don't use debuggers. You can skip a lot of deep thinking and go straight to the site of a problem. You can explore the actual execution paths in ways you'd never figure out from all the layers of templates and overloads. You can debug other people's code almost as easily as your own. You won't get caught by some subtly wrong assumption you're making when thinking through how things are supposed to work.
Debuggers are dangerous. They're insanely useful, but once you start depending on them, your ability to fix things without a debugger will atrophy. If I had the self-discipline, I'd ban myself from using debuggers for a month out of every year.
In my embedded work I avoided using debuggers because setting up remote debugging can be a PITA. I figured out a long time ago that building remote debug setup into my workflow will save oodles of time in the end. Still, I find myself using plain old logging more often than not - the bug may not be in C or C++, it may be on the JavaScript run in the interpreter embedded in that C++ application.
Yes. Something is wrong, but you can't tell how you got into that state.
If a core file is all you've got, then you do your best (and there are some nifty researchy tools that help with that). But if you could have a full recording instead of the core file, then of course you'd want the recording, because that dramatically increases the probability you'll be able to debug the bug (to near 100%, if the bug is in the code you've recorded).
> Rust and Go and all the new age languages never seemed to have heard of it despite proclaiming their commitment
Go rebuild / full rebuild times are fast enough. I don't think edit and continue would be that amazing with 1000 concurrent goroutines.
Rust will hopefully have edit and continue one day. Hopefully compiler will get faster as well... That said, I much prefer Rust compiler catching a significant portion of issues at compilation phase, thus not requiring debugging in the first place.
While rr seems to be very nice, there's still room for improvement. But I guess this kind of tool might be impossible on current hardware, at least without emulation.
One bit I’d add is the bad attitude towards debugging in optimizing compilers. It’s not obvious to me that optimizing for perf at the cost of debugging fidelity is a good trade off but here we are. Like, if you could save enough time debugging with better tools then you’d have more time to optimize. And maybe if compiler devs confronted full-speed debugging head on then we’d get to perf that is good enough for everyone and we’d all be more productive.
Or maybe it all sucks and all we can do is bitch about it.
gcc is actually pretty good at preserving debuggability through optimizations. clang? Not so much.
For clang users things would get a lot better if someone just fixed a lot of clang bugs.
Omniscient debugging creates new opportunities to overcome the impact of optimizations. E.g. are you trying to display the value of a variable after its live range ended? No problem, look back in time to when it was live. However, in many cases we need extensions to debuginfo to take advantage of that; e.g. DWARF can't express the difference between "variable outside live range, but has not been modified" and "we don't know where it is".
If we had a debug build and an opt build of the same program then there are some mad-science approaches that would be fun to try.
So I actually think the perf/debuggability tradeoff is not fundamental and with the right resources we could mostly make it go away.
I believe that this overhead in conditions where the codegen is otherwise behaving well is around 5%. So not terrible, but that’s my call of the price relative to -O3 or whatever.
In the comment about omniscience, here's an example of what I'm proposing: -- Debuginfo says variable 'x' is in register R1 in the PC range 0-10 -- At 10, R1 is set to something unrelated -- Debuginfo says in the PC range 10-20, 'x' is equal to its last live value (i.e. its value at PC=10) Pernosco would evaluate the value of 'x' at PC=15 by actually reconstructing the value in R1 at PC=10, using the omniscient database we already have.
Of course DWARF can't currently express this, because normal debuggers can't make use of it. Even in rr it would be quite expensive to compute.
If you're familiar with DW_OP_entry_value, it's a bit like that. That's a way for DWARF to say "the value of this variable (e.g. a parameter) is equal to the value of this subexpression (usually a register) on entry to the function", which regular debuggers can often obtain using stack unwinding. Pernosco doesn't do the unwinding, instead it (logically) warps back in time to the point where the function was called and evaluates the subexpression in that context.
This of course will only work for a little while until someone who has more time than money (or is a big company that wants to commoditize the tool) will build a command-line version of it on Linux. Weird UX and the need for a spaghetti of shell scripts to integrate with vim/emacs/vscode/sublime/ed will soon follow with an Eclipse addon that nobody will use a bit later. After a macOS port, assuming these are still a thing, Apple may create a nice looking UI and integrate it with Xcode or someone like Panic may create a good front end and sell it for ~$99.
Most people will keep using Windows, think Visual Studio has the best debugger and everyone will be happy.
Those that learn about the "overwhelmingly better approach" will consider the free one the best one and its spaghetti of shell scripts approach the obvious best approach, because they wont have any real deep experience with the paid tools (either the original one or the macOS shiny wrapper - which will be considered as unnecessary by most anyway) to properly judge.
Also either you start with rr as a base, in which case you need people with deep rr knowledge, and I know their names, or you build or buy your own record and replay framework --- more time and money.
Some other company could do it. Remember that we've gotten Navigator (now Firefox), Blender, OpenSolaris, OpenOffice, and the .net stuff. Governments can surprise us as well; the USA did that with Ghidra.
I'm not expecting it anytime soon, but surprises happen.
Certainly possible that someone has a secret tool that's as good or better. However, I expect the secret tools in this space are all focused on so-called 'security research', i.e. more like Tetrane and targeted at adversaries of the software under test, rather than its developers, which leads to a somewhat different feature set.
I belong in the first camp (Debugging to me means "running", It's not a tool I dig out to actually de-BUG something, it's just the tool I use to run my code every time).
It's relatively easy to get value out of a debugger in a straightforward synchronous userland process.
It's much harder in a complex embedded application full of threads, callbacks, asynchronous execution, and a target where you may need to set up something like remote debugging and potentially tweak the whole system config to make it possible in the first place. This is also more likely to be the kind of system that doesn't run from your fancy IDE..
If you add generative testing via spec or test.generative, you can simulate lots of the scenarios you’d otherwise be desperate to have a dump of the environment to inspect. If you get there, profilers usually tell the story in adequate detail. Profiling and debugging are related and often share responsibilities.
Often, you can't attach a debugger to the production system, but you can get logs. Also, you are often interested in the whole sequence of events leading up to the problem, not just the current state. And, to find where to look in the first place, you need to get an idea of what's wrong - a debugger doesn't help with that.
https://henrikwarne.com/2014/01/01/finding-bugs-debugger-ver...
Pernosco has something for that too. Since it has replay capability, it can replay with more logging than the original execution had. So to some degree, you can have your cake and eat it too. It's not perfect: I think it requires some level of integration with a given logging system, it's not going to do the full I/O to send your logs to where they would ordinarily go (syslogd or whatever), and you can't log values with debug-only code that wasn't compiled into the binary. But it's still a large boost to the fundamental tension around logging.
It's totally understandable that people prefer good logging to traditional debuggers which really suck. But you have missed the point of the OP that much better debuggers are possible.
On another note there's actually this thing I'd like to call "debuggability".
I find that if you just thinking of this as a measure of how easy your program is to debug and when making design decisions you think of "which design makes this thing easier to debug" your program will be simplified. To me mean high "debuggability" at design level means cohesive independent components which makes it easy to write unit tests for them which then subsequently makes it easy to debug them as well.
Other thing is sometimes I see people write code like this:
getFoo()->getBar()->doComputation(); doComputation(getFoo()->GetSausage());
If you used some temporary variables to store the results of those function calls again you'll make your program more "debuggable".
Good ones will let you set a breakpoint on a subexpression, so in this case you'd just highlight the call that you want, and it'll stop right before it happens.
step, then finish. Repeat until you're in the right subexpression.
Step will enter a subexpression. If it's not the one you want, finish will run until it's done. Repeat.
In the end, I gave up and built self-introspection tools directly into the program itself. It may not be a general solution, but it worked for me.
To be frank, I didn't use debuggers much at all throughout my tenure at university, my internship at Amazon, and my first few gigs in the games industry. I still had bugs, but I was able to solve them mostly through printf, gut checks, and dousing rods.
But when I got to AAA console game development, the leads had to show me how to use the immense power of the VS debugger, and now I have difficulty imaging life without it. Different scales of problems require different tooling to manage.
In python, for example I use: pycodestyle, pyflakes, mypy and logging, and bpython.
For web apps, reload time is the most important factor. Work on reducing the time to compile, reload and test.
The whole point of the OP was that just about all debuggers suck, but debuggers can be a lot more powerful.
Which is why projects have to built the debugging into the product in one form or another. A topic that is far to broad/long for a posting here, but most projects end up implementing in various forms if they live long enough.
I blame larger programming teams. It’s a chicken and egg problem: if you want to start using a commercial debugger, you need to buy it for everyone on dev team, so you can share your knowledge and benefit from others’ help; and you are not going to buy a debugger for everyone if you are not using it yourself.
Now we're exporting logs from a docker in docker on travis. O tempora O mores
I used to use it. It was so painful and limited that I think it helped only a single time, and was all in all a net loss in productivity.
I've only used it with C++ applications so far, but the WinDbg blog claims that it also works when debugging managed code with the SOS debugging extension.
Visual Studio confusingly has the somewhat-similar "Intellitrace", but AFAIK that only records previous states collected for certain events (breakpoints, first-chance-exceptions, ...). It can't rewind the program an arbitrary point like the time-travel debugger can, so it's not useful to follow data flow.
While at the outset this post seems like a rant, I believe the opinion expressed is very astute.
In a perfect world, we would all write programs as a composition of mostly small pure functions, with type systems that can enforce provable correctness, and a small part of the code would generate 'effects'.
We don't live in such a world.
We often inherit gargantuan codebases written in an extremely imperative fashion, with a lot of state management baked into functions thousands of lines long with conditional and loop blocks nesting 5 levels deep.
And there's a bug in one of those loops.
A good step debugger that we could reverse step and inspect would save us somewhat in such times of perils. But they don't really exist. Or if they do, don't integrate with your editor.
So needing debuggers suck. Debuggers themselves suck. Developers suck. Deadlines suck.
Sigh.
I definitely agree - I've watched people painstakingly stare at code, add print/log statements and waste enormous amounts of time -- and then I asked them why they don't just use a debugger and always got unsatisfactory answers that seem to hide the fact they have a hard time using the tool effectively.
Of course, like you say, using a debugger effectively in many settings is difficult. A multi-threaded program setting a breakpoint and inspecting state difficult, as finding the right thread to inspect can be difficult -- not to mention having other threads alter your debugger's display state by breaking once you're already inspecting the correct thread. I've had this happen with native software (e.g. C++), for which rr is great (but unfortunately I've worked mostly on Windows), but also for backend software written in higher level languages such as Python.
Using a classic debugger effectively in a multithreaded environment requires being aware of the issues you're about to encounter, and knowing how to deal with them with the debugger's tooling -- which ranges between difficult and not possible. For example, with Windbg - it is possible to break on the correct thread, but setting a conditional breakpoint (even a simple one) is nothing less than an incantation to someone who's never used it before. As such, the cost of entry is very high, and I've watched kernel driver developers use prints and asserts to debug.
While at my previous job, I thought that logs are a really crappy way of debugging an issue in production when it arises - if you have all the right information and can fetch it quickly, great. But often, that's not the case - you might end up needing to add logs, or worse - the issue will be in a hotspot where you simply can't log - since you'll have _too much_ data, or you can't log everything for compliance reasons.
I had a similar idea to Rookout's (and even almost accidentally raised money just by talking to the right people, although I was not prepared at the time to start a company), and discovered Rookout later on, going on to use them.
When I left my last job, I joined Rookout because -- like O'Callahan (and Pernosco) -- I'm very much a strong believer that the way debugging is today isn't good enough, and can be a lot better.
Nowadays Rookout's product for interactive debugging backend apps is pretty much complete, works well and fast -- and we're exploring other ways to make debugging easier.
Rookout's a bit different to rr/Pernosco in that it allows you to collect variables from anywhere during runtime (instead of allowing you to replay a recording) -- it's a bit similar to allowing you to add a log that logs locals() anywhere, except without changing your code, redeploying, etc.
Also replay debugging doesn't have to be complicated.
This enables replaying sections of a program with hot reloading by requiring that the data structures are serialized. It also catches low level exceptions so that replay and hot reloading won't crash the running program.
https://github.com/LiveAsynchronousVisualizedArchitecture/la...
It's similar to rr because you can go back and forth in time, but unlike rr it doesn't record much of the state (besides return values) and instead focuses primarily on the execution flow by the threads across the different methods, classes and packages.
There’s tons of languages. The closest I’ve seen to a sane debugging experience is vscode. It works the same across languages. Python, browser js, node js, go, Java etc
Mostly because they all share same underlying debug adapter protocol.
Now if you truly want to make debugging phenomenally better. Get those new apis in the protocol for reverse debugging and the likes. Then make it happen across a bunch of languages. I will pay It. I know most people will.
We just want something uniform that works
Log debugging is easiest. Why ? Because it’s simple AF. Running a 1000 things in parallel. Aggregate the logs and filter what you need.
Going through layers of virtual computation? Physical -> Node pool -> service -> pod -> container. Write to log, aggregate.
There’s been a lot of promise but haven’t seen anything that beats the simplicity of structured logs + good log analyzer tool.