Hacker News Reader

Comment by jpochtar | Hacker News Reader

As someone who's also worked on a debugger, I'm sad to say many programmers aren't working on problems that needed complex debugging. I couldn't understand why friends weren't interested in what I was working on, until I realized they were building CRUD apps all day.

There's little control flow to speak of in there, so if they had a problem, they could just dump locals() to get a good view of everything that happened.

When I went back to doing compilers work, I missed my debugger dearly. I also realized why I needed a debugger and my friends didn't: debuggers like this are particularly useful for work with compiler-like characteristics, of which there is unfortunately not enough.

View on HN

> I couldn't understand why friends weren't interested in what I was working on, until I realized they were building CRUD apps all day.

As someone who gets paid to clean up the cluster fuck that can be created by people building "crud" apps let me tell you better debugging tools would be a godsend.

In the old days with more monolithic apps you at least had one pile everyone shit in, and it was possible to get a handle on how it "functioned" and what its quirks were. Between micro-services and docker it isn't one big shit pile its a bunch of small ones, everywhere. And because of docker on occasion I now get to dig down to the underlying OS, because someone who had no right touching a production server, ever, built and tweaked a container and it "runs" so it went out. Did you know that you can run diff over ssh to compare and contrast the code on two containers? I can think of two rather heathy pay days in the last year where some Jr. on site engineer is the one who found the problem by going bottom up while I went top down.

The article states that "developers suck" and on a bad day I would say that the author is giving them too much credit. On a good day I realize that the issue is much deeper and that we hire technological magpies who want to chase shiny new objects (the javascript world is probably the worst at this). On a really bad day I realize that our entire industry is "wrong headed" starting from hiring. We test people on "can they write code" but no one ever asks the people walking through the door to read 50 pages of it. To find the one misplaced semicolon just to see if they picked up on any of the logic traps that were littered all over the other 49 pages.

So to you sir (or madam), I raise a glass for at least trying to build tools for me, the guy who fixes the "gnarly bugs" in crud apps. I wish you would go back and finish your work, the 6 of us who really appreciate you would happily buy you a beer.

The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in. It's a hard career and industry for this reason. We all know the systems we work on are full of subpar choices, design flaws. We all know the delta between what it is an what it could be. But almost nobody can reconcile that gap on their own. And multiply that by entire teams. It's orders of magnitude less likely an entire team is that skilled and enthusiastic.

I have three philosophical perspectives that help me better frame the software industry.

Number one: People aren't stupid. Rather, things are hard.

Number two: Writing software is like doing a trillion piece jigsaw puzzle.

Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.

"The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in."

It's not just that, but the typical company is too busy chasing new features and fighting fires with understaffed engineering teams on constant death marches to unrealistic deadlines. Nobody has time to do things right, though many of us want to. Instead the systems accrete endless layers of bandaids and half-assed solutions which everybody hates but no one has the luxury to fix or solve properly the first time.

You need strong engineering management to communicate the business advantages of well-made software. The benefits of good software are exceedingly hard to measure and understand as a businessperson, because design is a lot like risk management: you can't measure the cost of the disasters that don't happen. The only substitute is to communicate thoroughly on the business's own terms and to show examples of payoffs whenever they're tangible enough to make the point.

Number 3: MOst SOftware Engineers are bad at their job, they aren't anything close to Surgeons.

This I think is the reason most software nowadays is garbage, there is a lot of low skilled people creating it. I blame bootcamps and dumbed down CS courses for this. It's not necessarily the developers fault, they can't improve on the things they don't know they are bad at.

Computer Science curriculum was never intended to produce competent software developers. You're complaining about the wrong problem.

Out of curiosity, do you consider yourself the exception or the rule?

Number 4: there's never enough time.

People who could do better are pushed to ship a feature tomorrow because hey it's simple and it can't take more than a day or so, right? And then we have the next feature..

I haven't seen anyone pull off a large code base without ever going back and reiterating on the design and refactoring things as needed in an attempt to try and maintain the qualities we (developers) would want in software. Unfortunately there are too many places in the commercial world where the time for that work is never given.

In my experience the people who have said "I didn't have enough time" if they were given infinite time, wouldn't have done any better a job.

Funny, because aside from some cheap tools and coffee, time is pretty much the only thing a developer needs to do anything.

What I've experienced is that in open source projects, patches get rejected and rejected until everyone agrees they're good enough, and sometimes that takes months or years of iteration. Not good enough? Spend more time on it. That's how things improve.

Wanna write a test suite? Need time.

Wanna run a battery of static analysis tools and inspect their output? Need time.

Wanna run a fuzzer? Need time.

Wanna document things? Need time.

Wanna rework the buggy guts of your application? Need time.

Want to figure out the heisenbug that rears its head every once in a while? Need time.

Realize the API the guy before you wrote is crap and needs to be rewritten, and all components that depend on it need to be updated to match the new API? Need time.

Want to formally verify that the core mechanisms of your critical application are correct? Need time.

Need to figure out and rewrite the part of the program that slows everything down to a miserable crawl? Need time.

On the projects I'm on right now, I could think of months of stuff to do to get them to a state where I'm somewhat happy with them.

Every goddamn thing just takes time and not much else. And IME the commercial world presents the same problem over and over again, far too often: you don't have time, you can't bill your customer for all the work that should be done (if good software were a concern), and you can't block the thing they want just because you first want to fix three other things that'd take a while to do. Crap just accumulates, because people build in a hurry, on a bad foundation.

Even open source projects where deadlines and "we sold this as an item that takes three days of work" aren't a thing, time is always a problem because there's only so many people putting hours in.

"Never enough time" is an artificial construct. Set up a pace that gives you a manageable output, nobody will be able to measure you precisely anyways.

Then suddenly you learn to do the right thing ("right" means the one that is not stressing you out) faster. And you're a happy person.

How many managers have you heard giving importance to refactoring or maintenance? (Some pay lip service, but very few actually make sure that its gets done in my experience).

> How many managers

Or customers, for that matter. Sucks when there's weeks' worth of reworking guts, refactoring, and testing to be done and you need to bill someone for the time.

I don't think that they are the experts in this kind of trade-offs.

"Even systems with well-defined architectures are prone to structural erosion. The relentless onslaught of changing requirements that any successful system attracts can gradually undermine its structure. Systems that were once tidy become overgrown as PIECEMEAL GROWTH gradually allows elements of the system to sprawl in an uncontrolled fashion." http://www.laputan.org/mud/

There is that old adage "perfect is the enemy of the good", well I would argue that "working is the enemy of the good" as well. When your "average" engineer spends 2 to 3 years at a company how often do they have to pick up the messes they make? Rarely? Never? It doesn't pay to be a good citizen in that case - it pays to be (as another poster pointed out) fast.

> The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in.

If software is hard, and average people are not ~willing~ to put in the work to make it less so, doesn't that make them lazy? Willfully and willingly lazy at that.

> Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.

Look around you,how many engineers do you know who's code you would be willing to bet your life on? The problem is that every engineer thinks they are a surgeon (and a good one at that) when many of them wouldn't even make good nurses.

It doesn't make them lazy. You're not lazy for not wanting to hike the Oregon Trail.

No, it simply makes good systems brutally improbable.

Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.

> It doesn't make them lazy. You're not lazy for not wanting to hike the Oregon Trail.

If I walk the 1/4 of a mile flat loop at the park down the street from my house every day and call my self a hiker I would be a liar.

Really were talking about Dunning–Kruger, "If you're incompetent, you can't know you're incompetent". I go back to "stupid" as the clear winner, and add in that they are too lazy to get out of that rut (put in the work) and that the title "engineer" becomes a willful or delusional lie.

> Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.

To your earlier point, we give everyone with a pre med degree a scalpel and a patient with a traumatic brain injury and say "go" were going to have a lot of bodies. The case is clear that we are far too tolerant of bad code and bad engineers, and we don't have roles like "podiatry" or "nurse" to demote people down to where they can contribute and we can limit their harm.

In healthcare, podiatry and nursing aren't demotions. They are entirely separate career tracks from medicine. Most nurses and podiatrists do those jobs because they want to, not because they failed at becoming physicians.

> Did you know that you can run diff over ssh to compare and contrast the code on two containers

Good luck doing that with Windows containers and diffing the HKEY_CURRENT_USER hive.

Anyways, rollback the container and start over.

I haven't touched a production windows box in 10 years, and probably haven't run windows on the desktop for 15. Your comment did access enough of my windows PTSD to make me twitch. I have abused the ldap interface for active directory several times in those same years periods but I don't think that really counts.

The situation of the "live" (container) works and we can't figure out how to build a new one comes up more often than you think. Knowledge leaves the building. People run off in the weeds and then quit so they don't have to clean up the mess they just made and no one is willing to touch it. Some times an executive shits a kitten and your policy changes from "vendor everything" to "always use the latest and greatest version cause "security" . Sometimes in house staff are stupid, or incompetent, or lazy, or is smart enough to know that if they touch it they own it forever and choose to play one of the aforementioned roles. Sometimes the staff does know and no one in management listens to that guy or gal. Other times it is a minefield that every one dodges on their way to getting what ever else they need done, and no one addressed the elephant in the room till it causes a panic.

I think there are more versions and variations of the "we don't know how any more" tale that I haven't listed, and I am sure that my list is no where near complete however I'm sure you get the picture. The call aways starts the same, I always listen and nod and say not a problem, and use soothing tones to tell whoever is on the other end of the phone that the world isn't going to end. Some of them freak out when I ask for an NDA, and a development environment - many give in but some are unwilling to let me peek under the kimono. Most of the time I can tell them that "price" might get better (or worse) but it will be more accurate if they do so. It is good to get into a system, to talk to someone who can "solve your setup problems" because most of the time they will point you to a closet marked "dead body(es)" and in many cases your half way there. Sometimes it is simple and large (back to that elegant) other times there are a few corpses or an orderly grave yard, other times it is an Alderan level apocalypse and you shut the door and slowly back away.

At that point I can give a rate, and a guess at a time line for a fix (and yes I estimate like scotty from Star Trek). Most of the time the person on the other end of the phone is desperate enough to say yes. With that magic little word the hunt begins - and many systems are shockingly resilient to being ripped apart and abused. Often times things that look important can be ripped out, swapped out or shut off and things still happily churn along up to a point. From a global view you can quickly get to a city or block sized area and then the fun begins. The solution in more than half the cases is fairly simple, I'm changing code because requirements, directions and documentation artifacts have become more sparse as time goes on. Sometimes I'm quashing good ideas for the sake of consistency but most of the time its some sort of "cute" piece of code that is getting ripped out for the betterment of all involved.

Rare is it that I tell someone to really "start over", to burn the whole thing to the ground and not look back. Those closets with Alderan in them do exist and are frighting and fun when you find them. I do love a green field but I am wary of the company that offers me one because most of them make a whole separate set of mistakes on V2.

No one every reads any of the 475 billion lines of crap they are updating, who really knows if you are making things worse or better, or where any of it comes from. Most of us wouldn't eat a ham sandwich handed to us on the street by a stranger but will happily run code from 100's of other people on our servers with out even looking at it.

I spend my day on CRUD apps and disagree.

Often big CRUD apps with a lot of business logic tend to have higher level, "logic flaws" that generate bugs. In those cases having something like `rr` to trace the origin of certain values would be a godsend.

I think these apps tend to be very wide in breadth, where instrumenting the entire system can be hard. Languages like Python (with `pdb` being a single-line in the middle of a program) are great in these scenarios because it allows for a pretty exploratory strategy when debugging.

Unfortunately there's a lot of people who do "print statement debugging" because the tooling really isn't practical in most places, but it would definitely be useful.

In general, I find a sense of learned helplessness among coworkers. They don't feel like they can get much useful out of the debugger.

In a way they are often right, but in part because it's a self-fulfilling prophecy. If you don't use the debugger much, then excessive amount of indirection in many codebases is academic to you, the way (unfortunately) people think of color blindness, accessibility, hell internationalization for some. It's abstract, not something you have to deal with all the time. It doesn't affect you personally so you just ignore it (in other words, complete lack of empathy).

Once the codebase is full of shared mutable state, lasagna code, lavaflow antipatterns, callbacks, handlers, etc, then it really does seem like it's easier to pepper some print statements around and pray.

It's coherent, but wrong. Like a teenager's justification for how they deal with their messy room. Your life really would be easier and healthier if you cleaned some of this shit up, man.

There's another way out, which is that better debuggers can be much more effective at punching through that complexity. E.g., rr/Pernosco-style reverse dataflow tracking lets you jump from an observed value directly to where it was set, which often bypasses a lot of byzantine control flow. When a callback is invoked, we could show you where the callback was set as well as where it was called. Etc.

Having done enough maintenance programming its safe to say that lots of developers manage to add complexity in to what should be fairly simple systems.

Yes a debugger is necessary for debugging a system that has been written by someone else.

I'm not sure it's specific to "debuggers" or "CRUD", I've noticed a lot of developers are just uninterested in advancing their tooling, they don't use IDEs, they don't learn keyboard shortcuts, they don't try to understand build tools. Just just build simple habits and stick to them without stretching themselves to things that might multiply their productivity.

CRUD apps have their own kind of complexity, and would gain a lot by some kind of data-based debuggers that instead of focusing on control flow said where data changes (from what, into what), where it's coming from, and where it's going to.

I wish CRUD apps were as easy to do as a operating system or something.

Seriously. I do this for living. For relaxing, I made a language in the side, because is MORE relaxing.

---

In the case of debuggers:

1- Yes, most debuggers suck not only suck, are near useless. What exactly can I do with "0Xfff23dfblabla" when I try to poke at a value?

This simple thing fail almost everything (.NET, swift, rust, java, ...)

Ok-ish: python, foxpro, js, probably other scripting langs?

2- Skip over thirdy-party and/or standard libraries and focus on MY code. Too much stupid "step in/out" when get in cross calls to code I'm not interested. This also mean better filtering of modules? Libraries?

3- Be INTEGRATED with my logging or tracing calls! print debugging is so useful, but is disconnected from step debugging!

4- Have search capabilities: Let me search in the debugger for vars, functions, etc to step in.

5- Have all the breakpoints on view! I wish I can enable/disable breakpoints with simple clicks without jumping around code.

6- Yes, I wanna look at that array/map with a MILLON values there, Imagine if computers could search stuff like that. Also, how nice if looking at/scrolling(remember, the debugger have not search capabilities) a array of few hundreds not kill my 32 GB machine..

This is also key. I fear A LOT to debug if it have too much variables or if some of it hold large values, like above 20...

Also: Allow to put simple invariants: Stop when this function is called N times, or when this var become Y, etc

This is some of the minimal stuff. I have not use rr or similar but if I can't see simple values, no opaque, can't filter code out, etc is the same pain but with more sophistication, I fear...

Some other stuff, nice but maybe is ask too much:

- Diff of values. Most debugging is to see what have changed (also nice: only stop IF changed)

- The debugging overlay the editor. Put the values above the vars, color/mark the touched lines, tell me how many times X have changed...

- Pull in a SINGLE view all the code on the stack, ie: Instead of jump around dozen files, put the dozen call in a single view and jump into that instead.

- Find a solution if thing are around threads. What if I have 100 threads?

- Have a command window (alike terminal) as useful as FoxPro 2.6 DOS. Something close today is Jupiter or the console in the JS debug tools. But this requiere integration that is available on FoxPro and maybe not cross easy to other langs

A step debugger is just a part. Also:

- Structured logging is good... but where is the log viewer for that? How collapse all the noise from logging that is only a repeat of the same stuff? How filter to the section where ERRs/WARns happened? What can I do with a 1 GB of logs to truly find the issue?

- How put tracing that can I enable/disable at runtime?

- How put tracing at runtime, if I don't have put the code for tracing before? Could I say "Trace before/after the code enter this function call?

Heh. Although it's currently really only for system languages (so doesn't do much for your #1), you're basically describing the feature set of Pernosco.

Sounds to me that VS/.Net with OzCode installed does most of what you want.

Reply on news.ycombinator.com