Comment by jiggawatts | Hacker News Reader

Comment by jiggawatts | Hacker News Reader

I just had a conversation with a dev manager about how the vast majority of dev teams I've encountered use maybe 5% of the tooling available to them.

Debugging using just debug tools in the IDE is just scratching the surface.

A demo I like to do is Azure Application Insights, which takes memory snapshots of web server processes in production whenever they throw exceptions. Combined with source indexing, it lets you "debug" a release build from a month ago in a matter of seconds.

Just recently I was trying to do the sales pitch to a semi-technical product manager, and he mentioned that they were seeing "occasional login issues". While on a video conference session we managed to repo the issue: after entering a valid password, the application promptly reloaded the login prompt without any error message.

Sit down and think about how long you would estimate a bug like this would take you to fix: "Occurs in prod only, infrequent, no error message, just does nothing."

App Insights helpfully had a "debug snapshot" available for the error. It was literally a single click to load it in Visual Studio. It jumped straight to the line of code that crashed, where there was a comment along the lines of "we may be too lax in parsing this". The snapshot let me inspect the variables and... yes.. a bit too lax they were.

I did this in about two minutes while chatting with the manager. Boom. Solved!

I've had similar troubleshooting sessions in the distant past that took weeks or even months to resolve, because I just didn't have the tools available to me.

Tools make all the difference, and anyone who argues against this just doesn't "get it" in my opinion. I don't care how wizard you are at shovelling, you aren't going to out-perform the back hoe.

The devil is in the details.

You have to understand what the tool does, how it installs, and if it costs money, lordy you have to wait for it to be approved by 10 levels of management.

Another problem is that log indexers and the aforementioned Azure insights almost certainly would require boatloads of storage and processing costs. Programmer time is fundamentally disrespected in organizations, and debugging/QA is even moreso. Companies basically throw a laptop at both of them and say "good luck with whatever free tools are out there".

But your example was basically a stack trace, from Javaland, I wouldn't call it revolutionary in tooling and is ground-level features of Javaland IDEs for about a decade or more, but maybe C#/whatever is different.

I won't disagree with the core point, since I complained about a lack of visualization tooling for multithreaded apps above. We need more tools and tooling awareness.

> "Occurs in prod only, infrequent, no error message, just does nothing."

With proper logging, a few days?

- pull up bug report

- figure out user from logs usually based on what he was doing plus IP address + time issue ocurred

- look at what he was doing and what went wrong

- look at code, see what could possibly go wrong, fix all found issues, apply patch, test and deploy. (alternative - see what recent changes were done there from last few deploys to find possible culprits)

Now imagine small issues at scale, millions of users, lots of servers and services, complex architectures... where you can't even reproduce. Those are fixable with logs. Debugger won't really work unless you get a crash dump. Memory snapshot as you say might work - but to get the snapshot I assume it pauses the service to generate memory dump and takes it offline... so equivalent to crash/assert?, otherwise I'd be very curious how to get instant snapshots :)

Essentially it forks the process. The original “keeps going” unaffected, the fork just sits there doing nothing as its memory is written out to disk and then killed.

It’s not perfect, but it’s close enough to be indistinguishable in practice.

It “feels” like you attached the debugger to the production process. You get the heap, the call stack of every thread, etc… It makes it trivial to inspect things like “what was the database query and what result set did it return” or “which specific heap object was null”.

> Essentially it forks the process.

That is a neat idea, I'll definitely use it in the future where possible.

> The original “keeps going” unaffected

That's not really the case, copy-on-write pages means any write after the fork requires a new page in ram.

That will mean increased memory usage and slowdown of writes in addition to fork costs.

On something that uses a lot of memory and writes a lot it will be significant.

If the service is already optimized to run with just barely enough ram to keep costs down then this fork-and-dump is going to oom in prod.

> It “feels” like you attached the debugger to the production process.

Yeah, memory dumps are very helful, gdb + dump usually means bug is fixed barring any corruption that happened long before point of manifestation or business logic issues.

Is this just a pitch for Azure Application Insights, or do you have other example backhoes?

(If it’s just a pitch, it’s a great one! I just don’t use Azure currently).

I don't work for Microsoft and I'm not selling anything.

The point is general. Any time you hear terms like "time travel debugging", you should get excited, because someone out there has worked very hard to make your life easy.

It's just ridiculous how much effort has gone into some of these tools, and then has been ignored. It's a waste that bring tears to my eyes. You have all been given light sabers, and you hit each other with blunt sticks.

> and then has been ignored

They're not ignored, most are just not feasible in production due to both performance and memory overhead. They're still used for tests to check for issues (valgrind?)

If you can point to something that actually works properly in production without much overhead I'd use it in a heartbeat.

Also, I'd say proper logs are a form of time traveling debugging... so a system built on logs might work :)

I have heard it said[1] that the complexity of our modern government is a consequence of the computerization of its bureaucratic functions. In effect, the complexity of implementing crazily esoteric laws used to regularize how crazily esoteric the laws could be. No more.

If time-traveling debugging becomes table stakes then we will build systems that can only be debugged by time traveling debuggers. Specifically, because we'll lose our healthy fear of complexity beyond what logging and a normal debugger can manage. And that fear (more often called "experience") is about the only thing that causes the greybeards to push the non-greybeards for simple, elegant solutions.

I think we collectively go further, faster without the magic because magic makes us willing to discount complexity.

But I am something of a Luddite.

[1] By my father, a now retired Social Security Administration manager, roughly 1980 to 2015.

rr is time travel debugging, but you can't use it in production or running desktop applications normally, since it adds overhead, disables parallelism, and doesn't snapshot so replaying a bug that takes 3 hours to occur takes 3 hours.

> I don't care how wizard you are at shovelling, you aren't going to out-perform the back hoe.

80% of the time you want to know what's 10cm below the surface, and a shovel is every bit as good (better, since you don't have to go and start the backhoe).

If you're in the position of working on simple problems, it's very hard to justify the work of learning these tools. Back when I worked on a multithreaded database engine, I would have killed to have `rr`, but when I'm trying to get simple procedural code right it really does seem like a lot of hassle.

I wonder how much of dev teams using only 5% of the possible tooling is due to ignorance and how much is due to being burnt by believing claims about the magic of tooling that wasn't so magical (or was magical until the vendor went bankrupt), so they just stick to the classics.

+1 for AI (Application Insights). Is easy to set up , super powerful and has its own query language Kusto.

We did all kinds of things like like custom performance metrics for processes.

Then in my next gig we used sentry.io. Its ok as far as it goes but it's not a quarter of the tool AI is.

Related: Full story is an amazing front end recording and troubleshooting tool.

What language were you using here?

I completely agree. GUI debugging is the bare minimum and yet most people don't even do that.

In my experience it mostly happens when people that use very basic dev tools (e.g. Vim and GDB) set up a project. They don't use a proper debuggers so they don't take the effort to make it easy to use them, and it's often very difficult to retrofit later.

Reply on news.ycombinator.com