Stack Traces Are Underrated
karl.berlin
karl.berlin
Once you get comfortable with how they work and what information they contain, you can hit the ground running anywhere. Stack traces will teach you about the product architecture faster than anyone on the team can.
As you embrace them, you take the little bit of extra time to make sure they go well. For example, re-throwing exceptions correctly, properly awaiting results, etc. Very minor details that make all the difference.
A broader outcome of this enlightenment is preference for monolithic products. Stack traces fare poorly across web service and API boundaries. If you've only ever worked with microservice architectures, the notion of a stack trace may seem distracting.
Yes. People forget that the original concept of microservices, the AWS "everything must have an API", was to put in an accountability boundary across teams. Either the API behaves per its contract or it does not, you're neither expected nor really allowed to cross that boundary into the API to find out why it's doing that.
In an environment which is correctly doing "each microservice is a different small team", that helps. In an environment which is doing "one team maintains lots of microservices", this is nearly always an anti-pattern.
It's not hard to implement, it's just that the industry kinda gave up on using powerful RPC frameworks with language integration. gRPC is about the fanciest you'll get and that is a clone of Stubby, which was originally written for C++ in a codebase that banned the use of C++ exceptions entirely. So it doesn't try to solve that problem.
The trouble is that ties both ends to the same implementation language. Web-style RPC became popular because its simplicity meant it wasn't tied to any particular features of any particular language.
I hope that at some point someone standardizes a way to pass virtual stack IDs, exception data and other RPC metadata via HTTP headers so we can get back the sort of features you find in high end RPC stacks.
It's not hard anyway. You can just upgrade server frameworks to render the stack trace to either a string or a json data structure (or protobuf), and when an exception bubbles up you capture it, convert it to that structure, compress it, possibly encrypt it with a shared key (just to avoid leaks in case there isn't a proxy stripping the special header), base64 encode it and stick it into a special header. Then your HTTP client can be taught to look for that header in case of a 500, decode it, turn it back into an exception and rethrow it.
If you have a unified framework like Micronaut it's probably the work of an afternoon to throw together. The hard part is the crypto. The moment you introduce keys, corporate security teams will insist on things like frequent rotation even if it's just a backstop against badly configured frontend servers and not mission critical (if you leak a stack trace, ok, not great but not the end of the world). So it might be better to obfuscate in a different way that isn't meant to be fully secure, or detect if a request comes from an internal IP in a secure way, etc.
That's technically true, but the situation is not as dire. Many errors do not need stack traces. That so few carry a backtrace in Rust is mostly a result of the functionality still not being stable [1].
The I think bigger issue is that people largely have given up on stack traces I think, in parts because of async programming. There are more and more programming patterns and libraries where back traces are completely useless. For instance in JavaScript I keep working with dependencies that just come minified or transpiled straight out of npm. In theory node has async stack traces now, but I have yet to see this work through `setTimeout` and friends. It's very common to lose parts of the stack.
Because there are now so many situations where stack traces are unreliable, more and more programmers seemingly do lose trust in them and don't see the value they once provided.
I also see it in parts at Sentry where a shocking number of customers are completely willing to work with just minified stack traces and not set up source maps to make them readable.
I’m not sure there are many reputable modules on npm that minify without source maps, and if people aren’t using them I’d consider them to be making a poor technical choice, one that I would correct before contributing to the project.
Diffing two lengthy stack traces to find a divergence is perhaps the fastest way to debug a slew of bug types. Let alone just the ability to instantly click into a file/line even from console prints as you follow the execution path.
And my favorite part is being able to ignore / hide external modules and specific files in chrome’s debugger which allows for stepping through only your code, and evaluating much shorter traces. Something java needed decades ago.
When I do use print debugging I always use console.error to include the expandable stack trace as needed, I can’t imagine how slow it would be to not have that always, and have to resort to stepping and breakpoints to get around.
React is a good example of a library that is a transpiled mess when installed from npm. Sadly not the only one, there are many more popular libraries that look like this.
import asyncio
async def baz():
await asyncio.sleep(.1)
raise RuntimeError()
async def bar():
await asyncio.sleep(.1)
await baz()
async def foo():
await asyncio.sleep(.1)
await bar()
async def main():
await asyncio.sleep(.1)
await foo()
if __name__ == "__main__":
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
main_task = loop.create_task(main())
try:
loop.run_until_complete(main_task)
except KeyboardInterrupt:
main_task.cancel()
loop.run_until_complete(asyncio.wait([main_task]))
pass
[1]: the important line is "main_task = loop.create_task(main())"
At that point you have something like a "coroutine frame tree" instead of a "stack trace", where you potentially store multiple parent frames and source lines per coroutine frame. Could be presented something like:
$ python3 test_stacktrace.py
Traceback (most recent call last):
* File "/home/user/tmp/test_stacktrace.py", line 22, in <module>
| main_task = loop.create_task(main())
| * File "/home/user/tmp/test_stacktrace.py", line 24, in <module>
|/ loop.run_until_complete(main_task)
* File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
| return future.result()
* File "/home/user/tmp/test_stacktrace.py", line 17, in main
| await foo()
* File "/home/user/tmp/test_stacktrace.py", line 13, in foo
| await bar()
* File "/home/user/tmp/test_stacktrace.py", line 9, in bar
| await baz()
* File "/home/user/tmp/test_stacktrace.py", line 5, in baz
raise RuntimeError()
RuntimeErrorYou need to use the actual `await` syntax to get an async stack trace in node. Callbacks and raw promise work can't be seen by the async stack trace implementation which hooks into `await` points.
I can look at a stack trace, go "oh, function X is misbehaving after being called by function Y, from function Z", and work out what's gone wrong from the context clues, and other debugger info. As a game developer, with codebases that are big, semi-monolithic codebases, it's essential, especially when code crosses the gameplay/engine and engine/kernel barriers.
The languages that I work in that don't print useful traces are typically strongishly-typed system languages. So I miss them - sometimes having to step through offending lines of code in a debugger - but I also completely avoid a whole class of bugs that are responsible for most of my stack traces in Python.
TFA's example isn't one of these, but is a function that would have a return code checked and logged if erroneous. This class of bug also can't be inlined and makes an easy breakpoint-ee.
For smaller scale apps it's a godsend but I haven't worked in that environment in more than a decade so remote debugging is essentially useless in my work.
Sure, you can grep the log message but it can be difficult if it has some templating/formatting going on, and it can be pretty easy to end up with non-unique messages.
Also, for "normal" errors, you shouldn't need a stack trace. For example, "file not found" is, from the point of view of the developer an expected situation and should be handled with the same amount of care as it the file was present. You don't dump the internals to the user when you have successfully opened the file, so don't dump them when you haven't.
For unexpected errors (i.e. bugs), the crash, abort, panic, or whatever it is called in your language. These will usually give you a stack trace, or a core dump from where you can extract the stack trace and more.
What I would wish for however would be a standard feature in languages to display a stack trace on command. Many languages have it, but even when they do, they could be more prominent. This way, if you encounter an unexpected situation you want to debug without crashing and without a debugger attached, you can call it.
One problem with stack traces is maybe that they can be too verbose. E.g. if you print them for any warning you print to log (or stdout). Sometimes they will be extremely helpful for debugging some problem, but in many cases, you maybe don't need them (you know why you get the warning and/or you don't care about it).
You could also add more information to the stack trace such as local variables. That can be even more helpful for debugging then, but again adds more verbosity.
For example, we often use this to add information about relevant local variables: https://github.com/albertz/py_better_exchook
One solution to the problem with verbosity is when you have foldable text output. Then the stack trace is folded away (not shown in all details) and you can unfold it to see the details. See the DomTerm demo here: https://github.com/albertz/py_better_exchook#domterm
Some more on text folding:
https://github.com/PerBothner/DomTerm/issues/54
https://gitlab.com/gnachman/iterm2/-/issues/4950
https://github.com/xtermjs/xterm.js/issues/1875
https://gitlab.freedesktop.org/terminal-wg/specifications/-/...
Still, i do think returning the error as a return value is better than having a completely separate flow when dealing with exceptions. I like that it forces me to properly deal with an error and not just ignore it and think something like "meh, i'll get to this later". Because i will never "get to it later".
It allowed me to go some frames back - lines up, up in the stack. I don't recall the name of this debugger, nor what language it was. But I've never since seen this, yet very often wished I had it (for rust, javascript, python, mostly).
Did I misremember? Can such a thing exist? Does it exist?
Of course, this was in the hyper expensive Visual Studio Enterprise IDE, not something like VS Code, which high school me totally paid 4 grand for and didn't pirate at all.
These days, features like these are making it to free or cheap tools.
Merely being able to inspect the state of (local) variables further up the stack frame is a much more limited proposition, even if it can still be useful.
> yet very often wished I had it (for […] javascript […] mostly)
Both Firefox's and Chrome/Edge's devtools allow you to do that, don't they? Click on an entry in the stack frame and it takes to the corresponding code line and shows you the state of the variables relevant at that point.
They are most likely describing the much simpler feature of having the debugger drop stack frames (by force nuking the stack) and then starting over (with all the other side effects that occurred in the “prior” execution still present). This is a fairly common feature for exploratory debugging, but has the obvious downsides of leaving lingering side effects so is only fit for use in non-production environments at best like other “edit-and-continue” features.
You'll be dreaming of good old times of linear stacktraces.
Another tip: I have found that it is helpful to ask AI to "deeply analyze" (use those words) and think about the problem without providing a solution (say "don't reply with any code"). If you don't do that, it will take its first guess and then eagerly start outputing code that is still wrong and doesn't really identify or fix the issue. When you ask it to deeply analyze what's wrong and not reply with any code, it frequently finds the true underling problem, and then you can ask for how to solve it in the next step.