Trace your Python process line by line with minimal overhead | Hacker News Reader

Trace your Python process line by line with minimal overhead

github.com

56 pointsfywvzqhvnn13 comments

A problem I have with Python tracing is Django-specific: often my trace is a ton of Django internal code and doesn’t even point to my code that’s causing the problem.

Not sure if there’s a way to address this kind of issue. Or if it’s common elsewhere? Just this experience of “what I wrote is being run in the context of some library internals and so it spits out the internals.”

If you’re using django, try Sentry. It’s good at sorting out the noise from the signal, including separating parts of traces that aren’t relevant to your app.

The common solution is filtering. Almost all tracers will accept predicates to exclude Django per your example.

My issue is that the trace sometimes does not include any lines pertaining to my code. I’m really not sure how that can be true. I’ll have to save an example next time for the next time I bring this up.

I get similar errors with FastAPI, I believe it's due to compiled routes/code being run separately in the web server etc, it's really up to the Django and FastAPI devs to implement proper traces here.

I'm with you. I've had huge stack traces from Django, many many times, without even a hint of my own code in them. I've always assumed it's the extremely "extended" nature of Django - my class extends A which extends B and C, and C extends D, E and F and F uses a G metaclass and so on.

Any advantages/disadvantages compared to py-spy [1]?

[1]: https://github.com/benfred/py-spy

Advantanges - You cannot trace a Python process line by line in py-spy. Because py-spy does not attach the line's probe. By the way, py-spy is a profiler, beetrace is a tracer. - You can see the call/return hierarchy of a Python process because beetrace attaches function entry and function return probes.

Disadvantages - To trace the process, Python must be configured with the --with-dtrace option. Otherwise, you won't be able to trace it.

Looks interesting I will definitely try this.

For those that find this interesting, you might also like pysnooper - I use it all the time.

https://github.com/cool-RR/PySnooper

https://python.plainenglish.io/pysnooper-stop-debugging-pyth...

Does something similar exist for nodejs and java?

USDTs have significant overhead per active probe. I would not call that minimal. They are very cheap when inactive, but inactive probes can't do this.

If it can't be done cheaper then it's minimal; tracing line by line will always be expensive. I'm assuming (I scanned the readme & it didn't mention) what they've found is that USDTs are cheaper than the built-in `sys.settrace()` facility.

It can be done cheaper, although it may not be noticeable compared to the overhead of Python in general.

Reply on news.ycombinator.com