Hacker News Reader

Comment by superbatfish | Hacker News Reader

The redirect() method shown in this post only works for output that is written via the Python sys.stdout handle.

If your code links against C/C++ extensions that have internal printf calls, then you need something lower-level, which can intercept the data at the file handle.

There's a great solution for that on StackOverflow: https://stackoverflow.com/a/22434262/162094

View on HN

Even on the Python level it seems much better to use the logging module and attach a handler to sys.stdout. if all you need is to print some stuff to the console just attach a handler with logger.addHandler(logging.StreamHandler(sys.stdout)). You can attach other handlers for files or to send the logs via some network interface and you can silence or filter as desired. You can even attach metadata so that you can process the messages in different ways depending on what you want to do with that metadata. This can all be done separate from the calling code so the main logic just need to know log this message at this severity and all the processing filtering and transmission will be handled by the logger.

Could you give an example of "attach metadata"? Are you talking about attaching metadata to the logger object or handlers? How is that done? If I recall correctly, the only thing you can send with an individual log event is a string message, and an log level, no other metadata.

I looked into this because I love structured logging, and I wasn't able to figure out how to make that work easily with the default logging module.

A solution I have used is to creat a logging recordfactory similar to what’s described here: https://stackoverflow.com/questions/59585861/using-logrecord.... There’s also logging adapters as described here: https://docs.python.org/3/howto/logging-cookbook.html. The docs how-to also describes how to add contextual information via a filter. Not knowing your particular setup I can’t say how well any of these will work for you, but these gave me all the tools I needed to selectively attach IDs and other metadata to celery task logs so I can easily grep for the task ID or a few other unique pieces of data and easily see the whole log for a given task or for tasks that all used the same device, etc.

I'm a big fan of the logging module, but it can't solve the problem of silencing or redirecting noisy C extensions. For extensions which you control, maybe the ideal option would be to offer logging configuration, or callbacks, or even integration with Python's logging module. But for externally written extensions, your only option is lower-level manipulation of the relevant file handles. Python's sys.stdout and logging module are powerless in that scenario.

Reply on news.ycombinator.com