Hacker News Reader

Comment by gbuk2013 | Hacker News Reader

gbuk2013

This has been my experience. The pieces are simple, but being to able to understand what is going on in the whole system is not. Detailed, quality structured logging is very important, sent to a central location that can be queried (e.g Elastic Search / Kibana). I have come across several experienced developers who really don’t like he idea of verbose logging but in my experience it has been critically important for dealing with a mesh of micro services.

View on HN

"How does this even get invoked" and "Where would I go to find these logs" are the two questions that, if the answers are not obvious/obviously documented, there is a big problem with that particular system.

My current company for whatever reason does not value logging at all, to the point where for lots of our systems there just aren't logs/nobody has ever bothered to look for them. It's pretty astonishing to me, a person who has valued logging as one of the highest-value mechanisms to increase tracibility.

Prior to moving to software development, I worked 5 years in Support and consequently I have a murderous hatred of anyone writing software that is hard to debug and logging is a huge part of that.

I was actually greatly inspired by a product I used to support which had such detailed logs that you could troubleshoot almost any issue from the (rather huge) log dump. I try to do the same now.

I really wish all developers had to do support at the beginning of their careers and so they understand how to build things that the ops team can work with. Devops/SREs are one approach to this, but I'd really like to see it be more widespread.

I share your enthusiam for good logs, so am curious what made this product's logging great from a support perspective?

I wonder if they simply haven't seen a strong application of logging, and therefore don't know what they're missing.

Is there an opportunity in your daily work to sneak some good logging into a complex subsystem to show how valuable it can be?

For any new work that I do, logging will obviously be at the top of my priority list. Unfortunately, the systems that need the logging the most are the systems that I can only touch when they go down.

It's basically what happens when technically incompetent and inexperienced people wear management hats. If I was running a firm, the engineers would dictate to management, not the other way around.

I feel like this sentiment is easy to say, but basically impossible to implement.

Yep, you never know what logs you need until you need it. If there’s an economical factor coming into play (because proper telemetry will cost a noticeable amount of money), you can always choose to sample the more verbose logs.

Honest Q: Isn't that's where tracers like AWS XRay and Google Dapper come in? And somewhere down the line, solutions like Envoy/istio try to make the whole ordeal manageable?

You can implement distributed tracing with logs if every player in the system is propagating the headers appropriately. Tools like NewRelic APM and XRay will take care of that bit mostly for you.

Service mesh data planes have a wider scope, but they do make it much easier to implement distributed tracing (I think Istio includes this out of the box).

In the end though these tools can’t replace app logs because they only let you reconstruct what happens in between services, not internal to them.

Nothing cloud is relevant when your production network has no outbound access to the internet due to PCI. ;)

Not sure about the other two (will check it out) but we have have apps in 4 programming languages across several teams. Structured (JSON) logging is the easiest way of sending data to one place from literally everywhere.

Agree 100% - logging is expensive, in terms of performance hit and storage, but it is totally worth it.

In the medium to long term, the absence of logging is always more expensive than its presence.

Reply on news.ycombinator.com