You can't debug distributed systems with dashboards | Hacker News Reader

You can't debug distributed systems with dashboards

read.acloud.guru

Yeah, I'm sorry, but this is ridiculous clickbait written by hipster sysengs using fringe architectures to generalize about a gigantic range of complex systems and saying "all you need to do is sample some logs, guys, why are you still using dashboards".

You know what solves ops issues? System visibility connected to automation designed to diagnose root causes and provide simple solutions to common problems. Serverless adds extra complexity and points of failure, but that doesn't mean "you can't debug distributed systems with dashboards". Give me a break.

I clicked on that link ready to experience a revelation that was going to change how I debugged infrastructure problems.

I came away with nothing. I got nothing out of this article. I have re-read it twice now hoping that I was missing something, or had failed to understand some point.

The only point was that distributed tracing apparently solves all problems.

>System visibility connected to automation designed to diagnose root causes and provide simple solutions to common problems.

This. At the end of the say you gotta put the right stack trace in front of the right eyeballs so someone can connect the dots and figure out why your crap is burning down. Automation helps people who are not intimately familiar with every aspect of each system do that.

>but this is ridiculous clickbait written by hipster sysengs using fringe architectures to generalize about a gigantic range of complex systems

To be honest that can describe a lot of articles here

You can if your loggung structure is uniform, and includes a correlation ID. This happens every day... pretty simple actually.

This is literally the point of the article.

Charity is saying to structure your logs, add useful information to them from all portions of the transaction, and save the logs somewhere to query them across all fields in any combination.

That's what her product does.

This is in opposition to poor quality text logs, which typically contain almost no useful information, and time series aggregates, which either throw away all the context, or explode traditional time series databases when you try to add high cardinality fields to them.

Edit: I should add that "dashboards" in this context means the typical wall of time series charts you can buy from a bunch of different vendors.

Title is a bit clickbait, really.

Sounds simple. Unfortunately, having uniform logging structure in a non-trivial project is about as likely to happen as winning the lottery.

...only if you view software development like gambling. All problems can be methodically addressed; fixing log formatting is actually rather easy to track down.

Yes, in your own code, it is easily fixed... With third party code, like dependencies, it becomes more of a problem...

That's true. But you only need to re-format the events you care about from that dependency.

That's still a non-trivial problem, depending on how easy it is to adapt logs from a library; maybe with a proper logging facade, it's easy to attach those adapters to those logs, but I suspect most libraries don't have strong contracts around the formats of their logs, as it is. Maybe that's something to improve.

I use .Net and Serilog for structured logging. Each log entry can be a structure with name/value pairs instead of just a stream of text. Most of the time it’s JSON.

You have various outputs (sinks) where you can send your logs to almost anywhere - AWS CloudWatch to its own group/stream where you can trigger alerts, a NoSql database like ElasticSearch or Mongo where you can do queries against properties.

You can also have every log entry tagged with metadata like the application name/EC2 instance, AMI being used either globally or within a “context” (everything that’s called inside a using block).

Why anyone would do unstructured logging in 2018 is beyond me.

No it's not. You can start with something that snarfs data from your log messages and normalizes them. Then you just have to log enough data that you can extract it after. The normalization can connect IDs together to make the log data more accessible.

Now I have two problems: my app and a logging thing.

Console logging is the easiest but least effective method of debugging. Some people refuse to call it debugging because they believe a debugger or diagnostic tools are involved with the act of “debugging” (vs troubleshooting?)

Distributed systems mean you have to fall back to the most basic form of debugging.

You had better be getting something really good for that bargain. You’ve nearly permanently compromised your ability to reason about obscure bugs in the system in order to have it, so be sure it was worth it.

You need to take that basic form of "debugging" and turn it into instrumentation. Collect structured data and build tools to process it. Once you have a nice collection of data you can build profilers and execution traces, things classical debugging tools provide.

I've done it and the tools can work quite well. The same consistent instrumentation of your code also makes it easy to build really great dashboard monitoring.

All this time building tools costs money and adds complexity to your stack, so, yes, distribution had better be worth it for your business needs before you take it on. Pretty much like any engineering trade-off.

Do you mean something like this? (Rookout.com) https://youtu.be/bAxNWg6QYL8?t=62

What’s the pricing for the pitched product?

$70 per month and up, based on GB/day transmitted.

https://www.honeycomb.io/pricing/

I'm still wating for the day when these products use serverless-pricing too.

With CloudWatch you can literally do everything the article proposes just by sending your logs there - you get easy to create dashboards and alerts.

Nice, thanks for the info :)

Maybe I should to a course on CloudWatch. Seems like a powerful tool, but a bit convulted at times...

Such a massive unscrollable top bar that I had to open my browser's Dev Tools and remove it from the DOM it before reading the article.

Reply on news.ycombinator.com