Why are we still tailing log files? | Hacker News Reader

Why are we still tailing log files?

philipotoole.com

24 pointsotoolep19 comments

This is such a weird article. He's wondering why engineers ... look at log files? He's wondering why people still just manually look at a file instead of paying some bloatware SAAS service to use AI to badly analyze your own logs? What is this? Just a terrible attempt at shilling for Loggly or something? I'm so confused.

No I think the real insight is that production systems benefit from professional log monitoring tools more and dev environments benefit from real time light-weight on-demand immediacy of tailing.

He is increasing our understanding and awareness and warning us of the trap of tailing production systems because we get so used to doing it in dev.

I personally replaced tailing logs with lnav (lnav.org, also in Debian repos). Coloration, decompression, filtering, etc. with no learning curve made it an easy transition for me after hearing about it on HN. It isn't nearly so featured as a log aggregator like splunk, but it was a total tail replacement for me.

Ha, hadn't seen lnav before. Looks interesting. Proves my point, tailing will never die. :-)

Well, now that I look it, maybe it's not exactly a tailing tool. Interesting nonetheless.

Can you elaborate on why you don't think it's a tailing tool? lnav monitors the files it was pointed at and will load new data as it is appended or reload the file if it has been overwritten. Also, if you are scrolled all the way to the bottom of the view, it should automatically scroll the view to keep the latest messages in the view.

How do you know what to search for if you don’t tail the logs first?

We are? 550 words and no mention of Splunk? I might be preaching to the choir here, but if you're at the cattle-not-pets stage, you owe it to yourself to setup some form of centralized logging. Splunk (no affiliation other than being a happy user) is the premier application in this space, as evidenced by their pricing model, but there are open source (and also proprietary) alternatives available.

In particular, Splunk's query language makes it easy (once you figure it out) to build graphs, so a pie chart or bar graph of, say, how many requests across the whole fleet are hitting a particular endpoint.

I agree about the centralized logging -- it's my last point. My bigger question is why is anyone tailing log files anymore when the industry has built such a huge number of tools. It's curious, but tailing still has its place, at least emotionally.

That's why I'm such a huge proponent of Splunk, by name (and again, no relation, just a very happy user). Because of how easy it makes it to create graphs out of log files, the phrase "Tailing is dead, long live tailing!" is simply not true. After seeing that power of centralized logging, most people I've talked to are converts (except for the cost). And perhaps that's why tailing is still so popular. Deciding to invest and learn yet another language, especially a proprietary one that your next employer may not use, is a hard choice to make, given how much other stuff there is to learn.

In Splunk, something like "sourcetype=your_sourcetype | timechart count by status" gives you a chart of HTTPS statuses (200 vs 404 vs 503) over time. After learning that the language it becomes intuitive and is a really really powerful when trying to analyze logs because a visual graph is just so much better than tailing log files. If your centralized logging tool doesn't give you same access to graphing as Splunk does, I'm sorry but it's not good enough to replace 'tail'. Splunk really is the state of the art here and I've yet to see anything that's competitive. (But tbh, I've not looked very hard - the only downsize with Splunk is its cost once you get over a certain size.)

I don't know what audience this is for. I imagine most folks know this if they have concepts of observability and monitoring. For others, or when working on side projects, tailing is a perfect fit.

It might be useful to add a few tips for how to tail better, like using `journalctl -f` or `less +F` or ways to tail multiple files.

I don’t want to memorize what a certain graph does after I build it. I don’t want to fight ambiguous menu items. I don’t want to memorize a click path through a web app with dozens of menus and toolbars just to end up at yet another dialect of query language I’m forced to learn.

Oh, and: not to speak of that one person somewhere in the chain of command who keeps barking out orders to randomly disable Splunk ingestion for individual instances due to cost.

Centralized logging gives me headaches and pain.

Splunk is mentioned in the image showing 30 or so other logging services. It’s a crowded market.

Exactly, I didn't mention Elastic either, another hugely important player.

My server went down unexplainably and I manage to bring it back up. I'll be tailing every single log file I can find until I get a clue of what the problem is.

Am I wrong?

What is about your deployment or setup that you don't use your log search system instead? Assuming you have one? You could search all your logs files at once (for errors, for strack traces), instead of manually tailing each one, one after the other. Is that you don't even know what to search for?

You are making assumptions that the log upload system is working in the first place. It's just another moving part in the equation. Since logging normally shouldn't affect production traffic latency, you usually want to offload responsibility to a separate process/sidecar/etc. That requires intermediate storage as logs are generated, which can fill up (disk or ram, it's all finite). Since you don't want just any person sending logs for processing, you now have to ensure this separate process is authorized to upload data, which means you probably have another security token in play. The search system must index the data in near real time, which may lag due to data backlogs, service disruptions, etc.

It's hard to beat the power of a live log tail when you need it.

Yeah, agreed, if the logging agent (or whatever) isn't installed or running yet, it's going to be hard to get the logs into the centralized system.

> That requires intermediate storage as logs are generated, which can fill up (disk or ram, it's all finite).

That's also an argument for centralized logging. Don't store the logs anywhere near your production systems. We've all had the experience where a machine goes down because the disk is full.....due to log files.

Low latency is even more important than the article suggests.

If you have to wait 5-30min to view your logs through alternative tooling, you've added significant slowdown to your hypothesis/verification loop

Reply on news.ycombinator.com