Lnav Logfile Navigator | Hacker News Reader

Lnav Logfile Navigator

lnav.org

156 pointsalexzeitler30 comments

LNAV is great, i’m so happy to have found it recently.

I’ve made a Caddy server lnav configuration file a while a go, for who’s struggling to parse the default JSON logs produced by Caddy.

https://gist.github.com/vjanssens/3c6fb8393d87346323d939f172...

For anyone looking for a similar tool but with a Web UI I recommend taking a look at Logdy (https://logdy.dev/, https://github.com/logdyhq/logdy-core) Ps: I'm the author

Thank you, that's interesting. A small feedback about the website - it uses the "Space Mono" font for the installation snippet - it is a monospace font, but surprisingly, it collapses "fi" into a ligature, making the font subtly and weirdly non-monospace. This is wrong.

That's interesting. Would it be possible to embed it in a go program to have an easy way of visualizing logs out of the box?

not yet, but its one of the items on my roadmap

lnav is something I keep coming back to over and over really _wanting_ to like, but I've never managed to figure it out. I'm not sure exactly what the problem is, but I find the docs confusing and incomplete, and I always end up getting stuck and going back to Vim and/or VisiData.

Does anyone have any good tutorials or resources apart from the official ones?

I'm the author of lnav .. and not a very good writer, apologies.

I guess my main question would be, what are you expecting to get out of lnav? I use it primarily for merging log files together and just jumping around trying to understand what was happening. It has a bunch of other functionality, like using SQL for analysis, but that's not something I use regularly.

Really, a lot of the benefits of lnav are automatic, like uncompressing files, detecting log formats, tailing... So, if that's not something that comes up for you, it might just not be the tool for you.

I actually have this "not getting it" problem with VisiData/multitail. I start them up and they don't behave like I would expect when pressing hotkeys.

Thanks for writing lnav, it's a fantastic tool. I constantly take advantage of the automatic benefits you mentioned, and also love the ability to navigate a log minute by minute, interleave multiple logs within the same timeline, navigate through errors or warnings, and how easy it is to deal with spammy logs by using filter-in and filter-out. Thanks!

lnav is amazing and I use it often. I do have a list of gripes where I think it could be improved, so I'm just going to dump them here in case you're interested:

- regex101 support for quickly defining custom formats is just awesome. Versioning support is slightly broken however, probably because regex101 changed something, so there's no easy way to update the format once you've initially imported it.

- I feel like there's missing opportunity for integration between various features.

  - There are lots of different filtering capabilities, but there is no unified treatment of them. For example, `:hide-lines-before` and `:filter-out` are at their core the same type of operation: filtering. I should be able to pull up a list of all filters that are currently active and easily add new ones and toggle or delete existing ones.

  - I would expect to be able to create a new view of the data using SQL `SELECT`. A select statement is fundamentally about filtering out some rows (log lines), which feels like a filter, and selecting some particular columns (log fields) and hiding others. The latter point seems like it could be something that should be handled when https://github.com/tstack/lnav/issues/1274 is resolved.

> - regex101 support for quickly defining custom formats is just awesome. Versioning support is slightly broken however, probably because regex101 changed something, so there's no easy way to update the format once you've initially imported it.

There is a `pull` sub-command and it looks like it still works. Running the following will generate a patch file with the updated regex:

    lnav -m format <format-name> regex std regex101 pull

It creates a patch file since the original file might've been modified.

> - There are lots of different filtering capabilities, but there is no unified treatment of them. For example, `:hide-lines-before` and `:filter-out` are at their core the same type of operation: filtering. I should be able to pull up a list of all filters that are currently active and easily add new ones and toggle or delete existing ones.

Adding the time filters to the "Filters" panel sounds like a reasonable request. I've added https://github.com/tstack/lnav/issues/1275 to track.

> - I would expect to be able to create a new view of the data using SQL `SELECT`. A select statement is fundamentally about filtering out some rows (log lines), which feels like a filter, and selecting some particular columns (log fields) and hiding others. The latter point seems like it could be something that should be handled when https://github.com/tstack/lnav/issues/1274 is resolved.

There is the `:filter-expr` command (https://docs.lnav.org/en/v0.12.2/commands.html#filter-expr-e...), have you tried that?

> I start them up and they don't behave like I would expect when pressing hotkeys

That's funny, because that's been my experience with lnav! Not saying there's anything wrong with it though.

It's been a while since last time I tried it so I don't recall the exact stumbling blocks I ran into, but I think it was mostly around hotkeys not doing what I expected, lnav not recognizing log types I think it should have (Apache/Tomcat), and not correctly loading custom log parsers.

If you don't mind next time I try it I can give you more concrete info.

> That's funny, because that's been my experience with lnav! Not saying there's anything wrong with it though.

I tried to use the hotkeys from less/more/vim so that it would be somewhat familiar. I think people are frequently tripped up if files are not recognized as a log and just treated as text. Files treated as plain text are separated from log files, so it can be a bit confusing. Not entirely sure how to improve the experience there.

> lnav not recognizing log types I think it should have (Apache/Tomcat)

There are quite a few log formats builtin. But, since log output formats can be customized by admins, it's possible they deviate from the builtin ones and things won't "just work".

> not correctly loading custom log parsers

I've tried to improve error messages a bunch[1] and make it easier to trouble shoot configuration issues[2]. I'm sure more could be done, I just don't quite know what folks are tripping over without feedback.

> If you don't mind next time I try it I can give you more concrete info.

Feel free to file github issues or email support@lnav.org

[1] - https://lnav.org/2022/08/04/pretty-errors.html

[2] - https://lnav.org/2023/08/04/config-dump.html

You're too modest. I wrote a custom format correctly on the first try; CouchDB. Keys are chosen rationally. People won't care about SQL until they realize that the later lnav versions allow `;select * from access_log where 0x00 in decode(log_body, 'base64')`

it's worth it just for the navigation helpers alone: '2' will bring you to the next 20-minute-after-the-hour entry (pressing again takes you to the next hour+20 minutes, same for the other 0-5 numbers). E will bring you to the next error, W to the next warning. And O will take you to the next entry with the same correlation ID (request/session/whatever .. you define it). And adding shift to all those keys reverses the direction. And naturally we're just scratching the surface.

I recently started appreciating lnav with JSON logs, where you can recreate a normal log-like experience by picking fields that are displayed (but you can still press a key to see all the JSON fields when you need them). I do wish there was support for switching formats so I could switch between different "views" over the same data, maybe it will be possible someday :)

lnav was a godsend with a particular project several years ago where we had a server bombarded with IOT messages and had to create some order from all the chaos. I actually went and donated some money to the project then, it really made my life easier.

> I do wish there was support for switching formats so I could switch between different "views" over the same data, maybe it will be possible someday :)

I created https://github.com/tstack/lnav/issues/1274 to remember this

I just found this tool a few days ago using GPT-4o looking for a better way to navigate and search logs. I did try it now and it looks great.

The histogram view could be improved to use a highlighted center line instead of the top line, but its quite helpful (once you read about SHIFT-I)

I tried it recently but got overwhelmed by its interface. I didn't understand all the colors and what I could do at each point. I'll retry later as it seems worth it as replacement of grep/sed/.... , but I'll have to reserve enought time to slowly go through it.

Would anyone have a recommendation for a log mining / analysis tool? I am particularly interested in automated identification of message sequences across multiple users, but anything beyond a simple log browser would be helpful. (I work with hundreds of GBs of Logcat logs.)

Not a CLI tool, but I recently started using https://klogg.filimonov.dev/ klogg (which seems to be a successor of glogg) more and more often.

Is there something like this that's lightweight and works well on (non-WSL) Windows?

I have Windows Servers with tools that create text-based logs and it would be nice to have something that could tail them.

If you just want old-school `tail` and `less` you can get them with Cygwin.

These don't support multi-file or highlighting like `lnav` does, but even on top of Cygwin they're very lightweight.

LogViewPlus (https://www.logviewplus.com) is very similar to lnav and built for Windows.

Baretail is what I have used allot. Really old but works good. Like the simple config to color lines. https://www.baremetalsoft.com/baretail/

notepad++ is great for log files. it handles massive files with ease. I wish it ran on non win systems…

It's still just looking at one server right? If you have a fleet of webservers, or microservices in containers, ELK or similar seems like a requirement.

Pretty much, yes, it's not for dealing with a bunch of servers.

There's some basic support for tailing files on remote x86 machines (https://lnav.org/2021/05/03/tailing-remote-files.html). But, again, just small scale stuff.

I use it on my development machine and for going through logs attached to bugs. Those use cases aren't served by something like ELK/splunk/etc.

I ended up using vim for log files, it recognizes messages and log files and highlight them automatically and I can also edit them if needed.

> I can also edit them if needed.

That doesn't seem .. wise. lnav has support for filtering, bookmarking, and attaching tags/comments[1] to log messages so that editing the log file isn't required. The filters, bookmarks, tags, and comments are saved separately so they can be restored when the file(s) are reopened.

[1] - https://docs.lnav.org/en/v0.12.2/usage.html#taking-notes

Best log tool !

I've been singing lnav's praises ever since I first discovered it back in... 2016?

Reply on news.ycombinator.com