Use the 'tail' command to monitor everything | Hacker News Reader

Use the 'tail' command to monitor everything

blog.robertelder.org

181 pointsbelter94 comments

If you're tailing logs, can I suggest that you try out the Logfile Navigator (https://lnav.org). It really is possible to do better than tail/less/whatever when you just want to look at a local log file:

https://lnav.org/2013/09/10/competing-with-tail.html

I've never heard of this. Looks great, thanks!

Was going to mention lnav also. You don't even need to install or compile it. Great for that server managed by a contractor.

This looks really awesome, thanks!

I think inline pretty-print of JSON/XML/YAML in a log entry.. plus histogram of date/time.. I do these things a lot using various grep/sort/uniq etc pipelines

Wow!

Sql queries on merged logs, histograms, json log formatting. Sweet.

If you want... a fraction of the features, but still a nice experience, and you want a viewer in your editor, and if your editor is Emacs, try Logview mode - https://github.com/doublep/logview.

I mention it both because I find it very useful and because, seeing Logfile Navigator, I now see that it desperately needs to have more features :).

How does it handle merging log files that are in different time zones? I wrote a small script to merge log files from lots of different network appliances a long time ago, it was extremely useful for debugging problems that occurred across our distributed system. The most unfortunate part was that the appliances all logged in local time and didn't include their UTC offsets in the timestamps :(

Timezones are not automatically handled by lnav at the moment, which isn't great. Timestamps are parsed and treated as UTC internally, which usually works out fine. You can manually adjust all of the timestamps for a file using the ":adjust-log-time" command:

https://docs.lnav.org/en/latest/commands.html#adjust-log-tim...

That can be useful for doing minor alignments if clocks are slightly out of sync or doing timezone adjustments.

You can also adjust the time programmatically by updating the "time_offset" column in the "lnav_file" table:

https://docs.lnav.org/en/latest/sqltab.html#lnav-file

Sorry I don't have any examples written up to illustrate this at the moment...

Thanks, I see. At least with `:adjust-log-time` you can make it work!

With my old log merging program, you had to supply a regex with groups for the different timezone components and optionally a UTC offset. That worked really well but was a pain to set up. Typically I was using it to look at the same format of files all the time though, so in practice it wasn't that bad.

I'm not really a C/C++ person but maybe I'll try and hack on lnav a bit and see if I can figure out how to add timezone support.

Nice.

This: https://lnav.org/features#query-logs-using-sql

Reminded me of: logParser https://www.microsoft.com/en-us/download/details.aspx?id=246...

Yes, Log Parser was the inspiration for that feature in lnav. That's why the column names are basically same between the two (e.g. c_ip, cs_uri_stem).

I recommend using `less <FILE>` <shift-F> rather than plain `tail -f <FILE>` because you can ctrl-c to stop the tailing and then search the text using /. You can then resume tailing again later with another <shift-F>.

I love to press enter a few times to add a few empty lines to serve as a marker. Plus, I set my terminal scrollback buffer to 10000 lines (not a problem nowadays), so cmd-F in the terminal works pretty well. And I prefer the natural/kinetic scrolling of the terminal backlog to less as a fullscreen tty program.

Finally, as the article alludes to, tail can monitor multiple files at once.

> I love to press enter a few times to add a few empty lines to serve as a marker.

Same! If it weren’t for this, I’d use less, as the searchability is often handy.

But it’s funny how just one tiny (and possible unintended?) “feature” turns out to be so dang useful that I keep using tail.

Yeah, right? It's not even really a "feature" of anything, just an artifact of how teletype support was originally implemented in UNIX, and yet I've been hitting the enter key for my own "marks" for decades.

> But it's funny how just one tiny (and possible unintended?) "feature" turns out to be so dang useful that I keep using tail.

"Dangling by a trivial feature"[0] seems to be a pretty common issue when people recommend 'better' software.

0: http://prog21.dadgum.com/160.html

less has its own system of bookmarks: use m followed by a lowercase letter to set a mark and single quote followed by a single letter to jump to that mark. I assume that this means we can have up to 26 bookmarks, though I normally don't go much beyond two.

You can use capital letters in less too, so at least 52.

I found myself doing this as well and decided to write a simple tool to do it "automatically" for me.

You still can't use it with less, but at least it allows you to mark "segments" of the log without switching to that window and mashing enter.

https://gitlab.com/kevincox/watchlog

tailf is also a good replacement for tail --follow=name, it uses inotify internally to only look at the file when there is a change.

And you can also use the & command to filter (out) some patterns, while you're tailing.

Or hit v after Ctrl-C to open it in a full editor (vi or whatever editor you have as $VISUAL). Quitting the editor resumes tailing.

Okay i wasn't interested in the `less` usage in the GP comment because i use tmux to stop and view logs as they're passing by. However automatically opening in my CLI editor? That sounds sexy as hell!

Equivalently, one can run `less +F <FILE>`

>couldn’t you pause/unpause the stdout with ctrl-s/ctrl-q instead?

edit: child comment is a better solution.

couldn’t you pause/unpause the stdout with ctrl-s/ctrl-q instead?

uh... doesn't watch just run the command over and over? I would have thought this would just print the entire file every n seconds. Now using tail with that instead of cat....

This reads the entire file, so for large files you'll likely want something like `less -n +F` if you want to use `less`.

Another tip for `less`.

bind next-file and prev-file to N and P instead of the default :n and :p.

Makes it much more convenient to flip through files.

[for some reason macos doesn't have the lesskey command even though it has the manual for it. shrug]

How many times have you seen the last line of a file you're tailing be something like "00:00 Logfile turned over." Both less and tail stop doing their business when a log is rotated and the inode changes. Stop that by adding or extending an alias:

    alias less='less --follow-name' # still requires <shift>-f to follow
    alias tail='tail -F'

I've been using `tail -F` for decades but I forgot why!

reopens files replaced with a new inode

I use tail, head, tac, grep, watch etc all the time to pick out stuff I'm looking for, sometimes I use tail -f, but it's certainly a minority of my tail invocations.

You can stop output on a terminal (emulator) by exercising flow control. If you get overwhelmed by `tail` (or a similar tool's) output, hit ^S. If you are ready to resume, strike ^Q.

My less(1) tips are:

Use "-S" to toggle the "--chop-long-lines" option at invocation or in interactive mode. This mode shows only one line on the screen for each line in a file and is helpful when viewing a log with entries much longer than the screen width. When lines are chopped less(1) lets you scroll horizontally with <LEFTARROW>/<RIGHTARROW>. Chopped mode also works in tail mode ("F" command or "+F" option).

As others have mentioned the search "/pattern" and filter "&pattern" commands are awesome and they also work in tail mode.

The -J option toggles the status column which is helpful when using search patterns or paging around a file.

Since less(1) has lots of features and navigation methods it’s good to remember the "h" command to show the help page.

One of my recently learned tail tricks (which may be very obvious to most of you already but it’s new to me) is to tail -f /proc/<pid>/fd/1 (or 2 if you want to see stderr)

I have some scripts that, for example call zcat file.tar.gz | dd of=/dev/mmcblk1 bs=1M status= progress (that last bit it important if your version of dd supports it)

This way I can watch the output of dd in another console if I wanted. I keep finding other uses for it now.

Sorry for the (perhaps) simple question, but why pipe zcat to dd?

Doesn't this do the same thing as regular gunzip? Or do you prefer the zcat | dd so you can view progress?

Well I needed to pipe the output to dd either way. You’re correct gunzip piped to dd works just as well. I’m a reason useless cat is a thing. =)

People often say "you can refactor that line and save a useless invocation of cat"

  cat /var/log/file.log

Output too long, so

Press Up, |grep -e useful -e lines

  cat /var/log/file.log|grep -e useful -e lines

OK, that looks fine, but I want to tail it

Press Up, Home, deldeldel, tail -f (other terminal modes are available)

  tail -f /var/log/file.log|grep -e useful -e lines

Exact same command no matter what grep -e you have. You separate the filtering logic from the accessing logic. You can work on one side without worrying about the other. Want to change from cat to bzcat, or tail, or tac, or nc? No problem. Want to add a pre-filter (say strip snmpd errors before you add -n to number your matches in grep), it all stays the same.

Now imagine this without "useless" cat

  less /var/log/file.log

Press Up, Home, deldeldeldelgrep -e useful -e lines

  grep -e useful -e lines /var/log/file.log

Then either

Press up, home, del(n times), tail, end, | grep -e useful -e lines

Press up, up, |grep -e useful -e lines, home, deldeldel, tail -f

Cognative break, extra typing, potentially having to look at what you're typing too rather than muscle memory, all because someone says "cat is useless". My computer isn't from 1965, I don't need to justify every process I run.

I like the cut of your jib. I feel the same way, hence the zcat. =)

With zsh:

  </var/log/file.log
  </var/log/file.log grep -e useful -e lines
  ...

That is neat

'tail' is so neat and versatile.

I used it as a solution to allow me to trigger backups of minecraft worlds from within the game server my son and I share. Anything you put to the 'say' command is written to the game's log. I use

tail -F -n 0 ./logs/latest.log | grep --line-buffer <command-pattern> | sudo xargs -L1 ./some_script.bsh

to detect commands in the log and shuttle them to a custom script. In my case it zips a copy of the world and pushes it to blob storage.

I'm not a unix guy so figuring this all out was quite gratifying.

I never type the letter n as in "-n 5" when using tail or head. I just replace n with the number, e.g., "-5". Its a habit I never broke because I still haven't encountered a situation where it does not work.

Cool never knew this - this will make me about 20% more productive at tailing!

Had an industrial plant that used these very dated VBScript based controllers. There was a SQL Server for numeric logs, but nothing for textual logs and therefore no transparency into the system. The textual logs themselves were stored on the devices in a statically sized array, so if the machine rebooted or a lot of logs happened at once (say at interesting times like startup or during errors), it was common to lose any interesting logs.

Spent a while trying to come up with a zero-budget solution for this and during a literal shower thought "Why don't I just tail a log file?" I managed to alter the controllers code to transmit recently written log messages to a minimal server which would just prepend timestamp and origin to the message and append the line to a file. Then each control console would have a Windows-built tail binary that read the file over Samba, wrap it in a batch file that would customize the colors and text a bit, and Plant Tail was born.

The mostly computer-illiterate operators quickly fell in love with it as it provided an amazing degree of transparency to what they were used to and demanded an equivalent utility for all future deployments.

TLDR: Don't rule out simple solutions if you can leverage already existing infrastructure.

Not sure how standard it is, but at least on linux you can do "head -3" as a short-cut for "head -n 3". Same with tail.

I habitually type

     (head|tail) -33

to quickly fill the terminal window with text.

Edit: that | is an OR not a pipe.

Yes I do the same because in fact on Solaris 10/11, only -X works. There is no -n switch. So I believe -X is more reliable across unixes.

That's the old school unix way. "-n" was introduced with POSIX iinvmm

On tail don't forget to use -F to tail a fail that gets rotated.

Wow, I've been using -f and getting (extremely rarely) frustrated if the file rotates. I need to read more man pages, thank you!

I wish tail had an -F -like option, to do globbing itself. When I want to tail any file following pattern mylog.*, including file created /after/ the tail command was started...

https://unix.stackexchange.com/questions/39729/monitor-files...

I... Thanks! I wish I could upvote this 10 million times.

The Logfile Navigator (https://lnav.org) supports glob patterns and does what you want by default:

https://docs.lnav.org/en/latest/usage.html#viewing-files

So, you would just run it like so:

  $ lnav 'mylog.*'

Does anyone know an easy way to tail follow two logs at once on a pair of load balancing servers?

  mkdir foo bar
  sshfs foo.server:foo.dir/ foo/
  sshfs bar.server:bar.dir/ bar/
  tail -F foo/foo.log bar/bar.log

A method that doesn't involve sshfs:

  servers=(foo bar)
  pipes=("${servers[@]/%/.pipe}")
  mkfifo "${pipes[@]}"
  tail -F "${pipes[@]}" &
  for p in "${pipes[@]}"; do
    true > "$p"
  done
  for s in "${servers[@]}"; do
    ssh "$s" tail -F file.log > "${s/%/.pipe}" &
  done
  %tail

There's timing issues on this alternative. I just ran each statement separately in a prompt, and luckily it worked. The first `true` statement needs to run after `tail` has started trying to read the first argument, and once that dies, the second run of `true` needs to run after `tail` starts reading the second argument. Last time I did something like this, I just typed the statements without the loops and ran by hand.

Wow, I never used sshfs but this seems like a fantastic convenience.

You can ssh host1 tail -F log | sed -e s/^/host1:/

But you might eventually want to log everything to a third host. Use syslog to send logs across the network and configure your local software to log to syslog.

When you have 10k hosts then sampling 1 in N log lines is still useful as a canary, while avoiding being a firehouse for the central log sampler. If something heinous occurs you can get the unsampled log from the original machine, if you need it.

Much more sophisticated techniques abound, and others will comment I’m sure, but it’s also nice to have the syslog trail as a fallback. You never know in advance what’s going to fail and syslog will show you everything from your Apache logs to kernel bugs and faulty NICs.

> You can ssh host1 tail -F log | sed -e s/^/host1:/

That's only for one host. If you mean something akin to:

  tail -F <(ssh ...) <(ssh ...)

that wouldn't work, because `tail` would wait for the first `ssh/sed` to close their output (to reach the end of the first pipe) before starting to read the second `ssh/sed`'s output.

You can do:

  $ ssh ... >a.log &
  $ ssh ... >b.log &
  $ tail -F a.log b.log

You're right. That's pretty practical. I guess I try too hard to avoid writing to disk when I can.

Honestly Loki and Promtail are so easy to setup, and jump into not just tailing and searching but incorporating logs into Grafana dashboards (you are using Grafana already to monitor host and service metrics, right?), that they make sense to run for everything but the smallest scale deployment (that is, a single service on a single node).

Sorry to plug lnav again, but if the data volume is not "too much"[1], the latest version of lnav has support for tailing files on remote hosts that are accessible via ssh:

https://lnav.org/2021/05/03/tailing-remote-files.html

So, to tail "/path/to/log" on host1 and host2, you'd start lnav with:

  $ lnav host1:/path/to/log host2:/path/to/log

[1] - This is the initial release of this feature, so it probably has some performance issues. I'd be interested to know if it worked or not.

I tried but my timestamp is of the form %Y%m%d.%H%M%S: Do you have an option to just specify the timestamp format? If not, can you give an example of defining own format? Many thanks

I'll add this timestamp to the set of defaults for the final v0.10.0 version later today.

The documentation for the format files is here:

https://docs.lnav.org/en/latest/formats.html

The "timestamp-format" property in the format definition is an array of the formats to be tried beyond the builtin set.

Here are some example formats written by a user:

https://github.com/aspiers/lnav-formats

I think some of them define their own timestamp format.

Use syslog to send all the logs to one place. Either a third server or just have one send to the other if you can handle the load.

You often want to have some indicator that you are tailing the log rather than simply having the log been output to standard output directly. A colleague of mine once Ctrl-C'd a trading server which he thought was the tail of the servers log. This happened at an extremely embarrassing moment, and after he had spent hours rebuilding all the trades in the databases, and being constantly shouted at by irate traders, he had to take a couple of days off to recover.

So, my advice is always make direct server output stand out in some way (colour, whatever). And, of course think about trapping signals, though this is often not as easy as it might seem.

In the traditional unix/posix world of cli applications interactivity is a bit "frowned upon". I really wouldn't mind if an application of this caliber would trap ctrl+c and display an "ARE YOU SURE?" message.

Same goes for coreutils. I wouldn't mind an "OK" after a cp or mv so I don't have to type echo $? to confirm it didn't crash or something

I think Using Pipes in command lines is one of the best CS Invention. I feel that drawing parallels with Unix Philosophies, design and architecture can give many good ideas for non unix architectures as well.

I’m again reminded of “Flow-Based Programming”:

http://www.jpaulmorrison.com/fbp/fbp2.htm

I used to view all of my server request logs in one stream with tail, but I didn't like tail's output because it broke it into many lines with "header lines" in between, so I wrote a small PHP script[0] to tail every file separately and output each line with a (shortened) prefix of the filename. Maybe it'll be useful to someone (who has PHP on their system).

[0] https://gitlab.com/Nicd/php-multitail

https://www.tecmint.com/view-multiple-files-in-linux/

`watch` is better in some cases.

watch is so neat to look up /proc/net or ethtool and check for changes. I'm more of a while true ; do read -t 1 ; date +%s.%N ; do_stuff...

I use both! `tail` in one window for the log file from whatever program I am running. And `watch -n 0.1 ls -lh` in another to keep an eye on the file I am writing to. Sometimes I'll put a watch on the harddrive just to make sure I am not filling it gigabytes of crap.

Now if someone could tell me how to easily live tail a CloudWatch log stream... (without gaps, errors, delays, etc).

Step 1)

Don't use CloudWatch? :)

If that world exists, we don't live in it. I use cloudwatch.

I've been using https://github.com/lucagrulla/cw for that.

Seems to work ...reasonably enough.

I would love to see a FUSE filesystem for Kubernetes, CloudWatch, Data Dog, etc. Maybe something that lets you set up the folders/files based on your preferred facets, then read your logs with tail/less/grep as you like. It would be so much faster than clicking & scrolling in their web UIs!

I'm quite fond of multitail.

https://manpages.debian.org/experimental/multitail/multitail...

for a nice GUI log file viewer, I really like glogg ( https://glogg.bonnefon.org/ ) which is avaialble windows/mac/linux

I wish I could set a tail -F to watch a log file that hasn't been generated yet. It seems to complain about the missing file even after its appeared in the working directory.

Wait, i thought the flag “-F” does not error when the file does not exist and will start tailing it when it does.

ooo maybe I have been using -f instead all along

I just want to come back and say when I tested this with tail -F just now, it did NOT start actually tailing when the log file was generated. tail (GNU coreutils) 8.22

I'm doing indie game dev (Unity + Windows), and a log file comes in extremely handy. I personally use BareTail to track mine.

By default I would recommend tail -F with captilized F. This will reopen files after a log rotation automatically.

root-tail made me a Linux user. E DR11 helped, but it was the ease and expectation that I was supposed to be able to monitor the various system logs that convinced me.

Hmmm, my pihole is blocking me from viewing the site.

I’d check the logs but my cat is on my keyboard.

I do have cloudfront blocked which breaks many things, not sure yet if it’s related.

Your cat is a very eloquent typist

I prefer ‘less +F’

Reply on news.ycombinator.com