Hacker News Reader

Comment by hnlmorg | Hacker News Reader

It looks like it’s ostensibly a way of being able to dynamically peak into stuff that’s being passed through a pipe.

A bit like if ‘pv’ (https://linux.die.net/man/1/pv) and a pager (eg ‘less’/‘more’/etc) had a child.

I’ve typically solved this kind of problem with ‘tee’ but this looks a much cleaner approach.

It could be a very useful tool for any Sysadmins who still find they need to occasionally SSH into a server for real time log parsing (which is a dying habit these days). Or for DevOps and Developers needing to debug.

View on HN

>(which is a dying habit these days).

Hmmm, I don't know what that says about me. Checking logs is 2nd nature to me. I guess that makes me old if the "kids these days" are doing something else, but what is the something else? Just not looking at the data and spinning wheels debugging? New tools available? I'll admit I've never looked for anything other than the location of a log of a service I'm investigating.

We use a log aggregator and watch logs from hundreds of servers in a searchable & filterable interface like Kibana. We also use Loki and Elastic extensively. If you have money to spend, I think you'd be looking at Splunk or Datadog.

As a matter of fact, we do not have SSH access to our servers. Everything happens over a web interface that has been secured.

Usually you'd aggregate those logs to a centralised location and have some kind of visibility platform to parse them like Splunk, ELK, DataDog, etc. That same pipeline would also typically share at least part of the process for streaming your logs to a tamper proof archive.

So it's not that checking logs is a dying habit, it's process of SSHing onto the server and running shell commands to query them.

Though I'm not suggesting the latter is bad practice. In some domains it's suboptimal but in other domains it is a pretty reasonable response to fault diagnosis. IT is a broad field :)

>IT is a broad field :)

Yes it is. Today's assumption seems to be that everything is being done "web scale" with hundreds of servers in multiple datacenters across the globe.

I'm not so fortunate/unfortunate (depending on one's viewpoint) of having to deal with that kind of scale. I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

I'm not going to judge how you do your job because I clearly don't know anything about the problems you're tasked to solve. However I will disagree that visibility is a web scale problem. I've still found having a centralised visibility platform useful even when managing a very small number of servers. After all, you need to set up monitoring and (presumably) alerts when something fails anyway. So it's just another service on your Nagios (for example) server.

Plus if you have any kind of auditing and compliance where you need to retain tamper proof logs for any period of time then you've already done the other half the work by streaming those logs off the servers.

Even if you're not webscale, modern tooling is just so much better for visualizing website stats. Even if you have only one server, how do you make visualizations from the log file? You don't have to use something webscale and enterprise-grade like Splunk. I do recommend finding a modern tool that you like though. Something better than `awk`-ing a log file into CSV, loading that into a spreadsheet, and then fiddling with it to get a graph.

> I have less than 10 servers I maintain so a lot of these new fancy viewers is overkill for me.

There are several layers to the complexity of "web scale orchestration" however, and you definitely can get the same tooling/features on a much smaller scale with less complexity involved.

I.e. you can rent a few baremetal servers from hetzner, give them an internal lan interface and setup k3s on that internal network. Now you can add capabilities as you want and need them: I.e.

  * deploying longhorn for a distributed storage backend with easily configured backup.

  * Certmanager for cluster wide https certificate provisioning 

  * Promstack to get healthchecks etc

  * Loki for the just now mentioned logging aggregation

Each of these capabilities are usually just one deployment command (kubetcl apply/helm install) and will enable this capability for all existing and future deployments on the same cluster.

If you have a weekend to experiment and want to explore what it actually feels to use that kind of tooling I can only encourage you to maybe try it out. The whole experience has gotten pretty nice over the years, and while it is pretty much the same you already know... Just differently, it does solve it in a very reusable way, which makes it ultimately (at least now after about 1 year of usage) much more straightforward to figure out then just a bunch of shell scripts

After I got over that it's done wrong, (as in: not what I've come to know/expect) I kinda got to like it, especially for small scale stuff (a 4 node HA cluster with 64gb ram each costs slightly below 200€ per month on hetzner, and they're using nvme storage, so really fast IO)

Not everything you can SSH into that produces logs is necessarily a "server". Small embedded systems may not have the RAM or flash space to aggregate locally, and setting them up for remote logging may be problematic or at least inconvenient for one reason or another.

> Not everything you can SSH into that produces logs is necessarily a "server".

I never suggested that was the case. I just don’t consider then in scope for this topic.

I have managed numerous such devices over my career. In fact most of my home networking gear would fall into that category of “produces logs and SSH but not a server” but none of which runs anything UNIX-like (so you couldn’t install this tool even if you wanted to). However they do support a bunch of different tooling for pushing logs and metrics over the network. Which then brings me back to my earlier point about log collection.

Also if we take your example of embedded devices with low memory and storage, then installing additional utilities like this one when you can already do the same thing (albeit less conveniently) with busybox probably isn’t a good idea. So they’d be out of scope for this thread too.

I’m not going to say that there’s no such thing as a non-server platform that produces logs that this tool would be useful for (I don’t believe in absolutes!) however it is reasonable to assume that servers and desktops, ie more classical computing devices, are a vastly larger demographic for this tool. Hence why I focused my examples around that.

As an embedded developer I have different view. Linux is everywhere. I've been working within nothing but embedded Linux constantly since 2005.

I regularly install debugging tools onto target devices (strace, tcpdump, ...) even though those don't go into a production image.

Currently, I'm using pw on an embedded system, for capturing detailed traces from an application. It's very useful because it decouples the rate of the traces from the serial port, while still letting me get the info I need out of it.

The stripped executable is tiny: 20 kilobytes. It's so small I'm stashing it in a configuration flash partition which survives re-flashes of the kernel and filesystem, so I have the utility there no matter which image I load onto the target.

The alternatives these days is examining the combined logs in an aggregator like Splunk or ELK.

> for real time log parsing

Or other things!

  tcdump -i <if> -l | pw    # -l for line buffering

Now you have an interactive network monitor that, in some ways, has better ergonomics than Wireshark. Want IPV6 only?

  :g IP6

done. Okay, only packets from bob to alice:

  :g bob.*>.*alice

Only UDP:

  :g UDP

Forget UDP, how about http:

  :r
  :g http

Kind of thing.

Ah, thank you. My main question now is whether it decides what to show based on what it's already showing (ie deduplicates somehow) or is just straight-up sampling.

I wasn’t entirely sure of that myself but it does look as if you can filter results manually

Wanted to add websocketd [1] to your list. It's an amazing tool to stream debugging logs to another system where you can build your webapps that accumulate alerts.

Use it only for debugging builds and not for production builds (obviously).

[1] https://github.com/joewalnes/websocketd

Another way to simulate the behavior of this tool, if I'm understanding you correctly, is with software flow control, usually mapped as ctrl-s for XOFF and ctrl-q for XON. That lets you pause output while you look at it, and then resume once you're done.

You cannot add filters et al with them though. I guess this is comparable to the scroll back history on a terminal multiplexer

XOFF doesn't discard input; if you pause for too long, things will backlog, which could be a problem if we are reading directly from some program (rather than, say, reading the output of something disposable and disconnected from the source, like tail -f).

When pw isn't updating the display, it's still streaming and discarding the data, which is a big semantic difference. It means that the source of the data will not block, for one thing. But if you need to see all the data or search through it, then pw is the wrong tool, at least by itself. You need to make other arrangements to capture the complete data.

PSA: don't use linux.die.net. It's horribly outdated.

This particular man page is from 2012 or earlier.

That doesn't really matter in this instance, I just wanted some example of what pv was as it's a lesser known utility and that page happened to be the top non-spammy link in DDG.

Reply on news.ycombinator.com