Hacker News Reader

Comment by combiBean | Hacker News Reader

Was there not a third option: Calling the journalctl CLI as a child process and consume the parsed logs from the standard output? This might have avoided both the requirement to use CGO and also to write a custom parser. But I guess I am missing something.

View on HN

Ironically this is EXACTLY what the journald receiver for OpenTelemetry does, which, as they noted, is written in go.

Specifically because you're only supposed to use that OR the c bindings by design because they want the ability to change in the internal format when it's necessary.

Yeah looks like they missed the forest for the trees.

I see this kind of thing in our industry quite often; some Rube Goldberg machine being invented and kept on life support for years because of some reason like this, where someone clearly didn’t do the obvious thing and everyone now just assumes it’s the only solution and they’re married to it.

But I’m too grumpy, work me is leaking into weekend me. I had debates around crap like this all week and I now see it everywhere.

Also there is a --json (or -o json) flag for journalctl which will output line based json log entries. And it can simply be called with a Command as you pointed out.

This was the first thought that occurred to me too when I saw this post.

It's generally less robust to run CLI tools and scrape the output. Usually it isn't intended to be machine readable, and you have to handle extra failure modes, like incompatible tool versions, missing tools, incorrect parsers, etc.

It's the lazy-but-bad solution.

journalctl with -o export produces a binary interchange format. Would you rather have bugs or API rot from that, or in an internal tool?

journalctl is designed for these use cases and has options to solve those issues. The lazy part here is you not doing any research about this tool before dismissing it as "not best practice", which is exactly what the fuckups who wrote this article did.

We dismissed using journalctl at the very start. We’ve had similar experiences with other CLI tools: the moment you start embedding them inside a program, you introduce a whole new class of problems. What if journalctl exits? What if it outputs an error? What if it hangs? On top of that, you have to manage the subprocess lifecycle yourself. It’s not as easy as it may seem.

You can also argue that sd_journal (the C API) exists for this exact reason, rather than shelling out to journalctl. These are technical trade-offs, doesn't mean we're fuckups

Does Go really not have any libraries capable of supervising an external program? If you'd considered journalctl, why didn't you mention it in the article? As many have pointed out here, it is the obvious and intended way to do this, and the path you chose was harder for reasons that seemed to surprise you but were entirely foreseeable.

JFTR, of course it has a library for it https://pkg.go.dev/os/exec

> You can also argue that sd_journal (the C API) exists for this exact reason, rather than shelling out to journalctl.

Quoting from https://systemd.io/JOURNAL_FILE_FORMAT/

> If you need access to the raw journal data in serialized stream form without C API our recommendation is to make use of the Journal Export Format, which you can get via journalctl -o export or via systemd-journal-gatewayd.

Certainly sounds like running journalctl, or using the gateway, is a supported option.

I think a lot is riding on that “generally”. You’re right that the default approach/majority of cases should avoid shelling out wherever possible, but there are a large minority of situations where doing that does make sense, including:

Calling a CLI tool which will be present everywhere your program might reasonably be installed (e.g. if your program is a MySQL extension, it can probably safely assume the existence of mysqld).

The CLI tool you want to call is vendored into or downloaded by your wrapper program, reducing installation requirements overhead (this is not always a good idea for other reasons, but it does address a frequently cited reason not to shell out).

The CLI tool’s functionality is both disjoint with the rest of your program and something that you have a frequent need to hard-kill. (Forking is much more error prone than running a discrete subprocess; you can run your own program as a subprocess too, but in that case the functionality is probably not disjointed).

Talking to POSIX CLI tools in a POSIX compatible way (granted most things those tools do are easier/faster in a language’s stdlib).

I recently wrote some Go code for running containers and chose to use the docker CLI instead of an API client. The CLI is more well known and better documented, and this is what replacements like Podman support. When there’s a problem, it’s easier to reproduce it by running the same CLI command. It also meant I wouldn’t need a whole lot of dependencies, and we needed the docker CLI anyway.

Obviously you shouldn’t try to parse human-readable output.

> Usually it isn't intended to be machine readable

It usually is, because that is the UNIX philosophy and programs that intermingle output with layout often stop doing that, when they don't write to a terminal.

Reply on news.ycombinator.com