Hacker News Reader

Comment by conradev | Hacker News Reader

The author does make a valid point in that logs are harder to collect the more processes you add.

For example, I run gunicorn inside of a container with worker processes, and I had to do some trickery (writing to /proc/1/fd/1) to redirect the logs of the worker processes to the stdout of the master process so that they would show up in `docker logs`.

View on HN

It's almost like the idea of relying on stdout from processes in a container to collect logs is fucking stupid.

With all the hoops people jump through so they can say "we use Docker, we're cool" I think I shall refer to Docker fan-boys as Olympic Gymnasts from now on.

Why is it stupid ? Is the same approach that heroku advocates.

I have loads of containers logging to stdout and another container which collects all stdouts and send to a logging-as-a-service. Why is it a bad idea ?

> Why is it stupid

Let's compare logging via stdout, to say syslog, with a simple test.

How do you identify the difference between a warning and an error or an emergency, in a string sent to stdout? What happens if the message contains a newline. How do you know if what follows is a continuation of the log entry, or is a new log entry?

If your application dares to have two processes in the same container (i know, crazy concept) how do you separate them.

The "literally one process per container" idea is what both causes the issue, and causes the obvious solution to the issue to be even harder to solve for the cases where software doesn't natively support the syslog protocol.

Were you the one who downvotted me because you disagree? o.O

Anyways, this is dealt with in the logging-as-a-service (LAAS) that receives the logs... say you are using the python logger, you have different types of logging (DEBUG, INFO, WARN, ERR). If you just start the line with one of those statuses, your LAAS will know it should email you or something like that.

The print, or file, or whatever, it's just how your data is getting somewhere for the proper analysis.

You know if it's a new line or not because every log message starts with the same format, something like {APP, TIME, SEVERITY} and then the message...

So instead of having actually structured information in defined fields, you're relying on a specific service's format. So presumably you then have to try to get all the services you use (e.g. Apache / NGinx / HAproxy / Varnish / (my|postgre)sql / redis / memcache etc) to format logs in the same way, and do it to stdout...

> Were you the one who downvotted me because you disagree? o.O

Down voting on HN is broken[1], so I don't use it, ever[2].

1. In the sense of, it does the wrong thing, not in the sense of it doesn't do anything.

2. Except when i hit the wrong thing because of the complete lack of usability on a mobile device.

yup, I will tell you tho, it isn't really hard once you have a format... it's pretty easy to share the format between different services as it's just string formatting.

I currently have logs from the django app and nginx and was pretty easy to set up.

It might not be the best solution but I don't think it's stupid

If you drill through the docker source code, it becomes apparent that their are some rather serious issues to doing something other then this.

I really really want a LOG directive for docker where I can just setup some pipes which the daemon will collect and tag into it's log files (would make dealing with old Java processes a lot easier).

But actually finding somewhere to hook this logic in - huge task. A lot harder then grabbing stdout/stderr of the init process in the container.

You can use different logging drivers: https://docs.docker.com/engine/admin/logging/overview/

That does not answer the question on how to capture _different_ application logs.

Why not have the application log to a file on a data container?

Who will manage these volumes, directories and log files? Why not just use existing infrastructure?

I use containers for 10 years (they are called VServer), and I newer had a problem with standard setup: some less important logs are stored directly in container and rotated with logrotate, some more important logs (and error messages from less important logs) are forwarded to central server for analyzing, some are just disabled or discarded.

Well, now you've just created a new problem to deal with...

I'm a heavy docker (ab)user, but I haven't gone through the source code - i.e. perhaps there is some complexity I'm overlooking.

But couldn't Docker provide additional file handles for extra logging streams?

Programs which understand they live in a Docker world could hook into an special system call directly. Non-Docker-Logging-Enabled apps could have a helper program (conceptually similar to 'ip netns exec') remap STDOUT & STDERR.

The whole point of getting logs out of containers is to centralise and store them somehow.

This exact job is what Syslog (over a network) does.

Or maybe we can refrain from calling each other perjorative names? Wouldn't that be grand?

Normally, if you run Docker you won't have a single container anyway, normally you want to scale out. Hence you'd be better off using a centralized logging infrastructure like ELK or one of the many commercial solutions like Splunk etc.

True, but it's not uncommon to only route the stdout/stderr of each container to fluentd, logstash and the like such that the containers do not have to care about where the logs go (other than just writing to stdout/stderr). And now that we have logging driver I definitely would want to avoid custom in-container log routing.

Yes, it's good practice to have a separated logging container (eg logstash forwarder, filebeat etc), which accesses the shared log folder of the other container you want to actually forward the logs from. I'm recommending using file forwarder because this way you can simply categorize your logs by filename. Pushing all to stdout is a pain.

Yeah, the problem is getting them from each process in the container, then get them to the centralized logging. It can be done either way though. Especially with the syslog driver in Docker...

I don't follow. If you go with one service one machine model, not using a container is more or less the same than just scripting a repeatable install.

I'd say they are for isolation before they're for scaling out

Reply on news.ycombinator.com