Comment by bandushrew | Hacker News Reader

Comment by bandushrew | Hacker News Reader

Parent Full thread

interesting.

The discovery of the edited source is a nice, solid addition to the discussion.

Ultimately though, AFAICT you and Zed are looking at - and generally agreeing with - the same facts, ie. that epoll and poll perform better depending on the situation.

If I am understanding you correctly, where you both are disagreeing is that you are determined that no webserver will actually reflect the usage pattern that would make it worthwhile for Zed to implement this 'superpoll' feature, while Zed is thinking that it will be worthwhile.

Which makes you seem like two reasonable people agreeing on the facts and both coming to a different conclusion, unfortunate but it does happen (and always fascinates me).

I wonder if part of the reason for the differing conclusions is differing expectations for Mongrel2? you seem very focused on 'expected real-life scenarios for existing websites', but I do have the feeling that Zed is anticipating/hoping/expecting that Mongrel2 will have a broader set of uses and be used in other ways in addition to the standard webserver.

I wonder if part of the reason for the differing conclusions is differing expectations for Mongrel2? you seem very focused on 'expected real-life scenarios for existing websites', but I do have the feeling that Zed is anticipating/hoping/expecting that Mongrel2 will have a broader set of uses and be used in other ways in addition to the standard webserver.

I can't speak for him of course, and I may have misunderstood some of the details of these posts[1], but it seems like jacquesm is saying that the cases in which poll beats epoll turn out to not actually be high-scale use cases in any interesting sense (and his arguments seem compelling to me). In which case it does seem safe to not bother optimizing them.

I certainly wouldn't discourage Zed from trying though. Who knows what might come of it?

[1] I write a lot of Twisted code that uses epoll - I wrote the justin.tv chat server for example, which has scaled to half a million concurrents, and I believe I get the basic concepts of epoll/poll/kqueue, but I've never actually tinkered with the details.

He says "Also, you should take under consideration the workload of your system, if you are sure that you will have a very large portion of your fds active most of the time and that your worker pool is sufficiently large compared to the number of connections that you have that there is no reason not to use poll (think media/file servers), whereas if you are fairly sure that the majority of your fds are inactive and/or that the size of your workerpool is substantially smaller than the number of fds that you have open (think web servers) that there is no reason why you shouldn't use epoll.

That's the precise reason those two calls exist in the first place, and which one is 'best' (for whatever bit of extra throughput you can squeeze out of a machine because of this optimization) depends on your traffic."

To me it sounds as if overall he is saying roughly two things: (1) "There are server conditions where epoll works better, and server conditions where poll works better,"

and (2) "I dont believe that web servers specifically under real-life conditions are likely to be improved much by poll"

Clearly he and Zed agree on (1), that being the point of the related Zed post, and (2) might well be true, but I dont think it necessarily speaks directly to the use cases that Zed is hoping will eventuate for mongrel2.

Zed used a web server to test the differences between epoll and poll, and that has clearly focused the discussion somewhat, but from what I gleaned by reading his ongoing 'The Coming Of Mongrel2' series, he has a wider range of possible uses in mind for Mongrel2, and I am wondering whether that is where his and jacques conclusions are differing.

[1] I mean, clearly they are both entirely insane in a very real sense, but I do love to pretend that those around me are rational actors and enjoy projecting rational explanations for their behavior upon them.

sorry? why am I being downvoted? surely if you think Ive missed something in the above summary it would be more constructive to post a reply?

It may have to do with calling Zed and me insane. I don't know what you base your knowledge of me on but in defence of Zed he does not at all come across as insane to me, though he definitely does seem to have a pretty rough edge.

You got it perfectly, so even if you weren't speaking for me you might as well have.

> Ultimately though, AFAICT you and Zed are looking at - and generally agreeing with - the same facts, ie. that epoll and poll perform better depending on the situation.

Correct. For some value of 'better'. Clearly that difference will be larger if you do less other stuff.

> If I am understanding you correctly, where you both are disagreeing is that you are determined that no webserver will actually reflect the usage pattern that would make it worthwhile for Zed to implement this 'superpoll' feature, while Zed is thinking that it will be worthwhile.

That has to do with the way epoll is implemented and the expense of the calls to ship fds back and forth to the kernel epoll structure.

Some research has been done on this and it shows that even improving epoll did not matter enough to care about:

http://www.linuxinsight.com/ols2004_comparing_and_evaluating...

> Which makes you seem like two reasonable people agreeing on the facts and both coming to a different conclusion, unfortunate but it does happen (and always fascinates me).

Yep, that's pretty much it. But part of the different conclusion is in the input to the pipetest program, if you feed it unrealistic data you will get unrealistic answers. There is a hint in my piece about what is happening in the more realistic segment of the problem domain (the 0..2K threads section).

> I wonder if part of the reason for the differing conclusions is differing expectations for Mongrel2? you seem very focused on 'expected real-life scenarios for existing websites', but I do have the feeling that Zed is anticipating/hoping/expecting that Mongrel2 will have a broader set of uses and be used in other ways in addition to the standard webserver.

Which is his privilege.

And as I wrote in some other thread I'm hoping that he will succeed.

Reply on news.ycombinator.com