Poll vs Epoll, once again.
jacquesmattheij.com
jacquesmattheij.com
The discovery of the edited source is a nice, solid addition to the discussion.
Ultimately though, AFAICT you and Zed are looking at - and generally agreeing with - the same facts, ie. that epoll and poll perform better depending on the situation.
If I am understanding you correctly, where you both are disagreeing is that you are determined that no webserver will actually reflect the usage pattern that would make it worthwhile for Zed to implement this 'superpoll' feature, while Zed is thinking that it will be worthwhile.
Which makes you seem like two reasonable people agreeing on the facts and both coming to a different conclusion, unfortunate but it does happen (and always fascinates me).
I wonder if part of the reason for the differing conclusions is differing expectations for Mongrel2? you seem very focused on 'expected real-life scenarios for existing websites', but I do have the feeling that Zed is anticipating/hoping/expecting that Mongrel2 will have a broader set of uses and be used in other ways in addition to the standard webserver.
I can't speak for him of course, and I may have misunderstood some of the details of these posts[1], but it seems like jacquesm is saying that the cases in which poll beats epoll turn out to not actually be high-scale use cases in any interesting sense (and his arguments seem compelling to me). In which case it does seem safe to not bother optimizing them.
I certainly wouldn't discourage Zed from trying though. Who knows what might come of it?
[1] I write a lot of Twisted code that uses epoll - I wrote the justin.tv chat server for example, which has scaled to half a million concurrents, and I believe I get the basic concepts of epoll/poll/kqueue, but I've never actually tinkered with the details.
That's the precise reason those two calls exist in the first place, and which one is 'best' (for whatever bit of extra throughput you can squeeze out of a machine because of this optimization) depends on your traffic."
To me it sounds as if overall he is saying roughly two things: (1) "There are server conditions where epoll works better, and server conditions where poll works better,"
and (2) "I dont believe that web servers specifically under real-life conditions are likely to be improved much by poll"
Clearly he and Zed agree on (1), that being the point of the related Zed post, and (2) might well be true, but I dont think it necessarily speaks directly to the use cases that Zed is hoping will eventuate for mongrel2.
Zed used a web server to test the differences between epoll and poll, and that has clearly focused the discussion somewhat, but from what I gleaned by reading his ongoing 'The Coming Of Mongrel2' series, he has a wider range of possible uses in mind for Mongrel2, and I am wondering whether that is where his and jacques conclusions are differing.
[1] I mean, clearly they are both entirely insane in a very real sense, but I do love to pretend that those around me are rational actors and enjoy projecting rational explanations for their behavior upon them.
Correct. For some value of 'better'. Clearly that difference will be larger if you do less other stuff.
> If I am understanding you correctly, where you both are disagreeing is that you are determined that no webserver will actually reflect the usage pattern that would make it worthwhile for Zed to implement this 'superpoll' feature, while Zed is thinking that it will be worthwhile.
That has to do with the way epoll is implemented and the expense of the calls to ship fds back and forth to the kernel epoll structure.
Some research has been done on this and it shows that even improving epoll did not matter enough to care about:
http://www.linuxinsight.com/ols2004_comparing_and_evaluating...
> Which makes you seem like two reasonable people agreeing on the facts and both coming to a different conclusion, unfortunate but it does happen (and always fascinates me).
Yep, that's pretty much it. But part of the different conclusion is in the input to the pipetest program, if you feed it unrealistic data you will get unrealistic answers. There is a hint in my piece about what is happening in the more realistic segment of the problem domain (the 0..2K threads section).
> I wonder if part of the reason for the differing conclusions is differing expectations for Mongrel2? you seem very focused on 'expected real-life scenarios for existing websites', but I do have the feeling that Zed is anticipating/hoping/expecting that Mongrel2 will have a broader set of uses and be used in other ways in addition to the standard webserver.
Which is his privilege.
And as I wrote in some other thread I'm hoping that he will succeed.
These are two good developers who are drawing on their real life experiences to develop their opinions, backed up with some simple tests performed on local machines.
While it would be pleasant if they could provide hard data for a wide range of webservers to 'prove' their opinions it is hardly a realistic option for either of them without investing a degree of time and effort that is far greater in scope than the work being discussed.
I would prefer to see less of a cry for hard evidence of real life usage patterns from either side, and a greater acceptance of the fact that this is simply an interesting discussion of technical possibilities by some good developers.
Instrumenting varnish or apache would be a lot harder to do because I don't know those code bases well enough to know for sure that I'm doing it just right so all I can rely on there is the data reported by the 'status' modules of those programs, and for now neither of those suggest otherwise.
Zed attacks that data with a bunch of handwaving but to date has yet to show a single instance of a server which actually has an active-to-total ratio that is higher than 60%. I suspect such servers do exist, media servers, file servers and other other servers with relatively few connections, a large worker pool compared to the number of active connections and lots of long files being shipped across those connections.
But your typical web server is more likely to be stuck in an endless cycle of accept-read-write-close where both the 'read' and the 'write' can block for significant amounts of time compared to the number of bits shipped, and the backend can do the same thing once more. So idle is the norm.
If so, read the comments here: http://news.ycombinator.com/item?id=1570694
an actual discussion would be an interesting thing.
sure, its easier to just downvote someone as you go past without actually engaging sufficiently to understand or discuss the issues, but ask yourself...is that really who you want to be?
"Resist complaining about being downmodded. It never does any good, and it makes boring reading."