Hacker News Reader

Comment by junon | Hacker News Reader

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware.

Nevertheless, io_uring is certainly the better design and I'm happy to see progress still being made.

View on HN

Not disagreeing with you, but even if benchmarks are unfair, these reports are still a good illustration of progress. Unless there are some drawbacks to these optimizations, there should be no negative side-effects.

AFAIK as I know, these changes shouldn't negatively impact tasks that are not making any use of these features. So, even if improvements are not directly proportional to what is being reported, they're still real.

> Unless there are some drawbacks to these optimizations, there should be no negative side-effects.

Historically the benchmarks have been crafted in a way to make them seem faster than their real-world and well-formed opponents (e.g. the io_uring vs epoll benchmark). One issue that was pointed out was that the io_uring benchmark eschewed proper error handling in order to avoid some branches, whereas the epoll benchmark properly error checked. This reduced the benchmark times considerably, though no correction was ever published after the fact.

This is why in the Github issue I mentioned in another comment there are some people understandably a bit annoyed. I don't choose to jump to conclusions about Axboe's intent, and my original comment wasn't meant to do so.

Why wouldn't you report more rigorous benchmarks for typical use cases though? Why can't others reproduce it the same?

Nitpicking aside he has twenty years of Linux kernel development experience and does great things for everyones benefit. Congratulations Jens Axboe!

This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?

What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.)

The person you’re going after here probably felt compelled to counter the needless personal nature of your remarks. It’s difficult to experimentally verify relativity but we don’t criticize Einstein as a result.

Comparing very verifiable, applied science to very theoretical science is a strawman. Science doesn't care about feelings or tenure, so while his experience is relevant, it does not excuse the provably inflated performance figures that have been boasted historically.

> There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories.

Could you provide any links to these discussions?

Sure. https://github.com/axboe/liburing/issues/189

Original claims were in the 90% and above performance increase over epoll. Then issues were found, and the figure was adjusted to 60% over epoll. Then more issues were found, and now real-world performance tests are showing minimal speedups if any.

Unfortunately the sibling commentors don't see "computer science" as a science but instead as a "feel good hobby", it seems. My point wasn't to hurt feelings, it was to provide a word of caution with these sorts of groundbreaking claims with respect specifically to the io_uring efforts, as they have been disingenuous quite a few times historically.

I don't doubt Jens does fantastic work. I don't doubt that he's seen these speedups in very specific cases. But people are celebrating this as a win where they'd be skeptical of e.g. "breakthrough" treatments of cancer (footnote: in mice). It's the same thing.

But if you read the thread, you realize he is testing on a 2011 laptop, and the older architecture may be causing the issues. He reported a 3x speedup running the benchmark on a Raspberry Pi 4 in the same thread[0]

Someone claimed that you don't get to see the benefit of io_uring in a hypervisor[1], but they did not provide benchmark results.

[0]: https://github.com/axboe/liburing/issues/189#issuecomment-94...

[1]: https://github.com/axboe/liburing/issues/189#issuecomment-73...

If that's the best example you've got I'm writing your entire comment off as a steaming pile of FUD.

It's FUD to have personally run benchmarks across a number of platforms and machines and have seen very trivial, if any, speedup?

The referenced benchmarks are of poor implementation quality, don't even produce consistent results, and the specific linked issue is rife with discussion better suited to twitter than anything scientific or professional.

Some of the referenced benchmarks are by Jens himself. That's the problem. He has promoted tweets and press releases touting io_uring as being insanely fast when the benchmarks used to come to those conclusions were flawed to begin with.

That's literally the entire point I'm making.

Why did you omit the response you received from a netty developer that showed a large performance improvement[1]? Focusing entirely on the few cases where people report regressions and ignoring all of the reports of improvements isn't a good way to make an objective assessment.

[1] https://github.com/netty/netty/issues/10622#issuecomment-701...

I didn't omit it, and the Java example isn't the benchmark that was used to derive the initial >90% speedup that was tweeted at Jens originally.

Reply on news.ycombinator.com