Unmapped I/O improves userland direct disk performance by 35% ~ 135% | Hacker News Reader

Unmapped I/O improves userland direct disk performance by 35% ~ 135%

svnweb.freebsd.org

85 pointsantonios36 comments

From the title, I thought Microsoft had sponsored an improvement in generic FreeBSD.

From the revision, it seems the improvement is in a Hyper-V storage driver.

Indeed, Microsoft has done quite a bit of work towards increasing the performance of FreeBSD running on Azure / HyperV.

Yes, they're doing this with Linux too. I recall reading an announcement about 20kLoC worth of their patches to the Linux mainline kernel, in order to improve performance on Hyper-V.

I mean, this only makes sense -- OSes are no longer a core product in the server world; they're just a part of the application stack. The hypervisor / orchestrator is the critical part, and that's Hyper-V in the Microsoft world.

Also, this patch doesn't strike me as monumental. Really, it strikes me as one of the first things you would do when configuring a disk driver for a VM (mapped I/O is unnecessary overhead for most VM configurations as the hypervisor and/or SAN remaps it anyway). Has Microsoft not supported FreeBSD well under HyperV in the past?

Hyper-v kinda sucked with FreeBSD, mostly just overall poor performance. If you needed FreeBSD that would dictate what you ran it on, so hyper-v would be out. If you had to run hyper-v, FreeBSD would be out. Many enterprises find Windows to be mandatory, even if it's just running virtual machines, even if a better solution is out there. UNIX is often replaceable because "they're all the same."

Yeah, the cloud is finally catching up with the mainframes.

They'll have caught up enough when they have I/O coprocessors as general as Channel I/O. Are there any server standards close to that? And CPU's with the needed I/O accelerators?

Why would we want specialized processors when a modern CPU core is filled with dozens of CPU cores? And no, I'm not trying to be snarky, but Channel IO sounds like something from the days when CPU was rare and expensive but doesn't make sense in an architecture where lots of cores exist and IOPS and network are generally a bigger issue. What am I missing?

We want them very badly, because they have a wider bandwidth at the cost of slower clock.

What I don't get is why the GP is stating that servers need to get them, when every PC comes with specialized IO processors for decades already.

"when every PC comes with specialized IO processors for decades already."

No they do not. My main CPU (s) gets interrupted by my drivers nonstop. As networking or disk goes up, my apps slow down even if they don't depend on them since the CPU is absorbing the hit of many of those operations. That's totally different than a Channel I/O-like model where a dedicate CPU manages all that with apps just handing it off asynchronously in a high-level way, I/O processor doing low-level stuff, & scheduler making sure nothing slows down due to waiting on it. That's why my CPU is sometimes under 10% utilization when it could be working big time but mainframes are more like 90-97%.

Or maybe you meant they have specialized hardware... dedicated ASIC blocks... to accelerate TCP/IP processing, compression, filesystems, database operations, and so on? I didn't see those for decades either outside custom cards or chips on the motherboard. Definitely not in SOC's designed for them with smooth CPU integration. That's come online recently under semi-custom banner at AMD/Intel & all these Cavium-style processors that merge CPU's + HW accelerators.

So, there's a difference between what highly-efficient, I/O model some products have right now and what my desktop computer has. The servers are catching up with all kinds of fast buses and add-on cards. Need changes at CPU and OS levels, though, for max effect.

> Channel I/O-like model where a dedicate CPU manages all that with apps just handing it off asynchronously in a high-level way

Wheels being reinvented for some time now in high-load world. Polling drivers instead of IRQ-on-every-packet-arrived, smart NICs, CPU-pinned queues and processes. Looking at e.g. DPDK architecture, it's almost mainframe type of Channel IO actually.

They're getting really close. I'd say they basically have the benefits on the networking side given almost all the work went there. Polling drivers was a significant step, too.

The future looks more and more like Channel IO http://www.theregister.co.uk/2016/07/22/mellanox_punting_mul...

Didn't see that one so thanks for the link. Ok, now that is definitely Channel IO in new form. It has a bunch of high-speed hardware, a bus for it all to communicate, and [critically] efficient SOC to absorb all the work such hardware brings. Some of the software on their list are used in clusters and such that were considered mainframe alternatives for batch or performance-oriented workloads. So, yeah, generalizing that architecture to various types of hardware would constitute a Channel IO replacement.

I think one commenter thought "Can't you do the same with multi-core CPU's?" One advantage of Channel I/O was that they could use weaker CPU's since it didn't take as much work. Cost or efficiency advantages with the heterogeneous architectures. We see that in this one where they use lightweight, ARM multi-core instead of heavyweight x86. Leaves room for accelerators in SOC or just watts for other stuff in system. So, going in right direction again.

They're already used in cloud market for niche stuff like network processing. Something about each major task like networking, compression, or encryption being accelerated to Gbps leaving all those cores mostly idle for app logic us said to benefit throughout & latency. ;)

Last time I checked, a z-series IO processor was a normal processor that just want licenced to run application code. Same as how their COBOL processors are the same as their Java processors, despite being more expensive.

AFAIK microcode is the big part of zCPU, that allows reconfigure the way how zCPU works internally, despite the fact on precise hardware level it's still the same zCPU.

CPU bound workloads are relatively rare. Memory and IO are the bottlenecks that most people deal with today.

We want specialized processors and we have them and going to have them even more, in many flavours. GPU is there, FPGA is on the way ([1],[2]), network IO is half-there (look at Solarflare NICs).

[1] http://www-304.ibm.com/support/customercare/sas/f/capi/home....

[2] http://www.theregister.co.uk/2016/03/14/intel_xeon_fpga/

No, thank goodness, but not for lack of trying[1]! The I2O group seemed to favour channel I/O on the grounds that it was an excellent way to sell more proprietary hardware and restricted access to specifications. Fortunately they were not successful in simultaneously making the x86 world more proprietary and more expensive all at once.

[1]: https://en.m.wikipedia.org/wiki/I2O

I didn't know about that one. Sounds like a failed implementation of the concept. A better one that someone told me about recently was some embedded products combining a good CPU with a weaker one (eg Cortex M0) that absorbed & queued interrupts to preserve real-time properties. Sandia did something similar in SSP processor. lowRISC is experimenting with their minion cores.

So, I think there's still potential. Especially given the formal verification and security strategies for I/O programs were often different in academic literature than algorithmic ones. Specialized hardware might facilitate improving their security or availability in provable way.

> They'll have caught up enough when they have I/O coprocessors as general as Channel I/O.

Modern Zs have largely removed I/O copros - none of the Z9 through Z12 have dedicated offload for DASD.

Appreciate the tip. In that case, the servers could get ahead of them especially if they use simplified cores for interrupt-heavy processing and HW acceleration for algorithm/protocol state machines that rarely change.

I don't understand mapped vs unmapped io in terms of Hyper-V. Does that mean that io from a user process goes directly to a paravirtualized io device and bypasses the freebsd kernel?

I had the same question, or maybe even a more basic one: what do they mean by "unmapped I/O", and is there a different term for this concept in Linux? Does it perhaps relate to O_DIRECT? To sendfile() and splice()? This paper is the closest to an answer that I've found: http://www.cs.technion.ac.il/~dan/papers/cim-asplos-2016.pdf

I think the point is that, currently, BSD (at least with Hyper-V) defaults to a "mapped" zero-copy approach: when a read is requested, the user space buffer is mapped to a DMA accessible address, the device DMA's the data directly into the buffer, and then the DMA address is unmapped.

Unfortunately, the IOMMU operations to make this happen are not free, and have side-effects like a TLB flush. For small transfers, it's cheaper to let the DMA write to a persistent kernel buffer, and then have the kernel copy it to user-space. I think this traditional non-zero-copy approach is what they are referring to as "unmapped".

But I'm guessing. It would be kind if someone fluent in BSD could explain it correctly.

Can anyone eli5?

FreeBSD running on Hyper-V (Microsoft's virtualization layer) was slow; now it's faster thanks to code Microsoft contributed (EDIT: or simply sponsored, it's not clear).

I see now that the patch was sponsored by Microsoft, but for a moment I thought that this was a new kind of HN text ad.

We updated the title from ”Sponsored by Microsoft – Improves userland disk I/O performance by 35% – 135%” which was editorialized.

For those wondering what is going on, the original submission title started with "sponsored by Microsoft". It has since been edited.

Thank you! I got here after the title change and was terribly confused.

The style of this diff is bewildering. What's with the gratuitous reformatting of the whitespace? Makes it difficult for the reader to pick out the meaningful difference.

I'm looking on a phone, so maybe I'm missing it, but I don't see much gratuitous whitespace change. The only place I see that is in the declaration of hv_storvsc_request.

Diff tools that can ignore white space changes make code reviews a lot easier.

Or one can separate white space diffs from functional diffs and reduce the dependency on particular tooling choices. This often works a lot better because style diffs often reflow lines that -w doesn't handle.

You can when you don't work on a team with feature branches, but everybody who failed to understand CI seems to think its a wonderful way to work. And it turns out there are a lot of them.

Reply on news.ycombinator.com