Open-source drivers according to Habana

It's not just drivers. It's really about ensuring that the folks that maintain the kernel have a way to test the code they maintain. The reasons that we (the kernel maintainers) have for this requirement are varied. But, for me, it's really nice to have at least one open source implementation that can test the kernel code. Without that, the kernel code can bit rot too easily.

Even better is if an open source implementation is in the kernel tree, like in tools/testing/selftests. That makes it even less likely that the kernel code gets broken.

Disclaimer: I work on Linux at Intel, although not on drivers like this Habana one.

One of the points of having the drivers in kernel is that means they kernel can actually run on that hardware. In addition to allowing for testing as others have pointed out, it is also a way to make sure that drivers aren't used to restrict access to the hardware. It ensures the freedom of the platform.

Or they could pull an NVidia, and dedicate a whole in-house kernel team to maintaining an out-of-tree kernel module.

Thanks for this, without this I had no clue what the article was referring to.

It would be nice if there was a fairly stable kernel API (or ABI) so drivers like this didn't have to be in the kernel. Out of tree drivers are a nightmare to maintain.

I maintained some out-of-tree drivers for years at Myricom (version of myri10ge with valuable features nak'ed by netdev, MX HPC drivers). Doing this was a massive PITA. Pretty much every minor release brought with it some critical function changing the number of arguments, changing names, etc. RHEL updates were my own special version of hell, since their kernel X.Y.Z in no way resembled the upstream X.Y.Z It got so bad to support 2.6.9 through 3.x that the shim layer for the Linux driver was almost as big as the entire FreeBSD driver (where nobody cared that I implemented LRO).

There's another preceding case here, from 2020, involving Qualcomm submitting a similar driver (and to some extent Microsoft), which is quietly linked to and worth looking at for some other history: https://lwn.net/Articles/821817/

The situation there is that Habana already had their driver submitted very early I suppose, and I guess the resistance wasn't high enough at the time to keep it out. Qualcomm later came around and their own AI 100 driver was rejected, on similar grounds that would have kept Habana out, had they been applied at the time. (Airlie even called out Greg at this time.)

The later scuffle (your OP link) is because the Habana driver eventually wanted to adopt DMA-BUF and P2P-DMA support in the driver, which the original developers intended for the GPU subsystem, so they consider this over the line, because the criteria for new GPU drivers is "a testable open source userspace". So, that work was rejected, but the driver itself wasn't pulled entirely. Just that particular series of patches was not applied.

Microsoft had a weirdly similar case where they wanted a virtualization driver for Linux that would effectively pass GPU compute through to Windows hosts running under HyperV, for the purposes of running Machine Learning compute workloads -- not graphics. (The underlying Windows component to handle these tasks does use DirectX, but only the DirectCompute part of it.) But, it wasn't out of hand rejected on the same principle; it's more like a VFIO passthrough device conceptually, and didn't need to use any DRI/DRM specific subsystems to accomplish that. But the basic outline is the same where the userspace component would be closed source, so the driver is just connecting a binary blob to a binary blob. It doesn't use any deeply involved APIs, but it's also not very useful for anyone except the WSL team. It's a bit of an inbetween case where it isn't quite the same thing, but it's not not the same thing. Strange one.

As of right now, looking at upstream:

- Habana now has DMA-BUF support, as of late last year, so presumably the minimal userspace given above was "good enough" for upstream, since they can presumably at least run minimal testing on the driver paths: https://github.com/torvalds/linux/commit/a9498ee575fa116e289...

- Microsoft's DXGI/whatever-it's-called driver for compute is still not upstream, but I think they ship it with their custom-by-default WSL2 kernel (`wsl -e uname -a` gives me `5.10.16.3-microsoft-standard-WSL2` right now). It was not rejected out of hand but they also didn't seem to mind if it didn't land immediately. I have no idea what it's status is.

- Qualcomm's driver for AI 100 was completely rejected immediately and I do not know of any further attempts to upstream it.

- And there are probably even more cases of this. I believe Xilinx has a driver for their (similarly closed) compiler + runtime stack included in Vitis, and I doubt it's going upstream soon (xocl/xcmgmnt)

So the rules in general aren't particularly conclusive. But it looks like most accelerator designs will eventually fall under the rules of the graphics subsystem, if they seek to scale through P2P/DMA designs. As a result of that, a lot of people will probably get blocked, but Habana to some extent got a first-mover advantage, I think.

Arguably if people want to complain about SynapseAI Core being unsuitable for production use, to some extent, they should also share a bit of blame with the Linux developers for that, if they consider the drivers a problem. I think this isn't an unreasonable position.

But ultimately this comes down to there being two different desires among people: the kernel developers' concerns aren't that every userspace stack for every accelerator, shipped to every production user, is fully open source. That might be the concern of some people who are users of the kernel and Linux (including some kernel developers themselves), but not "them" at large. Their concern might be more accurately stated as: they have enough tooling and information to maintain their own codebase and APIs reliably, given the hardware drivers they have. These are not the same objective, and this is a good example of that.

If you want to use that, you need the closed-source driver.

The open-source driver is the minimal to make linux kernel developer happy.

…and their customers will simply not buy their product if it needs a custom kernel.

> If it were up to me, I'd have people hacking them

Cut this garbage shit out. Hack on the driver implementation instead.

> Because if I understand anything at all about open source, it's that you don't get to post closed source as if it is open source

If it was upstreamed into the linux tree, then they have open sourced, with the appropriate license. So the driver is not fully functional, but what they have included could be the building blocks for someone to add the additional functionality.

It is a shitty practice, and if you want to drive adoption, this isn't going to do it.

At least they didn't make a marketing statement that they're open source purveyors and software saviors.