The limits of open source with Illumos and OmniOS | Hacker News Reader

The limits of open source with Illumos and OmniOS

utcc.utoronto.ca

36 pointsjsnell25 comments

As has been pointed out on Twitter,

> this is funny because his main beef is actually due to systemic problems inside Intel, not Illumos or FOSS

https://twitter.com/kevinbowling1/status/714352429333086208

Since I can elaborate more on here vs twitter, I have a senior engineer spending 100% of his time fixing Intel drivers at my company for coming up on 2 years. The FreeBSD driver is somewhat related to the Linux driver but Intel has obligations to put out a copyfree implementation. I believe this was then ported to Illumos a couple years ago.

ixgbe works relatively well on Linux and FreeBSD right now, but there are still occasionally surprises and it is less efficient than other options. What's astonishing is that you can get better HW with _much_ better drivers from other companies for cheaper. The Chelsio t520-so-cr is unequivocally better than the intel x540 and costs less. Chelsio, Mellanox, SolarFlare are all good choices for Linux, FreeBSD. I think Chelsio has a Solarish driver for Illumos, not sure about the others.

Inside Intel, the Windows team, Linux team, FreeBSD team do not talk to each other. They appear to not be able to talk to the HW team either. Several large and influential companies have been trying to force Intel to clean up their FreeBSD drivers. They have taken action, and that action has been pretty disappointing. The Linux driver and commit logs are also illuminating since that would presumably have massive market share.

Intel's 40g parts have been fraught with issues at the HW level. The drivers were barely able to outperform 10g at release. This kind of slop is not normal. It should not be rewarded in the market. I had an uphill battle convincing old timers that Intel NICs went so far down hill from the good old days, but after a lot of analysis we have totally written them off for the next two years.

Is it truly hardware? I had to deal with radios for years, and the vendor-supplied driver was always poor. It came from some SOM company, who got it from the radio designer. It was always a demo driver, intended to show off the chip features but no effort put into performance or error recovery.

It seemed nobody in the 'chain of custody' of a driver had any incentive to make it work well, in a commercial setting. At best, it was consumer-quality. By that I mean, it worked until it didn't. For a radio, it meant if roaming jammed up then just take the radio dongle out and put it in again. Which in a commercial device (like a forklift touchpad) which had the radio sealed behind a panel, it was junk.

So I had to fix features, performance, bugs, timing, power management, the works. E.g. to get a radio driver fit for a WalMart distribution center forklift going 15mph, it had to roam in milliseconds and choose between 60 APs in radio range. And run for a 12 hour shift without recharging. The chipmaker driver was never, ever good enough.

Dollar for dollar, the Chelsio and Mellanox HW is better in terms of features. Better drivers and vendor support at lower purchase price make it a no brainer. I don't know about SolarFlare pricing but it is also better HW.

igb and ixgbe HW seem to be fair, but you can go look at the HW errata to judge for yourself. Intel had to recall the XL710 due to silicon issues. They also had a firmware incident that fundamentally changes the driver interface, so a particular driver will not work between different FW revs.

Chelsio told me they had Solarish drivers, I haven't ever personally tested them.

Solarflare had a 10GbE Solarish driver which they released at one point under CDDL without much engagement, and has recently been proposed for mainline inclusion with support for their 40GbE adapters as well [1].

Mellanox explicitly does not have any Illumos support for newer chips and my understanding is that they have not been receptive to any queries from the community. [2] [3]

Intel's 40g parts didn't arrive before my day job stopped involving high-speed networking every day, but their 10G stuff, as you say, worked relatively consistently across different platforms (though Illumos had some fun with the first few 10G HW revs after they started pulling common code).

I'd guess Intel's 40G parts were either a last blip or a first gasp of attempting to integrate their high-speed Ethernet bits and the "Intel Omni-Path Architecture" bits that are descended from their QLogic IB purchase.

[1] - https://github.com/gdamore/illumos-core/tree/sfxge-merge

[2] - http://lists.omniti.com/pipermail/omnios-discuss/2014-Januar...

[3] - http://omnios-discuss.omniti.narkive.com/FHhYvBui/mellanox-4...

To be honest, I think they are probably still better than Realtek or "Killer" NICs.

Yes, and the "big and gnarly" issues that he alludes to are in fact a driver issue that has been seen only by him and brought up exactly once by him on the mailing list -- and that was a year and a half ago.[1] There was lots of discussion at the time, the conclusions being that (1) he was advocating changes that were deemed unsafe and (2) that his most serious problems were seen on hardware known to be bad. The driver that he's referring to (ixgbe) is in very widespread production on illumos (albeit likely more frequently over optics than the copper that he has deployed); to the degree that there's a driver issue here at all (and that's not a foregone conclusion!), it seems likely that there is something specific to his environment that is inducing it. Certainly, with no one else seeing the issue and without better information from him (e.g., a kernel crash dump that indisputably shows an ixgbe-level issue), it's hard to see how anyone could expect any real progress to be made on this issue -- illumos or otherwise, open source or otherwise.

tl;dr: This in no way represents the "limits of open source" -- but it does highlight the limits of relying on other people to magically solve your problems for you.

[1] https://www.listbox.com/member/archive/182179/2014/10/search...

Are you sure your summary of that thread is really accurate?

To me it reads as Chris providing an exceptionally detailed bug report (including the exact code paths triggering the problem, and statistics from lockstat and dtrace on the lock in question). Nobody in the thread asks for more information (why would you want a "crash dump" for a non-crashing bug anyway?). Everyone seems to agree that the drivers are in fact taking spinlocks for long periods of time, while holding other locks. Nobody talks about "hardware known to be bad". What is talked about is how it's been too long since the drivers were last synced with upstream.

It is detailed, but in all the wrong ways: instead of describing the problem that he's seeing and offering data, he has jumped to a code path that he believes is inducing it -- without much in the way of supporting evidence. And yes, he talks about bad HW ("access to the second port on the card currently fails to acquire swfw sync"). The ensuing discussion is more of a desultory wandering than it is a deliberate investigation into his problem -- which isn't surprising, because he hasn't described a problem but merely an observed artifact in the system. (Long lock hold times can easily be misleading; when exploring latency bubbles, one needs to be very careful about tying observed behavior to the latency outliers, lest one discover problems without discovering "the" problem.)

So yes, I stand by my summary of the thread.

I'm using ixgbe (with copper) under SmartOS on three machines right now and have had zero issues.

So he's not willing to generate a kernel crashdump? Is that what you are saying? Not an attack, genuinely curious.

I'm saying that even if someone wanted to debug the problem out of the goodness of their heart, they lack the necessary data to debug it. A kernel crash dump may or may not be required; the discussion too quickly jumped to his (unverified) hypothesis as to the root of the problem to even know what data would be required.

He mentioned "several" issues. One was Intel Ethernet, but the other was a serious issue with the kernel:

on a server with 128 GB of RAM, over 70 GB of RAM was being held down by the kernel and left idle for an extended time. As you can imagine, this didn't help the ZFS ARC size, which got choked down to 20 GB or so.

That's a big issue. Is he supposed to periodically reboot his NFS servers to free up the idle RAM?

Here too we have data that borders on the anecdotal: he comes to conclusions first and then seems to look for data to backfill them. Should, as Garrett suggested, we adopt high water marks on magazines? Perhaps. Would this issue have addressed his situation? Unknown, because (once again) we have an issue that isn't being seen widely, represents suboptimal behavior (in contrast to fatal behavior), and lacks the hard data to be able to allow it to be definitively root-caused. Not a recipe for success, in any community.

I could be wrong but isn't this the normal memory management in Solaris an its derivatives? Even if it's not arc and it was used and now marked as inactive . The kernel keep its in a different "accounting bucket" , but it's still usable . What I suspect the op is ticked off with is actually arc back pressure and l1 arc eviction which can be horridly slow .

The solution is obvious: pay someone to do the work. That's how 80% of Linux kernel development is done. Just because it's free software doesn't mean that you shouldn't spend money to make it better.

Is ixgbe really that much worse than Chelsio's?

In a word: yes

Hardware as well?

Yes, but hopefully anyone doing this for real will compare whitepapers and errata for yourself, then lab test them. The T5 ASIC is in a different class than intel's parts, it can be used as several types of offload engine if that seems like a good fit for things like iSCSI, or it can be used as a very solid stateless offload NIC with tons of VFs for things like Netmap, zones/jails, or HW virtualization. Most people don't push NICs very hard, so it doesn't matter that much. If you do, you probably already have a strong opinion (i.e. glance at Netflix OpenConnect build docs).

I was planning 10G networking in my home lan, so I guess I'll eye Chelsio NIC then. Hope they're not too expensive.

Typically cheaper than Intel I bought from Netgate for my personal colo machine http://store.netgate.com/Chelsio/T520-SO-CR.aspx

Doesn't Chelsio offer copper 10G port NICs? 10G non-fiber (aka Cat.7) has been my plan.

They do, but twisted pair uses more power. You should use Twinax copper cables with the SFP NICs and switches if you don't want fiber.

Netmap support in the NIC as in FreeBSD Luigi Rizzo Netmap? What kind of support is it? Let me search....

Reply on news.ycombinator.com