Hacker News Reader

Comment by icelancer | Hacker News Reader

>> There are a lot of people gloating about this on Twitter and elsewhere, but to me it just seems sad; the ultimate morass of monoculture and an endgame with no competition. An ode to "the OS layer no longer matters, don't bother putting any effort into it someone else will". Bystander paradox.

>>I know Linux fans don't see it that way..

I am not a Linux fan per se (as I type this from my Win10 desktop), but I legitimately don't understand what your comment is about. I've read it a few times and I'm still not exactly sure what you are complaining about.

How is Linux a monoculture, or this the endgame... or this stagnation?

I can't figure out what you are saying, sorry.

View on HN

For big metal rigs like supercomputers, there is almost no choice but Linux. Combined with Linux' dominance of the server market in general, there's a risk emerging to get stuck in a monoculture on the OS layer.

Most companies, including very big ones with a long history of OS development are abandoning their efforts in favor of just using Linux. The perception is often that there is no innovation that can be done on the OS layer that will earn back its investment.

This leads to a lack of experimentation, tunnel vision, and in the long run stagnation.

Companies experiment on Linux everyday. Unless they want to use a different programming language, they can implement all sorts of innovation in the Linux Kernel without incurring in massive overhead costs of rolling their own.

I don't think the situation is nearly as bad as portrait here.

They can always use *BSD as a second choice too.

Additionally, where's the stagnation we are so worried about here? Any real world examples?

> Additionally, where's the stagnation we are so worried about here? Any real world examples?

BeOS? Singularity? Plan 9? Hell, just read the last 20 years of OS research and realize that large swathes of it will never get used because anything that gets implemented has to be compatible with the way the Linux kernel does things. (By way of analogy, consider the x86 architecture, also a '70s era technology, and the massive contortions Intel had to go through to maintain compatibility over the years while adding new features and how all the newer alternative architectures died in the marketplace because x86 was too entrenched.)

Fuschia and, ironically, Windows are the last hopes for advancing OS architecture and that's really only possible because the semi-monopolists that created them have enough cash and engineering talent to push them until they take off.

Sorry, maybe I wasn't clear.

We are talking about HPC computing and how Linux being the only viable choice leads to stagnation.

You are proposing those OSes advance the status quo (and I agree) but how is it that HPC computing is stagnated because Linux is the only viable option? If anything, HPC would be an awesome testbed for those experimental OSes and they still choose Linux.

Where's HPC stagnated because awesome ideas from other OSes can't be used?

Finally, are these other ideas so extremely awesome that they require a new OS from scratch (with all the uphill battle that it's to create it)? Can they be implemented on top of the Linux kernel? In which case, is Linux's success really such a bad thing?

Hey, Haiku's still around, and while we don't have time to experiment with such crazy ideas ourselves (keeping an operating system project afloat is no joke), we'll more than likely accept patches for such things ... provided they actually work and aren't just needless bloat, and especially if they fix or improve some area we've neglected :-p

I understand that there is experimentation and innovation inside and on top of the Linux kernel, but that still leads to some kind of local optimum, the best approach that fits within the design and implementation of the Linux kernel.

I'm not saying this is necessarily a problem, but it risks becoming a problem.

It's clear we're seeing reduced investment in Linux competitors in exchange for Linux flavours from large players the past decade or so.

The competition has moved inside Linux. The large companies all maintain their own patch sets. There are still ways they believe that they can make OSes better, they just don't believe they can compete with Linux in all the other ways, so they just write their own modifications for it instead.

Which is absolutely wonderful, because Linux is licensed under GPL, so the best improvements end up in the main tree and everyone benefits.

> Linux is licensed under GPL, so the best improvements end up in the main tree

Without taking sides in the debate, grsecurity/PaX patches are a counterexample.

That's a great example of https://news.ycombinator.com/item?id=15772233 (currently also on the front page), where something appears superior on the surface, but there are good reasons why it didn't "win".

"grsecurity/PaX" didn't show up in the main tree but is alive in a different branch or as patch. Various distributions and users are free to enable and use it upon demand.

It kind of prove Linux kernel is not mono-culture and natural selection and evolution are happening.

I believe the AOSP's kernel patches are also not mainlined - also proof Linux kernel are evolving very healthily.

So in essence Linux have become what UNIX was back in the day, a baseline for everyone from IBM to HP to build upon.

That's all baseless speculation and hand-wringing over straw-boogeyman.

A 9 year old can install Linux and experiment and "innovate" all she wants. Linux's open source nature and the community is what powers its ongoing evolution, and that is encoded in its essence. I fail to see how the vein of Linux innovation on big iron is any close to being spent.

A "winner" is all but guaranteed when it comes to slicing the smallest performance gains in any field. And this is a niche field of performance; as I said in another comment, Win10 is still superior to Linux in other compute-bound systems (HBM2 is incapable of being exploited to its fullest potential outside of Win10 due to AMD's poor compute/blockchain drivers for Linux, for example - if you need blazing fast memops for the price, nothing beats Vega architecture right now).

Stagnation is a good thing -- when it happens because all the hard problems in a domain has been solved.

More like all the easy optimizations have been exploited, and all that remains are the hard (and much less cost effective) ones.

The linux kernel has a certain architecture which informs the design of every OS built on top of it. There is effectively no competition of core OS architecture ideas anymore because linux won. Now, linux won because it was better, but is it really the right OS architecture for the 3d xpoint / quantum-mechanical hardware stack of the future? The monoculture makes it harder for people to experiment, and slows down discontinuous innovation in OS design.

Or, just imagine microsoft won and you substitute windows for linux in this thought exercise. Wouldn’t that be a problem for innovation?

Windows is not developed in the open. It'd be a very different situation.

From an architectural perspective, it's really *nix that won. To the degree that we worry about software monocultures in that vein, it really doesn't matter much if it's only Linux or if it's Linux + AIX + various other Unixes. That Linux is open source and allows for customization and variants makes it less of a monoculture but it still embodies a specific architectural approach. (Although GPU computation is one big change that's taken place.)

The TOP500 list does tend to be oriented to highly parallel systems because of the benchmark used. However, the different architectures like vector systems have largely gone by the wayside anyway. As CMOS scaling runs its course I expect we'll see more experimentation. Whether some of that requires a new operating system model as well remains to be seen.

>>Or, just imagine microsoft won and you substitute windows for linux in this thought exercise. Wouldn’t that be a problem for innovation?

There is no "winner" as far as I can tell in the grand scheme of things. We're talking about a set niche of computing. Win10 is still superior for a lot of GPU-clustering specific work; HBM2 memops via Vega architecture comes to mind as the Linux compute drivers are simply inadequate and the Windows tooling is significantly better.

Why isn't the promise of https://en.wikipedia.org/wiki/Singularity_%28operating_syste... being delved into? Who is pushing the microkernel ideal? Alan Kay has talked about how back in his time one had to do the full stack: hardware to kernelspace to userspace. Why is hardware being developed running Linux instead of an OS tailored to use that hardware?

Maybe it has to do with the Linux kernel being malleable enough to meet these specialized needs. My comment is a bit open ended, there's probably good reasons like the ideas I referenced had underlying issues. Microkernel security isn't really necessary on a supercomputer where anything running on it is more or less trusted code

I'm afraid security is important on typical HPC systems (at least university ones). I'm horrified to be assured cliinical data on the filesystems is secure when there are trivial privilege escalations on the system. We really do need a microkernel-type capability operating system, though most of what I see is failure to do system management or sane configuration of the batch system.

I recommend the book "The Supermen" about Seymour Cray, National Labs, etc and that will give you the necessary context to understand. Bottom line, it didn't used to be this way and supercomputing degenerates to lowest common denominator without starting from square one every now and then.

Reply on news.ycombinator.com