Hacker News Reader

Comment by SwellJoe | Hacker News Reader

Technically speaking, they were quite far ahead of the curve. While VMs were all the rage, they were shipping container-based products (their Sun refugees kept developing Solaris Zones and ZFS in this context). The technical abilities of the Joyent folks were well beyond just about anybody else for a long time.

One could argue they were too early. For several years they had the technology to beat Amazon, until Amazon's massive scale washed over the whole market. The founders did well, so it's not like they "lost", but it's kinda sad to see a technically visionary set of products reach a dead end (one could maybe even view this as the final commercial death of what Sun built).

Also worth noting, their stack is open source in ways that nobody else is, I think. Maybe Google is doing right with Kubernetes, but everybody else is consuming OSS software to build proprietary clouds and giving basically nothing back. Joyent were doing most of the work, and giving most of the tech away.

View on HN

I don't know if it was the immediate cause of their lack of success, but they had also brought with them from Sun the Sun attitude of doing stuff that their engineers thought was cool (and it probably was) without paying attention to how much people are willing to pay for it and whether their focus is correct from a market perspective. For example, Bryan Cantrill, Joyent's CTO, liked blaming Oracle's Larry Ellison for the demise of Solaris, on which Cantrill had worked, but by the time Ellison acquired the failing Sun, Solaris, cool technology and all, had been pretty much dead or dying for a while -- at the hands of Linux, and not much helped by the Sun attitude. Cantrill was furious and blamed Ellison of being monomaniacal about making money ("don't anthropomorphize Larry Ellison," he said) but perhaps it was Cantrill who was monomaniacal all along, not keeping his eyes on the market. Then again, luck plays such an important role in these things that nice narratives rarely give the whole picture.

Solaris's death at the hands of Linux can be attributed in part not just to "the Sun attitude" but specifically to the attitude of Bryan Cantrill personally. In response to Dave Miller explaining how Linux's networking was technically superior (at that moment) due to more care for things like TLB footprints and system call overhead, and also had more accessible technical support, Cantrill famously decided to respond by making fun of Miller for being a nerd: "Have you ever kissed a girl?"

https://www.osnews.com/story/28261/have-you-ever-kissed-a-gi...

https://web.archive.org/web/20110721130404/https://cryptnet....

Admittedly, this was in 1996, and it's possible Cantrill regretted this remark, or the nerd-hating attitude behind it, at some point in the intervening 23 years. But I've never seen him admit it. I see he's participating in this thread, so if you see him not replying to this message, that's him doubling down.

He has said he regrets it in a Reddit AMA[1]. I find it quite strange that you attribute the death of an entire company to the attitude of a single engineer in an email from 1996 where he made a silly remark. You'd think if Cantrill was that powerful of a king-maker you'd want to be much nicer to him. Not to mention that Cantrill himself is also clearly a nerd, so "nerd-hating" is a bit of a weird statement.

[1]: https://www.reddit.com/r/IAmA/comments/31ny87/comment/cq3e4y...

Consider the remark not as the cause of death of the company, but the remark as an instance of a general attitude, and that attitude fitting in quite well at the company.

I have never in my life dealt with engineers so competent, professional and kind as the Sun Microsystems folks were. If they were technologically arrogant, they earned it: their inventions outlasted the company and live on in many products.

I cannot name a single GNU/Linux "hacker" inventing anything. Everything they implemented is a shoddy copy of what the engineers at hp, SGI and Sun Microsystems did. They might have shoddily implemented it by writing the code from scratch, but they blatantly stole everything. For instance, the manual page for chkconfig(8) will even tell one that the command comes from SGI's IRIX.

While I definitely agree that Sun, HP, and SGI had amazing engineering this section of your comment is particularly ridiculous:

> I cannot name a single GNU/Linux "hacker" inventing anything. Everything they implemented is a shoddy copy of what the engineers at [...] did.

If we ignore the obvious counter-arguments (GCC, Apache, nginx, arguably git, and so on), and just look at your statement at face value -- you're saying that nobody from the largest free software community in the world has invented anything new in the past 10-15 years. What possible evidence do you have that this is true? It's not just wrong, it's offensively so.

Heck, ZFS-on-Linux is now the repo-of-record for ZFS -- that's a specific example of the GNU/Linux community implementing things alongside the rest of the free software community (and currently ZoL has more features than any other ZFS port).

It's also the case that those brilliant engineers did the same thing. Zones are arguably a re-implementation of Jails with some slightly different design goals. IOCP is objectively just a copy of kqueue (to the point where Cantrill said he wished they'd just ported kqueue). And so on. I don't think this is a bad thing at all, but it's quite strange to put them up on a pedestal to the point where you effectively say that any engineer who didn't work at Sun/HP/SGI in 2003 hasn't invented anything.

There are obviously plenty of examples of Linux not learning from others (Jails/Zones vs containers, kqueue/IOCP vs epoll, DTrace vs eBPF+bpftrace, and so on). But arguing that nothing innovative has come out of the GNU/Linux community ever is just awful.

> the obvious counter-arguments (GCC, Apache, nginx, arguably git...

...and Wikipedia, BitTorrent, the WWW (open-source, though developed on NeXTStep), the Objective-C compiler that made it possible, Emacs (and its idea of an extensible self-documenting editor), Perl, bash, the dpkg/apt system the app stores ape poorly, the CTAN and CPAN systems it derived from, most aspects of decentralized source control in the form of arch (and then later Mercurial and Git), rsync, Docker, Nix, GNU make, Python 3, gold (the linker), iptables, Enlightenment, the Hurd, MySQL, PNG, Pango, Numpy, IPython/Jupyter, Ruby, the X11 Render extension, LADSPA, Valgrind, asan/ubsan, basically everything on CPAN, all the inventions in x264, ...

I'm not sure Apache belongs in there, though. Rob McCool might take exception to being called a "GNU/Linux hacker", though I don't think Robert Thau would mind.

> There are obviously plenty of examples of Linux not learning from others (Jails/Zones vs containers, kqueue/IOCP vs epoll, ...

Oh c'mon, is the problem that Linux doesn't innovate enough or that Linux innovates too much? Epoll is an example of Linux arguably innovating too much.

Not a single one, huh? I'll help: https://git-scm.com/

There is a counter-argument to git -- namely Sun had TeamWare[1] (the prequel to BitKeeper) and thus you could argue that git is still a "shoddy reimplementation". But this of course ignores the fact that TeamWare was almost completely unknown outside Sun until BitKeeper came along (at which point it was just an element of BitKeeper's history), and that git is objectively superior in many metrics to TeamWare.

Personally I think GCC, Apache, and nginx are much better counter-arguments.

[1]: https://en.wikipedia.org/wiki/Sun_WorkShop_TeamWare

Quite unfortunately for all of us condemned to use GNU/Linux:

- GCC exists because Sun Studio compilers weren't free back then;

- Apache is the next generation NCSA web server;

- nginex is a re-invented Apache wheel.

Do you have any more examples to get corrected on? I'll be glad to set you straight, in the hope that unlike your Linux buddies you are capable of learning.

They are simply amateurs wanting to play engineers, but they never were engineers and never will be, and it shows in just how shitty GNU/Linux based operating systems are. Hacked up together by hackers according to the "hack it 'till it works, man!" motto. And they don't learn from their mistakes or from the mistakes of others, on top of being deeply convicted that they are the best.

Find me one technology invented in GNU/Linux which UNIX did not already invent and I will publicly retract my statement here. Just one.

Okay, here are two examples: rsync and restic[1] (an evolution of borgbackup[2]). You already know what rsync is, but the other two are backup tools that offer encryption and content-based deduplication. There is no distinction between a "full" backup and an "incremental" backup.

Now, I already know that you're going to say that "zfs send" already does these things -- but it really doesn't. First of all, ZoL only recently got encryption support so built-in encryption with ZFS only came around recently (and was developed by someone from the GNU/Linux community, by the way). Also, ZFS's dedup is so expensive that it's very strongly recommended that people don't use it unless they really need it. Deduplication with restic and borgbackup is content-based which means that it's far more resilient to shifting bytes in files and it's effectively free because everything is stored as a CAS (to be fair, it's only mostly free because people don't use it as a filesystem).

Again, I really am not bagging on Sun here. I just think it's quite ludicrous that you're saying that any engineer who didn't work at Sun pre-2010 never invented anything. I refuse to believe that you honestly believe that, purely based on how ludicrous of a concept it is.

[1]: https://restic.net/ [2]: https://www.borgbackup.org/

Sorry buddy, but https://illumos.org/man/1/filesync predates rsync.

Borgbackup is a third party, unbundled application. I don't see what it has to do with people hacking on GNU/Linux like mad.

"I just think it's quite ludicrous that you're saying that any engineer who didn't work at Sun pre-2010 never invented anything."

You didn't get it quite right: I'm saying that professional engineers worked at Sun Microsystems, hp and SGI. That's statement #1. Statement #2 is that people working on GNU/Linux are amateurs who didn't invent anything. Statement #3 is that because they are amateurs who are incapable of learning, they just keep hacking shit together haphazardly and will never be engineers like the people from Sun, hp and SGI. Especially SGI: SGI had the best engineers. Those people were way ahead of their time in every technical aspect imaginable.

"I refuse to believe that you honestly believe that, purely based on how ludicrous of a concept it is."

Believe it. GNU/Linux people are amateurs. They don't have it in them. They want to be thought of as engineers but they aren't. They just aren't capable of it. That's why GNU and GNU/Linux are garbage and will never be anything more than a pile of haphazard, shoddily slapped together hacks. That's the world we live in now, where IT is shit thanks to their shitty, shoddy work.

Dave Chinner is an ex-SGI engineer who has been working on XFS since the beginning of the project and is currently a Linux kernel maintainer. Brendan Gregg is ex-Sun and worked on DTrace and currently is helping improve Linux's tracing capabilities. Are they "amateurs who are incapable of learning" or "professional engineers"?

I've seen comments from you about how GNU/Linux is awful in every respect ever since you created your account ~3 years ago. I really can't imagine being so stuck in a mindset that you feel the need to spend so much of your time being angry about such a large community of people. Honestly, I just don't get it and I earnestly hope that this is just an online persona you have.

Dave and Brendan are former SGI and Sun engineers. They are refugees. They gave up. So once again we're back to what I said: either one came from SGI, or hp, or Sun Microsystems, but one sure as hell didn't come from the GNU/Linux being an engineer, nor will such a thing ever happen. For 20 years they've been hacking furiously on Linux and still the basic things don't work correctly. That's what happens when one hacks instead of system engineering solutions.

"I really can't imagine being so stuck in a mindset that you feel the need to spend so much of your time being angry about such a large community of people."

Computers were my life, my passion, my calling. They destroyed all that passion in me with their GNU and their GNU/Linux. I will not forget. I will not forgive. They made my professional life a living hell. I hate working in IT because of those people and their mentality.

"Honestly, I just don't get it and I earnestly hope that this is just an online persona you have."

  You must be the change in the world you wish to see.

And I believe that, deeply. I'm absolutely convicted about it. Even if I single handedly must teach apprentices, I will continue to fight until the last drop of blood against GNU and against GNU/Linux and teach people, person by person if I must, as I have been, showing them one on one how awful it is and how superior and easy a real UNIX like SmartOS is. Either until I die or manage to get out of IT. Because truly awful things like GNU and GNU/Linux deserve to be relegated to dark history of IT, and some good and beautiful, well thought out things like SmartOS are worth fighting for. I will continue to fight against ignorance and trend pandering and for enlightment. I will strain like buddha strained if I must. Some things are just worth fighting for and some things one must make a stand against, no matter the cost to oneself. I am forced to work on and with GNU/Linux because people think like you, but I will fight it and resist it every step of the way and I will continue to teach because working on Linux and with GNU/Linux is more work, not less work for me. That alone is more than enough to motivate me.

Ha ha -- wow! I had no idea that I possessed such power over time and space! As has already been pointed out to you (and has been pointed out repeatedly on HN over the years), I do regret the remark. So, there's that.

I would also note that the rise of Linux in the late 1990s and early 2000s corresponds with the performance of x86 surpassing the performance of the RISC microprocessors during that same period -- a time when the only other open source Unix was operating under a legal cloud from AT&T. It seems likely that these forces played a greater role than a twentysomething engineer shooting his mouth off on Usenet...

Those were the days!

I think you're right regarding x86 performance being a huge factor. Also my own experience, admining a number of Solaris systems in those days, I just found it more frustrating to use. The first thing I would do was put GNU userland on there to make a Solaris system more usable.

Later on when I had more say in what systems things ran on, I'd reach for Linux because it was easier. It was good enough. I could easily run it at home too, and it ran on cheap hardware.

I sometimes wonder how much that usability factor played a role...

Though AWS seems to be doing fine as a business, so maybe this theory about usability isn't so great.

When you could run Solaris on the same hardware (x86), it was noticeably slower than Linux.

That was over 15 years ago!

It's now faster than GNU/Linux on the same hardware and has been for the past ten years.

FWIW when I asked the "what's Joyent all about" question I never expected or hoped this would become a "let's blame Bryan Cantrill for random shit" flamefest. I'm not sure where all of that is coming from and I'm sorry if somehow I triggered all that.

You've been a great example to me and many others, in doing open source, in taking a principled stance (even if I subtly disagree with some of your principles) and also in showing that yes, it's totally possible to found a company with an incomprehensible business model, grow it, have it contribute lots and lots to awesome open source, and somehow despite all the free work and the incomprehensibilities, flip it for a profit. One day I hope to do the same and all of HN will be like "huh?"

Hey, thanks for that! The HN comments here today have been full of surprises, not least the intense personalization of it all! I believe that our impact in the world is much more in the people we inspire than in those we rankle -- so in that regard, it's deeply gratifying to know that I have been of service; thank you!

> full of surprises, not least the intense personalization of it all!

I'm surprised you're surprised by this, considering that you have contributed much to the personalization of such matters. The personal "psychologization" of complex social dynamics is pretty much the trademark of many of your talks. If you find psychological explanations attractive, why are you surprised when others do, too?

https://www.youtube.com/watch?v=Rqb4V9GxaBo

If I am not mistaken, Cantrill stole the line from this skit.

"The good old days." I actually went to a Star Trek convention as a kid back in the late 70's/early 80's, don't remember the year.

It was a different time. "Cool" meant something different back then. People really could say "Get a life!" and it was not an insult. It was truth. There was more to life back then than computers.

(There still is, but today this simple truth is being lost on many more people.)

This is patent nonsense. Sun lost because they were not price competitive. They thought they could command outrageous sums and 46% profit margins because they sold UltraSPARC-based servers and in the case of AMD / intel servers because they bore the Sun Microsystems logo. Meanwhile Linux ran on shitty personal computer hardware and was just good enough for enough kids who didn't have a clue about correctness of operation or reliability and still to this day don't, as is often evident from many, many comments on the subject of GNU/Linux here.

Sun Microsystems stubbornly refused to lower prices to undercut the generic shitty personal computer used as a server and that cost them the company. hp and IBM made the same mistake at expense of PA-RISC / HP-UX and POWER / AIX respectively. Oracle corporation is in the process of repeating that mistake. They all thought they could get away with outrageous profit margins and pricing. It's like once they started shipping servers to enterprises they all lost compass. Meanwhile, the whole market commoditized. They all wouldn't accept that. Now they've been reduced to rubble. hp "Enterprise" is missing in action somewhere peddling a server here and there, IBM started whoring itself around as a "re-invented" service company (the next "Computer Associates"?) and other than Exadata I don't know of a single business willingly buying Oracle servers. Yeah, it's very hard to predict how that will end. "Very hard".

At any rate, all of that has nothing to do with Bryan. Zero.

To be clear, I have never blamed Ellison for the demise of Sun or even Solaris -- but I do blame Oracle for re-proprietarizing OpenSolaris, and thereby ripping up the social contract of copyright assignment in an open source project. If it must be said, this wasn't an isolated incident -- and in hindsight, was merely the front of a conga line bad Oracle behavior.

As for my own putative ignorance of market demands, one of Sun's growth areas at the time of acquisition in terms of revenue was the group that I had co-founded within Sun[1][2] -- in part because we had listened carefully to the market, and built a product that it very much wanted.

Finally, if we want to engage in the perennial musing as to the demise of Sun, I think for my part I still broadly stand by my analysis from 2011.[3]

[1] http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-...

[2] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/

[3] https://news.ycombinator.com/item?id=2287033

"perhaps it was Cantrill who was monomaniacal all along, not keeping his eyes on the market"

Joyent made money and was acquired, all while producing metric shit tons of incredibly cool Open Source software. We should all be so "monomaniacal".

We have an art installation at the corporate campus where I work.

It's a parody of 1984, with inspirational quotes flowing over the entire building (300meters long). But they are not inspirational, rather artsy sometimes edgy and provocative.

The sentences change every day, and are controlled by an artist in California somewhere.

One of the sentences one day was: "Monomania creates success"

I keep a logbook of the strangest entries.

Would you happen to be working in Germany?

No, but maybe there are several?

There are 430 statements. Holzer’s work reflects both a political and with a focus on interpersonal relations and a moral perspective. With her fleeting forms – texts that sweep past – she symbolizes aspects of life itself. Holzer represents a melancholic and committed voice in contemporary art.

* Strange section *

In a paradisiac state the basic survival tasks you need to perform all seems clear and your objective is right there in front of you.

There is a period where you know you have gone wrong but continue

Monomania creates success

* Very dark section *

Scream so powerful that it empties all the kindness you could have shown and blackened the entire fabric of your soul

Form a noise so true that your tormentor recognizes it as a sound that can come from his throat

Murder has its sexual side

You are caught thinking about killing anyone you know

Sometimes you have no other choice but to watch something gruesome occur.

Just as a side note here, Google is doing Kubernetes as a commoditize your compliment strategy (https://www.gwern.net/Complement) so they can sell other cloud compute services. This makes it easier for people to migrate their on-prem workloads to Kubernetes, and then to the cloud. This commoditize much of what Amazon was attempting with ECS. Forcing Amazon to fast follow with EKS, arguably a worse and much more complex to deploy product than Google GKE.

It makes it so it's hard to make money off container orchestration, but you can still make money off what Google is good at, or at least thinks they are good at. Selling compute, proprietary databases, and stuff like TPUs.

My point is that yes, Google is advancing the whole industry by releasing Kubernetes as opensource, but it's also a strategy to sell more of GCP and deny some revenue to competitors.

It remains to be seen if it works. GCP is still a distant 3rd in the cloud race and has a long way to go to improve their enterprise support. (Their usability is much nicer though...)

> it's kinda sad to see a technically visionary set of products reach a dead end (one could maybe even view this as the final commercial death of what Sun built).

I think it's not really the end of the technology, but it is an end to a generally available public commercial offering of it. It sounds like they're still offering services, but only for "single-tenant", which after some careful reading (because I wasn't sure how to interpret that initially), I think means that they'll provide their software and managemest system for private install for large orgs in their private deployments, maybe along with a support/management contract?

That's basically putting technology out to pasture. When you've got a handful of large corporate customers that depend on a product, you can continue to extract money out of them for years or decades without the product having a future.

The question is whether they are still actively pursuing enterprise clients. If they aren't, then I agree. If they are, then it's not putting it out to pasture, it's just a different business model. There are plenty of companies that do this. My brother works for one that does it for large network storage deployments using proprietary technology, and I imagine they don't want to hear from you unless you are willing to put out hundreds of thousands to millions of dollars on a deployment (and yet they still have new deployments). I definitely wouldn't consider them putting the technology out to pasture because of that.

"but it's kinda sad to see a technically visionary set of products reach a dead end"

It's not a dead end since any company, public or private can use the code and the product (I for instance build my own modified SmartOS from source) to power their infrastructure.

Technically there is nothing to prevent anyone from taking the product and the code and starting the next Amazon. I know of at least one internet service provider in the UK who is leveraging SmartOS to provide commercial, for-profit service. There are probably more such businesses, but this is something which squarely falls under "I'm keeping my mouth shut and making money" category.

What would you state as the top reason(s) to choose SmartOS vs Linux for those use-cases?

1. full-blown virtual UNIX servers running at bare metal speeds (joyent-branded zones).

2. ability to run virtual Linux servers are bare metal speeds (and their applications).

3. simple and easy container management of 1. and 2. with imgadm and vmadm.

4. OpenZFS - easy capacity growth, checksummed data, silent data corruption detection and self healing.

5. lightning fast, faster than GNU/Linux performance on the same hardware.

6. same application portfolio as GNU/Linux (14,000 applications available with a simple pkgin command invocation).

7. ability to create virtual routers and switches out of thin air with dladm, join and partition physical links in terms of percentages, simplifying network topology in virtualized environments.

8. OS paranoid about protecting and self-healing itself and the data it hosts.

9. ability to pick between KVM (slower) or bhyve for paravirtualization of non-Linux OS's like Microsoft Windows, OpenBSD, FreeBSD et cetera (although this process is poorly documented).

10. NFS V3, V4, fiberchannel and iSCSI which work correctly.

There is one major downside (I don't consider it as such, but I know others do): one must pick the hardware compatible with SmartOS from the Joyent's hardware compatibility list, not the other way around: https://eng.joyent.com/manufacturing/bom.html.

But when it does fire up over the network and once one groks it, holy shit, it's like being taken into the future with a time machine.

This. Well put.

when eveyone was hooked on VM's and other nonsense they had 'branded zones' running.

The learning curve was high, but worth it once you had it running.

Sad to see them leave. wonder what i will do with my SmartOS based media server now?

For what it's worth, we're still maintaining SmartOS and it's still open source!

> Technically speaking, they were quite far ahead of the curve. While VMs were all the rage, they were shipping container-based products

> The technical abilities of the Joyent folks were well beyond just about anybody else for a long time.

None of that was ever true. Whether they had container-based or VM-based tech was a tiny unimportant detail for a cloud company. Although over engineering in this area certainly made it impossible for them to compete with more traditional hosting companies. And to be able to compete with Amazon they needed something they didn't have - distributed systems background, not Sun background, which was anything but.

"And to be able to compete with Amazon they needed something they didn't have - distributed systems background, not Sun background, which was anything but."

lol. "The network is the computer."

I'm reminded of something a smart engineer said at work today. "Our customers don't care how we implement the system, they just care that it works"

That's how you end up with unmaintainable solutions and code-bases which die under the cost of just trying to keep the system running, and because they lose the ability to evolve.

I guess I should have provided more context. It was in context of talking about a rewrite into another language to make something more maintainable and performant. To explain how hard that was to get right, and that if we messed it up, customers only really care that we messed it up.

Sun Microsystems literally invented the network is the computer. They were the first company ever to even begin to think in terms of distributed systems on a network scale (DEC had done some pioneering work on the hardware front prior to that with VMS, but not the network). The network file system, NFS, exists because Sun Microsystems engineers willed it so -- they wanted to multiply the computational power of smaller systems into a larger, distributed one.

They were a pioneer in distributed systems computing. Not many computer companies can claim that. Even fewer companies can claim engineers which hold patents which now power the InterNet, like Radia Perlman does for the spanning tree routing protocol. If that's not distributed, I don't know what is.

All of this experience is pretty shallow and completely irrelevant to distributed systems in the age of the cloud computing and the internet. For some reason Sun ignored the internet and didn't move into distributed systems at all. So, for example, this experience at Sun couldn't help the engineers in any way to build a distributed key-value storage for the cloud, while Amazon was publishing stuff like Dynamo paper.

You know, whenever I vow to stop explaining Sun, there is some yet-more outrageous claim that I just can't seem to summon the willpower to ignore, for fear that it will join a corpus of alternative facts that poison the wellspring of future generations.

So, in that spirit: Sun emphatically did not "ignore the internet" (?!) -- and was in fact a pioneer of distributed systems in many ways. Instead of rattling off the many important, early distributed systems that Sun built -- or the many pioneers of distributed systems that Sun employed -- I will vector you to a single Sun-authored paper circa 1994, "A Note on Distributed Computing"[1] (and to Chris Meiklejohn's discussion of it at Papers We Love[2]). The paper itself drips with wisdom that still feels current and important a quarter of a century later -- and it very much reflects the zeitgeist of the time at Sun. Please read the paper -- and more generally, take the time to educate yourself as to the history of our domain and the many people and companies who have made seminal contributions to it; the generations whose shoulders your work stands upon will thank you for it!

[1] https://github.com/papers-we-love/papers-we-love/blob/master...

[2] https://www.youtube.com/watch?v=z79mjsLdkpY

Just to add to the "ignored the internet" bit:

This is the company that had "the dot in .com" as its marketing slogan.

https://www.youtube.com/watch?v=njnNVV5QNaA

https://slashdot.org/story/00/04/20/1542217/sun-no-longer-th...

Ignored the internet in terms of distributed systems, which I assumed was obvious from the wording.

Look at that 1994 paper, none of the things in the paper show any background to build a distributed key-value storage or really anything for the cloud. I'm not sure why people are trying to claim otherwise. Even ZFS from a decade later reflects both a complete lack of experience in distributed systems and ignorance towards the internet.

"The network is the computer" doesn't feel like a slogan that a company ignoring distributed systems would use

Ahh yes, knew I was forgetting an important one...

Oh, and SUN was the Stanford University Network.

I guess I rephrase: they didn't do any of the distributed systems stuff necessary for the cloud if people so desperately want to call all the network stuff they did "distributed systems". Technically they were distributed systems, sure, just like anything client-server, but irrelevant to the challenges of fault tolerant geographically distributed asynchronously communicating systems.

... Does this statement come from having actually used things like Hadoop on Sun Grid Engine or any of Sun's other products? Or... what? What knowledge are you drawing on?

Edit: I mean, this is the company that gave us the 8 Fallacies of distributed computing, as they were 20yrs ago

Sun Microsystems literally invented geoclustering! Wow... just... wow.

> Sun Microsystems literally invented geoclustering!

They didn't. Not that any of it is important anyway. All 80s and early 90s networking tech was ridiculously broken, which should be obvious, since distributed systems field was just born at that time (first papers in like 1978). Things started to get useful only by the end of nineties. But by that time Sun's tech was nowhere near anything internet companies were doing. So later when things got to infrastructure Sun's engineering was really really behind, they had to start from scratch on anything that internet companies were learning and doing for many years. And this was very different time consuming knowledge, not something where someone could use their OS level expertise. Which also makes it very hard to compete with companies that are much farther ahead, especially with this silly "our tech is the greatest" attitude making it hard to even realize how far behind you actually were.

Sun Microsystems was the #1 computer company by revenue at the end of '90's. Everybody ran their hardware and Solaris; that's where the slogan we're the dot in .com came from, because it was true. If you had been around for it, you'd know it. Your comments date you...

I completely disagree, see my comment above. I was doing asynchronous distributed data replication and compute over a grid in 2001 using Jini and Javaspaces, and I was the one late to the party.

So old broken networking tech (Jini and Javaspaces) over a fast reliable local network (a grid) in 2001. How could this knowledge help anyone with fault tolerance, high-availability, partitioning/sharding, CAP consistency/availability trade offs, etc., which internet companies at the time were figuring out?

Why does ZFS reflect a lack of experience in the internet and/or distributed systems? Simply because it didn't follow a questionable tech trend of "everything is a DHT?"

Because they didn't seem to know any of the simple and nice ideas from distributed systems to solve pain points of managing space, since adding, removing nodes, sharding/partitioning is something you have to figure out pretty much from the start.

You seem to believe a shitty general key-value store with no SQL interface and no ACID guarantees qualifies as a distributed system, which is terribly shallow and misguided.

I worked at a company circa 2005 where we had a platform as a service built on top of Solaris 10 and zones, on-premise infrastructure which you now call cloud. Fully configured and serving Sun clusters would come into existence by merely DHCP booting from the OpenBoot PROM (Sun had to give us patches for the OBP for DHCP booting). I happen to know this, because I was one of the senior system engineers who helped architect it and then built it. We had it powering tens of thousands of servers. We were one of the first users of IP Multipathing in Solaris ("IPMP"), a technology Sun engineers invented to provide redundancy for network connectivity for the clusters. Amazon wasn't even a smear yet. That was in 2005. Which year is it now? How many years ago was that?

I believe ZFS represented a missed opportunity by spending all that effort on a local filesystem instead of trying to do something interesting in the distributed space. I even said so at the time, since I was working on distributed filesystems at the time, but that's just a strategic issue (and one that's quite arguable). It's not evidence of ignorance or lack of experience.

Well, on that part I was referring more specifically to the vdev mess and raidz, how you cannot just add a disk to the file system. Which is pretty silly, considering that at the time there were plenty of ideas how to do it much better, although not for disks, but for nodes in distributed systems. But apparently no one on ZFS team had any idea about any of it.

That was a pretty common pattern at Sun. Engineers there did a lot of great work, no denying it. Like many other projects, ZFS solved some pretty difficult and important problems while ignoring others. Which ones get talked about ad nauseam? We all make mistakes, of course, but some of those ignored problems were well known at the time to be worth solving. Did those self-professed "complete engineers" learn anything from those times they missed the target? The fact that Sun doesn't exist any more suggests an answer.

The problem is they cannot all be solved with just one single thing. Jeff Bonwick and his team knew that. They knew what was left on the table and they had to make that call. A completely fault tolerant, distributed system which can correctly arbitrate a write to a single shared resource (usually a file or even a block) from multiple nodes at the exact same time is a problem which has not been solved correctly yet, to the best of my knowledge and belief. I myself have spent considerable amount of time (20 years, to be precise) trying to solve that particular problem correctly for all edge cases. That problem is terra incognita in computer science and yet to be throughly researched, let alone understood. All attempts at solving that particular problem 100% correctly have not been successful, by anyone, but if you know of someone, please do tell, I'd love to be corrected on this and start using such technology immediately.

The ZFS team was well aware of this and that's why they made the trade-offs they did. They picked a subset of problems and they solved them. Even in hindsight, to me at least, it's obvious why.

But to suggest that Sun Microsystems doesn't exist because the company's engineers picked only a subset of problems they knew they could solve is... either terribly ignorant or wilfully malicious, or both, since malice often stems from ignorance. Sun Microsystems lost because they were too expensive, more expensive than garbage intel personal computer servers and because Solaris wasn't open source code in 1993. It's that simple.

ZFS doesn't work that way: it works on pools of storage. Disks are added to the pool.

Filesystems (or datasets) are carved out of a pool, not out of physical disks directly. There are vdevs on top of physical disks.

That's like people who are stuck back in the '80's because of ext3 arguing that ZFS is crap because it doesn't come with fsck or "some sort of repair tool" -- they just can't wrap their head around the ZFS concepts.

Moreover, Sun’s legacy in distributed system goes far beyond theory, people forget about Jini (and Javaspaces) a distributed computing platform that shipped by 2000 or before (I had the pleasure to work with and it was the only reason I would develop in Java), SOA? micro services? In-memory distributed computing? It was all there at work. Want P2P?, enter JXTA. There you go.

Whatever... They are now history and lately remembered by hanging over Java to Oracle... Everything else will be buried in history or 42nd page of Google search results

Because "Linux is the best", or some bullshit like that?

This is just willful ignorance.

Prime Computer Inc. had transparent remote disk access across PrimeNet in 1980. You could not distinguish being on a local disk vs remote disk, other than performance. PrimeNet ran over token ring (they also had fiber optic token ring running at Ford's main Dearborn, MI campus), full-duplex synchronous, and half-duplex dial-up synchronous network connections.

Prime had remote procedure calls. Their MIDAS product (index sequential files, basically key-value store with multiple indexes) allowed multiple nodes to access the same file by allocating a "slave" on the node hosting the file, forwarding all calls over the network, and returning call results. OS file access worked the same way, using the same slave for all remote procedure calls. Source code designed for local file access didn't require any changes to be used for remote files, just like NFS.

I'd say Sun had a pretty good example to go by for NFS. In fact, if you want to see this in action, I wrote a Prime minicomputer emulator and have remote disks available between 5 different revs of their operating system, spanning a 10-year period.

Telnet em.prirun.com 8001 and enjoy!

Wow, they invented the slogan. Sun alumni seem awfully thin-skinned about not getting credit for every little thing they did, which is why it's pretty unforgivable to deny DEC credit for anything more than "some work on hardware". I was working on DECnet-based remote file access software in 1989. It wasn't new, but NFSv2 was. DEC already had real clusters doing real work before Sun did. Within the company there was already a globe-spanning LAT network. I had friends at Apollo who were also way ahead of Sun wrt distributed computing. AT&T's RFS was at least contemporary with NFS, as were Netware and Vines. Sun was definitely a player in that space and made some seminal contributions, but to say they were the first even to think about it is unmitigated BS.

I am not a Sun Microsystems alumni. I wish I was, just as I equally wish I were an SGI alumni.

Yes DEC did have clusters, and I remember booting my Alpha workstations from a Sun SPARCStation 20 over DECNet as an AutoClient and I know DEC had hardware clustering and VMS working together as a distributed system, but since I came at the end of that era and only worked with Ultrix and DECUnix, I don't know any more about it which is why I couldn't go into detail. I only go into detail when I know about something and usually I've done that something myself.

Reply on news.ycombinator.com