Hacker News Reader

Comment by zzzcpan | Hacker News Reader

> Technically speaking, they were quite far ahead of the curve. While VMs were all the rage, they were shipping container-based products

> The technical abilities of the Joyent folks were well beyond just about anybody else for a long time.

None of that was ever true. Whether they had container-based or VM-based tech was a tiny unimportant detail for a cloud company. Although over engineering in this area certainly made it impossible for them to compete with more traditional hosting companies. And to be able to compete with Amazon they needed something they didn't have - distributed systems background, not Sun background, which was anything but.

View on HN

Sun Microsystems literally invented the network is the computer. They were the first company ever to even begin to think in terms of distributed systems on a network scale (DEC had done some pioneering work on the hardware front prior to that with VMS, but not the network). The network file system, NFS, exists because Sun Microsystems engineers willed it so -- they wanted to multiply the computational power of smaller systems into a larger, distributed one.

They were a pioneer in distributed systems computing. Not many computer companies can claim that. Even fewer companies can claim engineers which hold patents which now power the InterNet, like Radia Perlman does for the spanning tree routing protocol. If that's not distributed, I don't know what is.

All of this experience is pretty shallow and completely irrelevant to distributed systems in the age of the cloud computing and the internet. For some reason Sun ignored the internet and didn't move into distributed systems at all. So, for example, this experience at Sun couldn't help the engineers in any way to build a distributed key-value storage for the cloud, while Amazon was publishing stuff like Dynamo paper.

You know, whenever I vow to stop explaining Sun, there is some yet-more outrageous claim that I just can't seem to summon the willpower to ignore, for fear that it will join a corpus of alternative facts that poison the wellspring of future generations.

So, in that spirit: Sun emphatically did not "ignore the internet" (?!) -- and was in fact a pioneer of distributed systems in many ways. Instead of rattling off the many important, early distributed systems that Sun built -- or the many pioneers of distributed systems that Sun employed -- I will vector you to a single Sun-authored paper circa 1994, "A Note on Distributed Computing"[1] (and to Chris Meiklejohn's discussion of it at Papers We Love[2]). The paper itself drips with wisdom that still feels current and important a quarter of a century later -- and it very much reflects the zeitgeist of the time at Sun. Please read the paper -- and more generally, take the time to educate yourself as to the history of our domain and the many people and companies who have made seminal contributions to it; the generations whose shoulders your work stands upon will thank you for it!

[1] https://github.com/papers-we-love/papers-we-love/blob/master...

[2] https://www.youtube.com/watch?v=z79mjsLdkpY

Just to add to the "ignored the internet" bit:

This is the company that had "the dot in .com" as its marketing slogan.

https://www.youtube.com/watch?v=njnNVV5QNaA

https://slashdot.org/story/00/04/20/1542217/sun-no-longer-th...

Ignored the internet in terms of distributed systems, which I assumed was obvious from the wording.

Look at that 1994 paper, none of the things in the paper show any background to build a distributed key-value storage or really anything for the cloud. I'm not sure why people are trying to claim otherwise. Even ZFS from a decade later reflects both a complete lack of experience in distributed systems and ignorance towards the internet.

"The network is the computer" doesn't feel like a slogan that a company ignoring distributed systems would use

Ahh yes, knew I was forgetting an important one...

Oh, and SUN was the Stanford University Network.

I guess I rephrase: they didn't do any of the distributed systems stuff necessary for the cloud if people so desperately want to call all the network stuff they did "distributed systems". Technically they were distributed systems, sure, just like anything client-server, but irrelevant to the challenges of fault tolerant geographically distributed asynchronously communicating systems.

... Does this statement come from having actually used things like Hadoop on Sun Grid Engine or any of Sun's other products? Or... what? What knowledge are you drawing on?

Edit: I mean, this is the company that gave us the 8 Fallacies of distributed computing, as they were 20yrs ago

Sun Microsystems literally invented geoclustering! Wow... just... wow.

> Sun Microsystems literally invented geoclustering!

They didn't. Not that any of it is important anyway. All 80s and early 90s networking tech was ridiculously broken, which should be obvious, since distributed systems field was just born at that time (first papers in like 1978). Things started to get useful only by the end of nineties. But by that time Sun's tech was nowhere near anything internet companies were doing. So later when things got to infrastructure Sun's engineering was really really behind, they had to start from scratch on anything that internet companies were learning and doing for many years. And this was very different time consuming knowledge, not something where someone could use their OS level expertise. Which also makes it very hard to compete with companies that are much farther ahead, especially with this silly "our tech is the greatest" attitude making it hard to even realize how far behind you actually were.

Sun Microsystems was the #1 computer company by revenue at the end of '90's. Everybody ran their hardware and Solaris; that's where the slogan we're the dot in .com came from, because it was true. If you had been around for it, you'd know it. Your comments date you...

I completely disagree, see my comment above. I was doing asynchronous distributed data replication and compute over a grid in 2001 using Jini and Javaspaces, and I was the one late to the party.

So old broken networking tech (Jini and Javaspaces) over a fast reliable local network (a grid) in 2001. How could this knowledge help anyone with fault tolerance, high-availability, partitioning/sharding, CAP consistency/availability trade offs, etc., which internet companies at the time were figuring out?

Why does ZFS reflect a lack of experience in the internet and/or distributed systems? Simply because it didn't follow a questionable tech trend of "everything is a DHT?"

Because they didn't seem to know any of the simple and nice ideas from distributed systems to solve pain points of managing space, since adding, removing nodes, sharding/partitioning is something you have to figure out pretty much from the start.

You seem to believe a shitty general key-value store with no SQL interface and no ACID guarantees qualifies as a distributed system, which is terribly shallow and misguided.

I worked at a company circa 2005 where we had a platform as a service built on top of Solaris 10 and zones, on-premise infrastructure which you now call cloud. Fully configured and serving Sun clusters would come into existence by merely DHCP booting from the OpenBoot PROM (Sun had to give us patches for the OBP for DHCP booting). I happen to know this, because I was one of the senior system engineers who helped architect it and then built it. We had it powering tens of thousands of servers. We were one of the first users of IP Multipathing in Solaris ("IPMP"), a technology Sun engineers invented to provide redundancy for network connectivity for the clusters. Amazon wasn't even a smear yet. That was in 2005. Which year is it now? How many years ago was that?

I believe ZFS represented a missed opportunity by spending all that effort on a local filesystem instead of trying to do something interesting in the distributed space. I even said so at the time, since I was working on distributed filesystems at the time, but that's just a strategic issue (and one that's quite arguable). It's not evidence of ignorance or lack of experience.

Well, on that part I was referring more specifically to the vdev mess and raidz, how you cannot just add a disk to the file system. Which is pretty silly, considering that at the time there were plenty of ideas how to do it much better, although not for disks, but for nodes in distributed systems. But apparently no one on ZFS team had any idea about any of it.

That was a pretty common pattern at Sun. Engineers there did a lot of great work, no denying it. Like many other projects, ZFS solved some pretty difficult and important problems while ignoring others. Which ones get talked about ad nauseam? We all make mistakes, of course, but some of those ignored problems were well known at the time to be worth solving. Did those self-professed "complete engineers" learn anything from those times they missed the target? The fact that Sun doesn't exist any more suggests an answer.

The problem is they cannot all be solved with just one single thing. Jeff Bonwick and his team knew that. They knew what was left on the table and they had to make that call. A completely fault tolerant, distributed system which can correctly arbitrate a write to a single shared resource (usually a file or even a block) from multiple nodes at the exact same time is a problem which has not been solved correctly yet, to the best of my knowledge and belief. I myself have spent considerable amount of time (20 years, to be precise) trying to solve that particular problem correctly for all edge cases. That problem is terra incognita in computer science and yet to be throughly researched, let alone understood. All attempts at solving that particular problem 100% correctly have not been successful, by anyone, but if you know of someone, please do tell, I'd love to be corrected on this and start using such technology immediately.

The ZFS team was well aware of this and that's why they made the trade-offs they did. They picked a subset of problems and they solved them. Even in hindsight, to me at least, it's obvious why.

But to suggest that Sun Microsystems doesn't exist because the company's engineers picked only a subset of problems they knew they could solve is... either terribly ignorant or wilfully malicious, or both, since malice often stems from ignorance. Sun Microsystems lost because they were too expensive, more expensive than garbage intel personal computer servers and because Solaris wasn't open source code in 1993. It's that simple.

ZFS doesn't work that way: it works on pools of storage. Disks are added to the pool.

Filesystems (or datasets) are carved out of a pool, not out of physical disks directly. There are vdevs on top of physical disks.

That's like people who are stuck back in the '80's because of ext3 arguing that ZFS is crap because it doesn't come with fsck or "some sort of repair tool" -- they just can't wrap their head around the ZFS concepts.

Moreover, Sun’s legacy in distributed system goes far beyond theory, people forget about Jini (and Javaspaces) a distributed computing platform that shipped by 2000 or before (I had the pleasure to work with and it was the only reason I would develop in Java), SOA? micro services? In-memory distributed computing? It was all there at work. Want P2P?, enter JXTA. There you go.

Whatever... They are now history and lately remembered by hanging over Java to Oracle... Everything else will be buried in history or 42nd page of Google search results

Because "Linux is the best", or some bullshit like that?

This is just willful ignorance.

Prime Computer Inc. had transparent remote disk access across PrimeNet in 1980. You could not distinguish being on a local disk vs remote disk, other than performance. PrimeNet ran over token ring (they also had fiber optic token ring running at Ford's main Dearborn, MI campus), full-duplex synchronous, and half-duplex dial-up synchronous network connections.

Prime had remote procedure calls. Their MIDAS product (index sequential files, basically key-value store with multiple indexes) allowed multiple nodes to access the same file by allocating a "slave" on the node hosting the file, forwarding all calls over the network, and returning call results. OS file access worked the same way, using the same slave for all remote procedure calls. Source code designed for local file access didn't require any changes to be used for remote files, just like NFS.

I'd say Sun had a pretty good example to go by for NFS. In fact, if you want to see this in action, I wrote a Prime minicomputer emulator and have remote disks available between 5 different revs of their operating system, spanning a 10-year period.

Telnet em.prirun.com 8001 and enjoy!

Wow, they invented the slogan. Sun alumni seem awfully thin-skinned about not getting credit for every little thing they did, which is why it's pretty unforgivable to deny DEC credit for anything more than "some work on hardware". I was working on DECnet-based remote file access software in 1989. It wasn't new, but NFSv2 was. DEC already had real clusters doing real work before Sun did. Within the company there was already a globe-spanning LAT network. I had friends at Apollo who were also way ahead of Sun wrt distributed computing. AT&T's RFS was at least contemporary with NFS, as were Netware and Vines. Sun was definitely a player in that space and made some seminal contributions, but to say they were the first even to think about it is unmitigated BS.

I am not a Sun Microsystems alumni. I wish I was, just as I equally wish I were an SGI alumni.

Yes DEC did have clusters, and I remember booting my Alpha workstations from a Sun SPARCStation 20 over DECNet as an AutoClient and I know DEC had hardware clustering and VMS working together as a distributed system, but since I came at the end of that era and only worked with Ultrix and DECUnix, I don't know any more about it which is why I couldn't go into detail. I only go into detail when I know about something and usually I've done that something myself.

"And to be able to compete with Amazon they needed something they didn't have - distributed systems background, not Sun background, which was anything but."

lol. "The network is the computer."

I'm reminded of something a smart engineer said at work today. "Our customers don't care how we implement the system, they just care that it works"

That's how you end up with unmaintainable solutions and code-bases which die under the cost of just trying to keep the system running, and because they lose the ability to evolve.

I guess I should have provided more context. It was in context of talking about a rewrite into another language to make something more maintainable and performant. To explain how hard that was to get right, and that if we messed it up, customers only really care that we messed it up.

Reply on news.ycombinator.com