Joyent Public Cloud EOL
docs.joyent.com
docs.joyent.com
But oh, they're cancelling the public cloud, which I assume was the hosting service? Then what are they now? I went to their site but I totally didn't get it, I'm obviously not the target audience.
So, what does Joyent really do, how to they make money, and how does it make business sense for them to do all that illumos/zfs/node etc etc stuff? Are they an AWS competitor? A Heroku competitor? An Oracle/ex-Sun competitor? I really can't place them.
One could argue they were too early. For several years they had the technology to beat Amazon, until Amazon's massive scale washed over the whole market. The founders did well, so it's not like they "lost", but it's kinda sad to see a technically visionary set of products reach a dead end (one could maybe even view this as the final commercial death of what Sun built).
Also worth noting, their stack is open source in ways that nobody else is, I think. Maybe Google is doing right with Kubernetes, but everybody else is consuming OSS software to build proprietary clouds and giving basically nothing back. Joyent were doing most of the work, and giving most of the tech away.
As for my own putative ignorance of market demands, one of Sun's growth areas at the time of acquisition in terms of revenue was the group that I had co-founded within Sun[1][2] -- in part because we had listened carefully to the market, and built a product that it very much wanted.
Finally, if we want to engage in the perennial musing as to the demise of Sun, I think for my part I still broadly stand by my analysis from 2011.[3]
[1] http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-...
Joyent made money and was acquired, all while producing metric shit tons of incredibly cool Open Source software. We should all be so "monomaniacal".
It's a parody of 1984, with inspirational quotes flowing over the entire building (300meters long). But they are not inspirational, rather artsy sometimes edgy and provocative.
The sentences change every day, and are controlled by an artist in California somewhere.
One of the sentences one day was: "Monomania creates success"
I keep a logbook of the strangest entries.
There are 430 statements. Holzer’s work reflects both a political and with a focus on interpersonal relations and a moral perspective. With her fleeting forms – texts that sweep past – she symbolizes aspects of life itself. Holzer represents a melancholic and committed voice in contemporary art.
* Strange section *
In a paradisiac state the basic survival tasks you need to perform all seems clear and your objective is right there in front of you.
There is a period where you know you have gone wrong but continue
Monomania creates success
* Very dark section *
Scream so powerful that it empties all the kindness you could have shown and blackened the entire fabric of your soul
Form a noise so true that your tormentor recognizes it as a sound that can come from his throat
Murder has its sexual side
You are caught thinking about killing anyone you know
Sometimes you have no other choice but to watch something gruesome occur.
https://www.osnews.com/story/28261/have-you-ever-kissed-a-gi...
https://web.archive.org/web/20110721130404/https://cryptnet....
Admittedly, this was in 1996, and it's possible Cantrill regretted this remark, or the nerd-hating attitude behind it, at some point in the intervening 23 years. But I've never seen him admit it. I see he's participating in this thread, so if you see him not replying to this message, that's him doubling down.
I would also note that the rise of Linux in the late 1990s and early 2000s corresponds with the performance of x86 surpassing the performance of the RISC microprocessors during that same period -- a time when the only other open source Unix was operating under a legal cloud from AT&T. It seems likely that these forces played a greater role than a twentysomething engineer shooting his mouth off on Usenet...
You've been a great example to me and many others, in doing open source, in taking a principled stance (even if I subtly disagree with some of your principles) and also in showing that yes, it's totally possible to found a company with an incomprehensible business model, grow it, have it contribute lots and lots to awesome open source, and somehow despite all the free work and the incomprehensibilities, flip it for a profit. One day I hope to do the same and all of HN will be like "huh?"
I'm surprised you're surprised by this, considering that you have contributed much to the personalization of such matters. The personal "psychologization" of complex social dynamics is pretty much the trademark of many of your talks. If you find psychological explanations attractive, why are you surprised when others do, too?
Later on when I had more say in what systems things ran on, I'd reach for Linux because it was easier. It was good enough. I could easily run it at home too, and it ran on cheap hardware.
I sometimes wonder how much that usability factor played a role...
Though AWS seems to be doing fine as a business, so maybe this theory about usability isn't so great.
It's now faster than GNU/Linux on the same hardware and has been for the past ten years.
[1]: https://www.reddit.com/r/IAmA/comments/31ny87/comment/cq3e4y...
I cannot name a single GNU/Linux "hacker" inventing anything. Everything they implemented is a shoddy copy of what the engineers at hp, SGI and Sun Microsystems did. They might have shoddily implemented it by writing the code from scratch, but they blatantly stole everything. For instance, the manual page for chkconfig(8) will even tell one that the command comes from SGI's IRIX.
> I cannot name a single GNU/Linux "hacker" inventing anything. Everything they implemented is a shoddy copy of what the engineers at [...] did.
If we ignore the obvious counter-arguments (GCC, Apache, nginx, arguably git, and so on), and just look at your statement at face value -- you're saying that nobody from the largest free software community in the world has invented anything new in the past 10-15 years. What possible evidence do you have that this is true? It's not just wrong, it's offensively so.
Heck, ZFS-on-Linux is now the repo-of-record for ZFS -- that's a specific example of the GNU/Linux community implementing things alongside the rest of the free software community (and currently ZoL has more features than any other ZFS port).
It's also the case that those brilliant engineers did the same thing. Zones are arguably a re-implementation of Jails with some slightly different design goals. IOCP is objectively just a copy of kqueue (to the point where Cantrill said he wished they'd just ported kqueue). And so on. I don't think this is a bad thing at all, but it's quite strange to put them up on a pedestal to the point where you effectively say that any engineer who didn't work at Sun/HP/SGI in 2003 hasn't invented anything.
There are obviously plenty of examples of Linux not learning from others (Jails/Zones vs containers, kqueue/IOCP vs epoll, DTrace vs eBPF+bpftrace, and so on). But arguing that nothing innovative has come out of the GNU/Linux community ever is just awful.
...and Wikipedia, BitTorrent, the WWW (open-source, though developed on NeXTStep), the Objective-C compiler that made it possible, Emacs (and its idea of an extensible self-documenting editor), Perl, bash, the dpkg/apt system the app stores ape poorly, the CTAN and CPAN systems it derived from, most aspects of decentralized source control in the form of arch (and then later Mercurial and Git), rsync, Docker, Nix, GNU make, Python 3, gold (the linker), iptables, Enlightenment, the Hurd, MySQL, PNG, Pango, Numpy, IPython/Jupyter, Ruby, the X11 Render extension, LADSPA, Valgrind, asan/ubsan, basically everything on CPAN, all the inventions in x264, ...
I'm not sure Apache belongs in there, though. Rob McCool might take exception to being called a "GNU/Linux hacker", though I don't think Robert Thau would mind.
> There are obviously plenty of examples of Linux not learning from others (Jails/Zones vs containers, kqueue/IOCP vs epoll, ...
Oh c'mon, is the problem that Linux doesn't innovate enough or that Linux innovates too much? Epoll is an example of Linux arguably innovating too much.
Personally I think GCC, Apache, and nginx are much better counter-arguments.
- GCC exists because Sun Studio compilers weren't free back then;
- Apache is the next generation NCSA web server;
- nginex is a re-invented Apache wheel.
Do you have any more examples to get corrected on? I'll be glad to set you straight, in the hope that unlike your Linux buddies you are capable of learning.
They are simply amateurs wanting to play engineers, but they never were engineers and never will be, and it shows in just how shitty GNU/Linux based operating systems are. Hacked up together by hackers according to the "hack it 'till it works, man!" motto. And they don't learn from their mistakes or from the mistakes of others, on top of being deeply convicted that they are the best.
Find me one technology invented in GNU/Linux which UNIX did not already invent and I will publicly retract my statement here. Just one.
Now, I already know that you're going to say that "zfs send" already does these things -- but it really doesn't. First of all, ZoL only recently got encryption support so built-in encryption with ZFS only came around recently (and was developed by someone from the GNU/Linux community, by the way). Also, ZFS's dedup is so expensive that it's very strongly recommended that people don't use it unless they really need it. Deduplication with restic and borgbackup is content-based which means that it's far more resilient to shifting bytes in files and it's effectively free because everything is stored as a CAS (to be fair, it's only mostly free because people don't use it as a filesystem).
Again, I really am not bagging on Sun here. I just think it's quite ludicrous that you're saying that any engineer who didn't work at Sun pre-2010 never invented anything. I refuse to believe that you honestly believe that, purely based on how ludicrous of a concept it is.
Borgbackup is a third party, unbundled application. I don't see what it has to do with people hacking on GNU/Linux like mad.
"I just think it's quite ludicrous that you're saying that any engineer who didn't work at Sun pre-2010 never invented anything."
You didn't get it quite right: I'm saying that professional engineers worked at Sun Microsystems, hp and SGI. That's statement #1. Statement #2 is that people working on GNU/Linux are amateurs who didn't invent anything. Statement #3 is that because they are amateurs who are incapable of learning, they just keep hacking shit together haphazardly and will never be engineers like the people from Sun, hp and SGI. Especially SGI: SGI had the best engineers. Those people were way ahead of their time in every technical aspect imaginable.
"I refuse to believe that you honestly believe that, purely based on how ludicrous of a concept it is."
Believe it. GNU/Linux people are amateurs. They don't have it in them. They want to be thought of as engineers but they aren't. They just aren't capable of it. That's why GNU and GNU/Linux are garbage and will never be anything more than a pile of haphazard, shoddily slapped together hacks. That's the world we live in now, where IT is shit thanks to their shitty, shoddy work.
I've seen comments from you about how GNU/Linux is awful in every respect ever since you created your account ~3 years ago. I really can't imagine being so stuck in a mindset that you feel the need to spend so much of your time being angry about such a large community of people. Honestly, I just don't get it and I earnestly hope that this is just an online persona you have.
"I really can't imagine being so stuck in a mindset that you feel the need to spend so much of your time being angry about such a large community of people."
Computers were my life, my passion, my calling. They destroyed all that passion in me with their GNU and their GNU/Linux. I will not forget. I will not forgive. They made my professional life a living hell. I hate working in IT because of those people and their mentality.
"Honestly, I just don't get it and I earnestly hope that this is just an online persona you have."
You must be the change in the world you wish to see.
If I am not mistaken, Cantrill stole the line from this skit.
"The good old days." I actually went to a Star Trek convention as a kid back in the late 70's/early 80's, don't remember the year.
It was a different time. "Cool" meant something different back then. People really could say "Get a life!" and it was not an insult. It was truth. There was more to life back then than computers.
(There still is, but today this simple truth is being lost on many more people.)
Sun Microsystems stubbornly refused to lower prices to undercut the generic shitty personal computer used as a server and that cost them the company. hp and IBM made the same mistake at expense of PA-RISC / HP-UX and POWER / AIX respectively. Oracle corporation is in the process of repeating that mistake. They all thought they could get away with outrageous profit margins and pricing. It's like once they started shipping servers to enterprises they all lost compass. Meanwhile, the whole market commoditized. They all wouldn't accept that. Now they've been reduced to rubble. hp "Enterprise" is missing in action somewhere peddling a server here and there, IBM started whoring itself around as a "re-invented" service company (the next "Computer Associates"?) and other than Exadata I don't know of a single business willingly buying Oracle servers. Yeah, it's very hard to predict how that will end. "Very hard".
At any rate, all of that has nothing to do with Bryan. Zero.
It makes it so it's hard to make money off container orchestration, but you can still make money off what Google is good at, or at least thinks they are good at. Selling compute, proprietary databases, and stuff like TPUs.
My point is that yes, Google is advancing the whole industry by releasing Kubernetes as opensource, but it's also a strategy to sell more of GCP and deny some revenue to competitors.
It remains to be seen if it works. GCP is still a distant 3rd in the cloud race and has a long way to go to improve their enterprise support. (Their usability is much nicer though...)
I think it's not really the end of the technology, but it is an end to a generally available public commercial offering of it. It sounds like they're still offering services, but only for "single-tenant", which after some careful reading (because I wasn't sure how to interpret that initially), I think means that they'll provide their software and managemest system for private install for large orgs in their private deployments, maybe along with a support/management contract?
It's not a dead end since any company, public or private can use the code and the product (I for instance build my own modified SmartOS from source) to power their infrastructure.
Technically there is nothing to prevent anyone from taking the product and the code and starting the next Amazon. I know of at least one internet service provider in the UK who is leveraging SmartOS to provide commercial, for-profit service. There are probably more such businesses, but this is something which squarely falls under "I'm keeping my mouth shut and making money" category.
2. ability to run virtual Linux servers are bare metal speeds (and their applications).
3. simple and easy container management of 1. and 2. with imgadm and vmadm.
4. OpenZFS - easy capacity growth, checksummed data, silent data corruption detection and self healing.
5. lightning fast, faster than GNU/Linux performance on the same hardware.
6. same application portfolio as GNU/Linux (14,000 applications available with a simple pkgin command invocation).
7. ability to create virtual routers and switches out of thin air with dladm, join and partition physical links in terms of percentages, simplifying network topology in virtualized environments.
8. OS paranoid about protecting and self-healing itself and the data it hosts.
9. ability to pick between KVM (slower) or bhyve for paravirtualization of non-Linux OS's like Microsoft Windows, OpenBSD, FreeBSD et cetera (although this process is poorly documented).
10. NFS V3, V4, fiberchannel and iSCSI which work correctly.
There is one major downside (I don't consider it as such, but I know others do): one must pick the hardware compatible with SmartOS from the Joyent's hardware compatibility list, not the other way around: https://eng.joyent.com/manufacturing/bom.html.
But when it does fire up over the network and once one groks it, holy shit, it's like being taken into the future with a time machine.
when eveyone was hooked on VM's and other nonsense they had 'branded zones' running.
The learning curve was high, but worth it once you had it running.
Sad to see them leave. wonder what i will do with my SmartOS based media server now?
> The technical abilities of the Joyent folks were well beyond just about anybody else for a long time.
None of that was ever true. Whether they had container-based or VM-based tech was a tiny unimportant detail for a cloud company. Although over engineering in this area certainly made it impossible for them to compete with more traditional hosting companies. And to be able to compete with Amazon they needed something they didn't have - distributed systems background, not Sun background, which was anything but.
They were a pioneer in distributed systems computing. Not many computer companies can claim that. Even fewer companies can claim engineers which hold patents which now power the InterNet, like Radia Perlman does for the spanning tree routing protocol. If that's not distributed, I don't know what is.
Yes DEC did have clusters, and I remember booting my Alpha workstations from a Sun SPARCStation 20 over DECNet as an AutoClient and I know DEC had hardware clustering and VMS working together as a distributed system, but since I came at the end of that era and only worked with Ultrix and DECUnix, I don't know any more about it which is why I couldn't go into detail. I only go into detail when I know about something and usually I've done that something myself.
Prime had remote procedure calls. Their MIDAS product (index sequential files, basically key-value store with multiple indexes) allowed multiple nodes to access the same file by allocating a "slave" on the node hosting the file, forwarding all calls over the network, and returning call results. OS file access worked the same way, using the same slave for all remote procedure calls. Source code designed for local file access didn't require any changes to be used for remote files, just like NFS.
I'd say Sun had a pretty good example to go by for NFS. In fact, if you want to see this in action, I wrote a Prime minicomputer emulator and have remote disks available between 5 different revs of their operating system, spanning a 10-year period.
Telnet em.prirun.com 8001 and enjoy!
So, in that spirit: Sun emphatically did not "ignore the internet" (?!) -- and was in fact a pioneer of distributed systems in many ways. Instead of rattling off the many important, early distributed systems that Sun built -- or the many pioneers of distributed systems that Sun employed -- I will vector you to a single Sun-authored paper circa 1994, "A Note on Distributed Computing"[1] (and to Chris Meiklejohn's discussion of it at Papers We Love[2]). The paper itself drips with wisdom that still feels current and important a quarter of a century later -- and it very much reflects the zeitgeist of the time at Sun. Please read the paper -- and more generally, take the time to educate yourself as to the history of our domain and the many people and companies who have made seminal contributions to it; the generations whose shoulders your work stands upon will thank you for it!
[1] https://github.com/papers-we-love/papers-we-love/blob/master...
This is the company that had "the dot in .com" as its marketing slogan.
https://www.youtube.com/watch?v=njnNVV5QNaA
https://slashdot.org/story/00/04/20/1542217/sun-no-longer-th...
Look at that 1994 paper, none of the things in the paper show any background to build a distributed key-value storage or really anything for the cloud. I'm not sure why people are trying to claim otherwise. Even ZFS from a decade later reflects both a complete lack of experience in distributed systems and ignorance towards the internet.
Oh, and SUN was the Stanford University Network.
Edit: I mean, this is the company that gave us the 8 Fallacies of distributed computing, as they were 20yrs ago
They didn't. Not that any of it is important anyway. All 80s and early 90s networking tech was ridiculously broken, which should be obvious, since distributed systems field was just born at that time (first papers in like 1978). Things started to get useful only by the end of nineties. But by that time Sun's tech was nowhere near anything internet companies were doing. So later when things got to infrastructure Sun's engineering was really really behind, they had to start from scratch on anything that internet companies were learning and doing for many years. And this was very different time consuming knowledge, not something where someone could use their OS level expertise. Which also makes it very hard to compete with companies that are much farther ahead, especially with this silly "our tech is the greatest" attitude making it hard to even realize how far behind you actually were.
Filesystems (or datasets) are carved out of a pool, not out of physical disks directly. There are vdevs on top of physical disks.
That's like people who are stuck back in the '80's because of ext3 arguing that ZFS is crap because it doesn't come with fsck or "some sort of repair tool" -- they just can't wrap their head around the ZFS concepts.
The ZFS team was well aware of this and that's why they made the trade-offs they did. They picked a subset of problems and they solved them. Even in hindsight, to me at least, it's obvious why.
But to suggest that Sun Microsystems doesn't exist because the company's engineers picked only a subset of problems they knew they could solve is... either terribly ignorant or wilfully malicious, or both, since malice often stems from ignorance. Sun Microsystems lost because they were too expensive, more expensive than garbage intel personal computer servers and because Solaris wasn't open source code in 1993. It's that simple.
I worked at a company circa 2005 where we had a platform as a service built on top of Solaris 10 and zones, on-premise infrastructure which you now call cloud. Fully configured and serving Sun clusters would come into existence by merely DHCP booting from the OpenBoot PROM (Sun had to give us patches for the OBP for DHCP booting). I happen to know this, because I was one of the senior system engineers who helped architect it and then built it. We had it powering tens of thousands of servers. We were one of the first users of IP Multipathing in Solaris ("IPMP"), a technology Sun engineers invented to provide redundancy for network connectivity for the clusters. Amazon wasn't even a smear yet. That was in 2005. Which year is it now? How many years ago was that?
lol. "The network is the computer."
When they first started they used FreeBSD jails IIRC.
The main selling point is/was being closer to the hardware and thus getting better hardware utilization. Virtualizing at the machine level was a lot slower (especially before Intel VT-x and AMD-V came along) and so one wouldn't lose (say) 30% right off the top:
Nope. Solaris Zones.
[1]: https://www.google.co.uk/amp/s/davidpaulyoung.com/2016/06/17...
Joyent/SmartOS was the one that added KVM support and a bunch of other features to the kernel, but I think it was merged into IllumOS. It's quite limited and has a clunky interface. When I tried to contact anyone there to add UEFI boot support there wasn't even so much of a: "we don't care".
The problem with SmartOS, Illumos and the rest of the Solaris world is that while for Linux you can find heaps of books, forums, stackoverflow entries there are 3 books for OpenSolaris. SmartOS has seemingly great design, but it's absolutely nontransparent on how to use it. The documentation is subpar, lots of the wiki reads more like an experiment than an actual documentation.
If you hit a bug, be prepared to spend hours on github issues to see if someone found a solution to it(might as well skip it because it's unlikely).
If you use Triton and the API for it all is good, but if you want to use a SmartOS cluster directly, everything thing will be fine until you make a mistake or try to do something non standard and then good luck to you.
There's a great/fun talk by Bryan cantrill about how management killed Solaris. But if you ask me it's only one half of the story. The other half is elitism.
If you want people to engage in your open source project it needs to be decently reasonable to work with and you should be able to get some sort of feedback, even if that feedback is "lol, what you're doing is completely wrong why don't you RTFM here".
But the attitude I saw was more like: "Our stuff is greater than everything else so why should we bother with you".
My apologies again for your experience, and I don't think it's indicative of the community. I would encourage anyone who wants to get involved on the mailing lists or hop into #illumos and/or #smartos on freenode![2]
When first using OpenBSD this was jarring, because I just wasn't conditioned to expect that system level features would have useful man pages, as that was almost never the case in Linux.
As a simple example of this, here[1] is the OpenBSD man page for ifconfig. Note that it contains liberal references to other man pages of similar quality and completeness where a protocol or functionality might be better explained. Once I learned this, very little outside help was required to setup OpenBSD firewalls stateful failover of firewall connection state as well as VPN tunnels with automatic IP failover with CARP. All from a handful of man pages.
So, that said, does OpenSolaris have good man pages? If it does, it might be that RTFM is the correct initial response to most questions, as it would be the up to date and definitive information about the what and how of things. A simple comparison of ifconfig[1] makes it look somewhat good, but that doesn't mean it's kept up to date and accurate. Can anyone comment if this is the case?
But yeah the structure of the OpenBSD man pages looks ok. Where you put proper structured documentation doesn't really matter IMHO.
It's conceptually different. You can easily use the OpenBSD manpages, because they map nicely to every other Unix system concept.
Whether it accurately explains the capabilities of etherstub I don't know, but it seems to do a good job explaining what it is and is used for. The basic usage is detailed within one of the examples. I'm not sure what you're comparing it to that you think would be a superior way to document it though. I honestly can't imagine it being better in Linux in any way. Windows, it would probably be explained better, but you might have to pay a few hundred dollars for the documentation or training, whether it be from Microsoft or a third party.
> It's conceptually different. You can easily use the OpenBSD man pages, because they map nicely to every other Unix system concept.
Not any more than SmartOS, I'm sure. IIRC, CARP is an OpenBSD designed protocol to compete with Cisco's proprietary VRRP. pfsync is their tool and protocol to sync PF (their own custom firewall implementation) states across a network link. sasync is their own protocol and implementation to sync IPSEC security associations across the network. These were eventually adopted and ported to FreeBSD, but they are far from "every other Unix system concept".
I view OpenBSD and IllumOS/SmartOS to be even more similar in approach now than I did prior to this discussion.
Here is an old, historical thread that gives some context:
https://news.ycombinator.com/item?id=4391669
I have a hazy recollection of their being part of the "free thumper" startup program that Sun Microsystems offered, briefly, and then later, blowing up one of those thumpers and losing a whole clouds worth of data... here it is:
https://adtmag.com/articles/2008/01/21/strongspaces-10day-cr...
Later, they were briefly (in)famous for voiding everyones lifetime shared hosting agreements:
https://en.wikipedia.org/wiki/TextDrive
"Customer backlash to the announcement turned out to be fierce."
But it got them cash and publicity, and didn’t kill them when they cancelled it, so, on balance, it made their business possible
The death knell really being the abandonment of illumos-gate as the authoritative ZFS upstream. If old-school diehards like Matt Ahrens are throwing up their hands and saying that the momentum in ZFS is in ZoL, the battle is over IMO.
node.js was always a diversion IMO. I think it was a play to maintain relevance, something Joyent has always acknowledged a need to do, but I don't think they've ever had the manpower or the buy-in to go full throttle.
I don't know if this is the end of Joyent as such or not, but IMO it's finally time for Cantrill et al to turn in. They've undoubtedly fought the good fight, but illumos is a farce at this point. Better to accept that their old-school big Unix development model is a thing of the past and put those efforts to something more practically useful, like FreeBSD. At this point, they're torturing the legacy of Solaris more than maintaining it.
[1] https://vimeo.com/230142234 (slides: https://www.slideshare.net/bcantrill/platform-as-reflection-...)
Of course I'm not suggesting that you forsake your values. I'm sure anyone who has followed you for any amount of time knows that will never happen. There is no stronger advocate for the principles upon which your work is based, and your fervor and commitment is extremely admirable.
My suggestion is merely that it's perhaps time to acknowledge that the market is not going to adopt illumos in a major way. An outflow of that is considering the possibility that the considerable engineering resources at your command may be better deployed in attempting to instill and carry forth the same crucial, badly-needed principles and values into projects that have more realistic prospects, to the extent possible, rather than remaining 1000% committed to a lost cause, where the principles and values will flare out in obscurity. "If a tree falls in the forest..."
illumos is an immense contribution to our collective engineering heritage and I entertain no illusion that it should be totally iced. Personally I would consider it a glorious resurgence to see it more widely adopted, and to see it become the basis of a new generation of systems. SmartOS is simply a joy compared to the train wreck that passes for systems management in today's mainstream, and I sincerely hope that the wheels turn to make it a more viable option. The ongoing open-source contributions of the faithful volunteer keepers of the flame will be invaluable in leaving that possibility open.
While there is no doubt that illumos's open-source heritage ensures it will live on, we must be realistic and admit that for all practical intents and purposes, the sun has set on the project's general commercial viability (at least for the time being -- the thing about sunsets is, they're followed by sunrises).
The question I pose is whether your advocacy may be more effective in bringing the values and contributions of yourself and your team in force to more active venues, than by continuing to toil in the last fleeting fragments of the twilight of Solaris's commercial viability.
I personally know many young engineers who admire your talks but have no exposure to the actual process of working with you directly. What would change if they got that? Would there be a Cantrill Youth? What's the likelihood of passing on that flame within illumos v. expanding your horizons to other groups?
Sometimes a reference implementation is intended to be just that, and nothing more—a reference, for other implementors of other systems to crib ideas from, especially as pertains to how the smaller ideas cohere architecturally in practice (which is hard to communicate by just writing papers about the individual ideas, or building them as standalone libraries, or building them into other systems where they must interoperate with components that don’t cohere into the desired architecture.)
Or, to put that another way: sometimes the best textbook is the codebase of a good implementation. And some people are more interested in writing those textbooks, than in personally getting their ideas into production systems. Especially because it’s far easier to refine the ideas in the context of such a coherent presentation of them.
Someone else, who reads such a “textbook” codebase, can put the ideas to work. But the ideas themselves will be better-developed for having had time to stew in the same pot as other novel ideas. This serves the people who put the ideas into practice as well.
I'm sure they're fine with it being Their Own OS that's only actually used for the Samsung Internal Cloud Thingy, while the source is public for anyone to take so it won't disappear from history.
You're asking Rembrandt to stop painting realistic paintings because Picasso nowadays generally passes for art. What you don't consider is that there is nothing stopping Rembrandt from continuing to paint his realistic paintings for his own sake and pleasure.
This is by far the best explanation I have heard. Suddenly Joyent makes sense to me after all these years. They have been continuing to do what Sun did.
In a very similar way to how Sun followed the fatal strategy of simultaneously trying to make money selling proprietary Unix hardware, while also making their software open source and porting it to more popular architectures, while simultaneously developing Java as a portable programming language intending to erase all differentiation in underlying architecture/OS differences …
Yes: Joyent is (was) a hosting company trying to compete with AWS, claiming technical superiority in the software stack. Yes they also make an OpenStack competitor. And had their own programming language runtime, node.js, which seemed to be an odd implementation choice for their classic old-school unix tech. All of the above.
Sun would have done better to have made SPARC hardware more accessible than to have open sourced Solaris. Hindsight is 20/20, but now that we've reached the limits of single-threaded performance Sun's emphasis on multi-threading and specialized ISA extensions would have made both Solaris and SPARC competitive today.
Getting there would have required cannibalizing their enterprise income, though, and that's difficult if not impossible for any company to do. They made the gamble on Solaris because it was less risky--major enterprises were always going to stick to Sun's Solaris--but low-end SPARC hardware absolutely would have hurt their bottom line.
That this came from a company essentially selling mini computers.
In fact, we are trying to do multi-tenant all over again. To my mind, this cycle of containerization is still trying to fulfill all the promises we were given in the early 90's about memory protected, pre-emptive multitasking OSes that never came true. Which itself wasn't the first time (it's all copying mainframe ideas, badly). Given all of the problems coming out about speculative execution and data breaches, I'm already curious what the next set of promises will look like.
I man not really, Joyent is a hosting company (cloud & virtualisation). That's built upon an Illumos distribution (SmartOS) but SmartOS is not their business.
SmartOS may've been indirectly implicated in revenue, but let's be real, illumos and Joyent have always been tied at the hip. Joyent was never going to become a Linux-centric alternative public cloud like AWS, GCloud, etc.
If there had been less ideological fervor and with Samsung's backing, maybe they could've offered more serious competition to the incumbents, but the Solaris style oozes out of Joyent's every pore. It's fundamental to their way of thinking. That purity and commitment is certainly admirable, but it's simply not commercially viable anymore.
Joyent is proof that large scale cloud can be done with SmartOS.
Before AWS launched, Joyent (in my opinion) had one of the best hosting services. We ran everything on them, and especially liked Zones as a feature.
Separately, no one has mentioned that Joyent also had a full web application framework at one time (mail, calendar, etc.) that competed with the nascent Google GMail offerings. Joyent EOL'd back around 2010 if I recall correctly, but technically it was an impressive accomplishment.
If you look into their docs, it's a layer above components such as Docker, VMs, cloud hardware, even Openstack.
The main selling point is/was being closer to the hardware and thus getting better utilization. Virtualizing at the machine level was a lot slower (especially before Intel VT-x and AMD-V came along) and so one wouldn't lose (say) 30% right off the top:
The one escape out of FreeBSD jails that I'm aware of (CVE-2005-2218) involved not a bug in the jails code, but by burrowing out via devfs.
So it seems that OS-level virtualization/security has a better track record.
The story of the BSDs and illumos is really a tragedy about the complete and total disconnect between technical quality+value and technology adoption. People utilize technology products as fashion statements first and foremost, and the Linux fashion was easily the most desirable -- indeed, aside from PC v. Mac, it's the only one generally known by average technophiles.
As a technical entrepreneur, I've discovered the sad reality that it's really only the most superficial 10% of any functionality that drives virtually all adoption. Everything else is marketing, even in supposedly heavily-technical endeavors like running an open-source kernel. The more you invest in engineering beyond that initial superficial component, the less you have to invest in brand promotion.
One can choose to gratify their own sense of righteousness and ignore this, and that is certainly a valid option to take, but then they shouldn't be surprised to see their lunch eaten by someone who took the other side of the bargain.
Linux have 18 millions lines of code in drivers. It can run on almost any chip/architecture. Distributions are fully featured and most of the things you can imagine have first class support.
I do not care how beautiful is architecture of BSD kernel if I cannot run my code on it.
It's true that tech has a popularity-breeds-popularity effect, but I don't think it's mainly fashion (at least, for industrial use of tech like PLs used commercially), but network effects around available knowledge and skilled users.
Abstract technical superiority often isn't a commercial win without reliable availability of people able to use it effectively at costs that make it efficient. There's situations where buying the limited knowledge pool, or paying to grow on house skill, is worthwhile, but often the “worse” solution that is well-known is better practically
That was true at one point in time (i.e., no known breakouts). I don't know if that is still true, and if it is not, when that breakout CVE was published.
There is still a great deal of administrative/technical value in using anything-but-Docker if you can tolerate the social stigma, but who can? The CxOs need that Docker/Kubernetes feather in their cap so they can look cool at conferences.
Suggesting SmartOS/illumos gets you sideways looks at the very bare minimum. When I stated that I was thinking about using it here over a year ago, my inbox was immediately flooded with accusations that I was part of a Joyent-bankrolled cabal, because clearly no sane ordinary person would consider such a dead platform. In the interim, OmniTI has gone belly-up and OpenZFS has deemed ZFS on Linux the rightful heir to the ZFS throne, and now the Joyent Public Cloud is shuttering. If the impression that you had to be part of a conspiracy to use SmartOS for a greenfield project was reasonable then, it's only more reasonable now.
The sun may yet rise again on illumos some day, but it's time to accept that it's well into the night now.
There are also some really subtle Linux compatibility problems. For instance: IIRC a SmartOS malloc will not fail. Ever. But it might not return either. There's a really complicated mark/sweep type virtual memory thing and the documentation is good, for the nasty node stuff, and kinda terrible otherwise.
Finally, and with no lack of irony, you just can't run SmartOS on the cloud. For a start it needs to do its first boot off a CD and subsequent boots off a USB drive-type-arrangement. The hard part is that most cloud providers run a butchered Linux kernel and not a fully generalised VM - and on vultr at least, while they run a proper VM, for some reason the processors don't "come up".
I've not looked at all this stuff for at least a year so it may have changed. Either which way I believe the majority of the community is now on OmniOS - which I always got on better with anyway.
Actually I'd say the Triton install is not only trivial, but should be a model for other similar systems. During my testing with Intel NUCs, I had a usable system deploying containers in under 20 minutes. Most managed Kubernetes systems cannot even provision in that time frame.
> you just can't run SmartOS on the cloud.
The installer may lead you to the USB and install path, but it is not necessary. I built Vagrant boxes that had SmartOS on a regular disk, and I believe that more recently SmartOS has been running well in GCP. There is no reason it could not run in AWS or Azure, either.
when was the last time you used k8s?! I mean as of now there is even the expirmental k3os which boots a fully fledged k8s in seconds. you can even built a cluster with it. sadly HA backplane is still not available since k3s is still early in its development.
How many systems did you need to install Triton?
How much memory did(do) the individual systems running Triton have?
Which documentation did you follow to install Triton?
No it does not. I boot my SmartOS instances with PXE, DHCP and TFTP from the network. Served by a Solaris 10 intel server, no less. No CD's or USB images anywhere in sight.
In fact, I came up with a trifecta scheme where three SmartOS servers will all run TFTP and DHCP servers, thus being able to boot each other and all the other SmartOS servers over the network. In the event where all three would lose power simultaneously, THEN I reach for a USB stick with the emergency boot image. And I'll have two of those. But barring such catastrophic loss of power simultaneously, there won't be any CD's or USB sticks needed.
The network is the computer.
When you compared the utility of JCP "tiny" instance (1 CPU worth of shares, 256MB of RAM) to a t2.micro or similar instance in AWS, the difference was astounding. Part of this is the CPU credit mechanism on T2 instances, but I'd wager that a good part of it was being in OS-level virtualisation.
Public cloud != Private cloud
It looks like at the very least they still offer private "cloud" hosting.
The control plane work likely won’t be made available (it was Linux specific, anyway). :-/
As for both the FreeBSD work and control plane work, it was good times, good memories, and a great team while it lasted.
I mean, if you want to publicly comment on your time at Joyent, please! I'm still pretty angry about it, and I left the better part of a year ago.
Or uh, so I heard.
I'd love to see and read about some of the things you ran into, and how you accomplished some of those high speed transfers.
https://github.com/joyent/freebsd/tree/projects/VPC
https://www.slideshare.net/mobile/SeanChittenden/freebsd-vpc...
Last I heard Samsung had acquired them.
This is false, as someone who was actually there can attest. Node.js couldn't even run on Solaris when Joyent finally got involved, and Node.js technically and in terms of growth had already taken off thanks to Ryan Dahl.
Source: I'm the guy that ported Node.js to Solaris, and I have nothing to do with Joyent, then or now.
Random kind inspiring people on the interwebs (most of whom I never met in person!). Incredible.
edit: also had one of their early "VC" deals for lifetime hosting. Very interesting feeling to have something you don't have to pay for anymore on a monthly basis, inspires to make the most of it. Got a ton of mileage out of it and wasn't upset about it being sunset, by that time hosting was quite cheap already.
edit2: When they started to adopt OpenSolaris I took a closer look and eventually ended up with it as my main desktop system back then & learned about ZFS and dtrace. Magical, fascinating times.
Please tell me this is a play on the word Voyeurs
RIP Joyent public cloud (and the last remnant of TextDrive's legacy).
The Joyent email + calendar + notes web app was also quite impressive at the time. So many memories.
ExpanDrive's "Cloud drive client software" is also well worth looking into.
I have stayed away from them forever since, and warned everybody I could about their terrible business practices.
I also have experience running SmartOS on Leaseweb (they purchased Uniquity Hosting where I setup SmartOS instances on numerous servers in numerous datacentres.
Decisions like this are long in the making, this is just the public announcement for the rest of us.
The statement implies that Samsung's hunger for resources had grown so large that there is no more room for the public hosting side of the business, but considering how high performance SmartOS zones are (bare metal performance), I find that incredulous. How is that even possible?
But do let us, for the purpose of this mental exercise, presume that Samsung has indeed such great hunger for resources that public side of the business has to go: surely a chaebol as large and as financially powerful as Samsung has the funding to keep both its internal infrastructure hunger and the public side of the business sated, and easily at that?
Another option could be that the public side of the business was losing money, but considering just how trivially easy and fast SmartOS zones are to provision, and how little financial resources it costs to run them (it costs me literally peanuts and I run my entire datacenter at my own premise and expense), it is hard for me to fathom what could have possibly went wrong there: this technology is so advanced and ahead of its time that even a single customer running on it would be profitable, a bonus, and Joyent has more than one customer...
Running a public cloud requires a massive amount of people, unlike what you seem to think. It takes one or two orders of magnitude more work to do anything public rather than internal. Bet there is no resource and no budget to keep running their public cloud anymore. It's a small company, don't have unlimited resources.
Second, there are no revenues, no future and no clients on their public cloud. Solaris for all intent and purpose is dead (it pains me to say that but that's a fact). What companies run Solaris in this decade? None. Plus hosting competition is at an all time high, DigitalOcean and other on the cheap side, AWS and Google on the enterprise side.
Samsung is a massive company and all massive companies have massive internal systems and need for internal infrastructure. I bet there is plenty of work that needs to be done internally that is much more valuable than public cloud. There they go.
If you've worked in any F50, you would know that the sheer amount of servers and desktops is larger than an AWS datacenter. They are all cloud hosting, internal or external, whether that's their main business or not.
I don’t imagine your average Fortune 50 has millions of servers.
Consider that a Fortune 50 has hundreds of thousands of employees. With one desktop per employee that's already way more hardware to manage than an AWS datacenter.
Now think of how many servers, mobile devices, network gear, access points there are to run the business. 60k servers is not uncommon. A million devices is nothing special.
The order of complexity for public cloud/multi-tenant is much higher than for private/single-tenant, especially given the recent onslaught of CPU vulnerabilities.
Still a big loss. I imagine it must feel like a defeat, especially to Bryan, given the corporate culture at Joyent. Open sourcing triton is great, but it still needs hosting for people to use it. Amazon/Google/Microsoft could certainly cut costs if they leveraged bare metal so who knows.
It's like coming to a race track with a car and being disqualified because one's car is too light and powerful.
Many of us Solaris Engineering refugees have gone on to do other things. And trust me, for me it was hard and still is because I don't yet get to use the new Linux eBPF stuff daily and w/o privilege for the equivalent of DTrace USDT, and I also don't get to use ZoL, and because systemd is a very poor not-SMF, and because there is no FMA for Linux, and because epoll makes me sad, and so on and on. But I don't dwell on all the mistakes made by Sun's executives from 2000-2010 (and they made so many terrible mistakes), not often. Since that seems popular in this thread, I'll tell you what I think those mistakes were (some of which Oracle is still making):
- clinging to dying business (SPARC, J2ME) to the
point that other, better business was forgone to
avoid cannibalizing / competing with the dying
business
- killing Sun PS (2002)
- insisting on unacceptable-to-Google licensing
terms for Solaris (2002)
- suspending Solaris x86 development (2002)
- selling Sun to Oracle
Oddly, the torch of OS innovation has passed to MSFT. Not so odd, really, when you realize that MSFT learned the lessons of stack ranking (don't do it). And MSFT already had learned the lesson that to survive disruptions you must pivot. Bryan clearly learned those lessons early, so I think he'll be fine.
- I'm furious about all the attention RISC V undeservedly gets while OpenSPARC is vastly superior in terms of mnemonics and virtual register windows and a free instruction every branch not taken and is GPL licensed;
- not devoting enough attention to i86pc Solaris was a major mistake;
- not selling cheap fully Solaris integrated desktops was the death knell: people knew Solaris from their student days, and universities had it because Sun made and sold workstations; once Sun decided to stop that, talent acquisition was stopped: desktop is needed for OS adoption;
- UltraSPARC T3 and T5 are blazingly fast, I know because I ran them, but now it seems too late;
- it wouldn't be too late if the RISC V nonsense pandering stopped and some OpenSPARC designed servers and workstations and even tinkertoys showed up by enthusiasts, like they did with Raspberry Pi;
- I'm glad Sun bought MySQL because Oracle subsequently destroyed it; MySQL is nowhere to be found now, and that is good so, because it was a shitty database which silently corrupts data; it deserved never to be made in the first place; that is one good deed in the world of damage which Oracle caused;
- killing OpenSolaris destroyed all that good will; had the project continued, it would have directly competed with GNU/Linux because we saw a constant monthly influx of GNU/Linux refugees on opensolaris.org forum, people who actually needed to get things done with their hardware;
- biggest mistake was that UltraSPARC based servers weren't cheaper than an average intel personal computer tin bucket server. That really killed them. And they would argue and refer me, a customer, to their internal "pricing committee" instead of listening.
What happens when a business doesn't listen to its customers?
"Bryan clearly learned those lessons early, so I think he'll be fine."
Indeed, and I wish him well and much success. He truly deserves by his deeds let alone his thoughts to continue being successful.
The death of MySQL is certainly a good thing, but it's not really dead. People use it still.
SPARC was a dead end.
I too wish Bryan well. And the rest of the Sun diaspora. I miss them!
On the other hand, the single-tenant customer is plausible. I know Samsung invested in a lot of technology in the past five years. The Viv team is still around (acquired around the same time as Joyent), and there is quite a bit of IoT Samsung is building into their appliances.
The tenant is now too large, please move out.
i also miss their xmpp bot for some weird reason.
But I mean, in honesty it's long overdue. Technical superiority loses out again.
Bryan, are you around?
Then there is COMSTAR, a reference fiberchannel and iSCSI implementation. illumos inherited the reference NFS V3 and V4 implementations from Solaris. The runtime linker is so advanced in respect to versioned interfaces that it looks like it's magical. The list could go on and on and on... Linux is nowhere near any of that. Linux barely works somehow but when it comes to running serious infrastructure and data integrity, that's not enough. Just because most people think bash completion and GNU commands are what makes GNU/Linux superior doesn't mean that it actually is: it's very crude and unreliable underneath, especially since you have armies of people constantly hacking on it and breaking kernel interfaces, especially in networking. Try running a firewall / VPN company with it to see what a nightmare it is. And that's just one instance.
And then there is cloud: nothing that GNU/Linux has comes even remotely close to how simple, secure and fast Solaris zones in SmartOS are. Literally nothing: I can pull an image with imgadm and instantiate a bare metal performance UNIX®️ or GNU/Linux virtual server in under 25 seconds with vmadm. No Kubernetes or Docker or OpenVZ or Terraform can do that, and those are all afterthought bolt-ons, while imgadm and vmadm are core of SmartOS and its reason of existence.
It feels that you're building something to avoid HW issue but they will happen no mater what so it's better to invest in HA with a mix of standard servers + SW.
And solaris zones... all the modern tooling are built around Linux / Docker / Kubernetes ect ... so why should I use something that no one uses and lagged behind in term of tooling? Solaris zones are doing 1% of what a modern cloud is capable ( like Kubernetes )
Linux is the present and the future if you think with a small company you can change that you are very wrong, by the time SmartOS get something you have Intel, IBM, RH, Google, AWS that all work on Linux that have thousands of people working on many more things, not to mention HW implementation, and that's not even talking about HW integration, pricing, security, support ect ...
"if you think with a small company you can change that you are very wrong"
You must be the change you wish to see in the world.
Mahatma GandhiNick at mnx io
Fun times!
Edit: It's also how Twitter used Virtualmin in their early days, as they were hosted at Joyent. (I didn't know this until Evan Williams told me at a YC event. That was pretty cool.)
It was a pleasure dealing with the Virtualmin team on issues we had on Virtualmin on Joyent and TextDrive's shared hosting services.
Replying to the "Wins over GCE/AWS above that I can't reply to":
They had various:
* The ability to run workloads at near bare metal speeds (their using Solaris Containers).
* Manta (ability to perform compute operations on their storage service without having to download files and reupload files).
* Their work on getting a better user experience in zones with pkgsrc (thanks to all the hard work of Filip and Jonathan).
* Better performance and having the ability to spend less money on resources and reducing the instance count for folks moving off AWS to Joyent. I was involved in a number of these migrations.
Linode (founded 16 years ago), and Digital Ocean (founded 8 years ago), are still alive and kicking, not to mention more traditional hosting companies like HostGator.
The amount of people who are unsatisfied with DO's VM product and would be satisfied with a k8s product is tiny. The amount of people who would like somewhere to run functions and not care at all about infrastructure is growing rapidly.
If they're not skating to where the puck is moving they won't be around much longer.