Announcing Joyent Cloud | Hacker News Reader

Announcing Joyent Cloud

joyentcloud.com

89 pointstimf44 comments

For the EC2/CloudX,Y,Z users here. Have you compared your cost with the cost of running your own cloud on top of dedicated servers from a traditional provider?

I know that with EC2 you can "follow" your requirements and start/stop instances, but even by considering a perfect predicting tool which would cut 50% of my instance requirements with EC2, I am still cheaper having 2 times more hardware than needed with my traditional provider.

I pay $150/month for the equivalent of a high memory extra large instance. I can ask to have them in different data centers all linked together within a private lan (VLAN). I look again and again, I never reach the point where it would be cheaper to run in the "public" cloud. I run my own cloud with KVM/Ganeti.

I think the cloud is nice if you really have a system which can follow your demand over the day/week but you need to be quite big for it to start to save you money. You can go a very long way on a single box...

I've heard about people who keep enough dedicated servers to handle their medium load, then dynamically adjust with EC2 (and add more dedicated servers to follow the growth).

If you want just 1 or 2 mid-sized servers running 24x7 with traditional software on them like an RDMS, AWS is almost entirely the wrong choice.

A single AWS instance is not reliable, and isn't designed to be particularly reliable.

The real power of AWS isn't the ability to stop and start cheap instances quickly, it comes from services like S3 and Elastic Load Balancing, and SimpleDB.

If you build your systems on top of these extremely reliable services, you can achieve uptime much greater than you can achieve with 1 or 2 standalone servers (whether they've got the word "cloud" in their name or not), while not spending the massive amounts on high availability or fault tolerance that you previously had to pay.

In my case it is more like 5x24GB RAM servers supporting 20 to 50 VMs depending on the load coupled with a very reliable managed NAS for backup. I can start/stop some instances using my provider's "cloud" if needed, but found that in practice, I always have room on my own hardware.

For the reliability, over 10 years of hosting, I cross my fingers, I have not been hit by any major incident. So, my services have something like a 99.96+% availability and up to now, no data loss or things like that.

When reading how much thinking NetFlix had to put in to adjust to the variance in the performance of the VMs/networking/etc. in the cloud (Cf. http://techblog.netflix.com/) I am happy as a very small shop to not have to think too much about this. A bit yes, but not so drastically.

This is why I tend to be happy having "a bit more control" and the current tools like Ganeti are making the management part of the VMs as simple (if not simpler) than with EC2.

Anyway, thanks a lot for the feedback, it kind of confirm that I am not totally stupid. :D

"A single AWS instance is not reliable, and isn't designed to be particularly reliable."

Source?

An AWS instance is simply an OS running on a virtualf (xen?) machine. On the largest instances you may in fact be the only virtual machine on the physical hardware.

Why would you feel more comfortable running a single instance outside of AWS? Your safety is just an illusion. Your MTTR is out the window. When you go down, youll have an outage that goes like this: "we'll be back in a day or two when I order new hardware and drive it to the colo. Also, the database will be time warping to the last backuP 24 hours ago."

Running on a single machine is never safe and AWS is cheap enough and comes with enough tools (ELB, RDS, maybe EBS) to allow you to become truly redundant and single fault tolerant.

> "A single AWS instance is not reliable, and isn't designed > to be particularly reliable." > Source?

Reading the AWS EC2 forums for any length of time or launching your own EC2 instances into production and watching them fall over from time to time.

All arbitrary, but I don't think anyone that has deployed on EC2 with more than a handful of servers would ever describe it as an overly stable service.

> Why would you feel more comfortable running a single instance outside of AWS? Your safety is just an illusion.

In theory, absolutely agree. It sounds like you were agreeing with the OP to be honest, but just getting pedantic about single-AWS instance vs single-dedicated instance in another hosting company.

Sure, best practices dictate trusting a single point of failure is not a good idea.

In practice, my dedicated deployments (at RimuHosting if that matters) are infinitely more stable than my EC2 deployments and I think that data point is worth something -- not going to bet the farm on it, but I'm also not going to treat my dedicated servers like I would flaky EC2 VMs.

> Running on a single machine is never safe and AWS is cheap enough and comes with enough tools (ELB, RDS, maybe EBS) to allow you to become truly redundant and single fault tolerant.

You are agreeing with the EwanToo from what I can tell...

AWS seems really stable to me. Hundreds of days of uptime are the norm, if the instances even run that long.

However, there IS a pretty huge gap that exists between managed colocation and EC2. There are dedicated servers that can be ready in an hour or so, cloud-like baremetal "instant" provisioners, and über-VPS providers like Linode and Rackspace Cloud.

> Have you compared your cost with the cost of running your own cloud on top of dedicated servers from a traditional provider?

For me, using EC2 and other Amazon web services isn't just, or even mostly, about cost. It's also about ease of use, flexibility, reliability and scalability, where AWS wins hands down compared to attempting to replicate the same myself using dedicated servers.

AWS are not primarily in the business of renting servers, they're in the business of selling their operational expertise in building and managing reliable, massively scalable distributed systems, and doing that well is very, very hard.

where are you hosting and getting a box that big for so cheap?

OVH. http://www.ovh.co.uk or http://www.ovh.com

It is a crazy team, they are fan of optimization, they build their own datacenters with a PUE below 1.15. The good news, next year they open datacenters in the US/CA.

Interesting, so they are comparing to the OnDemand pricing of AWS, not including that 3-year reserved instances on AWS are somewhere in the ballpark of 48% cheaper, BUT, they have this footnote at the bottom of the pricing page that seems like a really interesting differentiating approach[1]:

    No Charge For Data Transfer Up To 20TB Per Month Per
    Account. 100TB free data transfer for customers over 
    500GB memory

On AWS, 20TB == 2,000GB * $0.10/GB (avg) = $200

and I suppose if you have enough SmartMachines on lease that you are pushing 500GB+ total of memory in your account, they up that to 100TB of transfer.

It looks like overages are a fairly reasonable $0.08/GB[2]

I think we are finally witnessing bandwidth becoming a race-to-0 for big cloud providers and I love it (I know there have been prediction-papers here on HN before about how it will eventually become free or near-free, it is just interesting to see the different ways we can get there). Best of luck to Joyent and it's great to see another strong competitor in this area!

ASIDE: Anyone have a geographical map of the data center locations you can deploy in the Joyent Cloud? One of the big appeals of AWS is having decent global coverage and I am trying to find more information about that form their site with no luck at the moment...

UPDATE: For anyone interested, just got this back from the Joyent team about the location of the data centers:

  We launched today with two datacenters.  WEST is in 
  Emeryville CA and SW is Las Vegas NV.
  
  Inside of my.joyentcloud.com you will have a choice of 
  datacenters when provisioning a server.

[1] http://www.joyentcloud.com/products/pricing-comparison/smart...

[2] http://www.joyentcloud.com/products/smartmachines-2/options/

Performance claims and DTrace support aside, the costs are similar for instance-hours for low to medium-sized instances.

But completely different if you are doing more than 15GB of outgoing bandwidth: "No Charge For Data Transfer Up To 20TB Per Month Per Account"

http://www.joyentcloud.com/products/pricing-comparison/smart...

As their default (and quite interesting) offering is SmartOS, I'd like to ask if someone has some recent experience with (Open)Solaris networking. I still remember the day when it (especially its x86 incarnation) was renowned for having syscalls that were slow as molasses – IIRC, actually a big boon for the then-fledgling Linux.

How does it fare at memory allocation, especially under load? How good is their select successor (i.e. their equivalent to epoll/kqueue)?

As I recently discovered, you cannot run a software firewall of any kind inside a solaris container. Everything else about SmartOS seems quite great to me, but my evaluation term was limited.

Actually that is not the case. In SmartOS you have two options for networking. You can run a zone in "IP exclusive" mode which will give the zone full control over its networking stack. This means you can run a firewall or even change the IP addresses, etc from within the zone. SmartOS makes that apply to VMs as well. There are also anti-spoof mechanisms built into the OS to ensure that you dont get unfriendly neighbours.

Here is a rough overview: http://en.wikipedia.org/wiki/OpenSolaris_Network_Virtualizat...

I would say that the syscalls issue was gone as of Solaris 9 and Solaris 10 was very fast. The VM subsystem in Solaris is superior, and memory allocation using zones is excellent. Basically a Solaris zone is similar to an OpenVZ container.

Under load and under memory pressure, Solaris is as good as or better than Linux.

Despite all this, I am in the process of moving remaining Solaris systems to OpenVZ-based containers due to factors other than technical excellence and more to do with familiarity (clients are just more familiar with Linux than Solaris, though 90% of the shell etc. is the same).

Looks like a good competitor for AWS. I think as a customer it is always good to have some high quality competition amongst the vendors. This will push AWS to try to replicate or better the features being offered by them and also keep them from raising their prices too much.

The list of customers for the joyent cloud looks quite impressive. So this might already have been tested quite thoroughly by these customers and doesn't look like as if it is launching just now.

I think they should introduce a free tier like AWS's micro instance to allow ppl to test drive this new cloud.

http://www.joyentcloud.com/products/pricing/smartmachine-pri... -> 404.

Yeah. I found this: http://www.joyentcloud.com/products/pricing-comparison/smart...

This is a better link (not the comparison to EC2 pricing): http://www.joyentcloud.com/products/pricing-2/smartos-linux/

That's a worse link to me, mostly because they don't include the "Joyent Cloud Advantages" section like in the EC2 comparison. Those are very relevant points, especially 20TB bandwidth per month.

Their regular pricing page and their EC2 comparison page don't seem to match. One example - On the regular pricing page, XL has 32GB of memory, 760GB of disk, and costs $1.12 per hour. On their EC2 comparison page, in two places XL has 16GB of memory, 480GB of disk, and costs $0.64 per hour. In another place on that page it has 8GB of RAM and 240GB of disk and costs $0.64 per hour.

Pricing page is 404. Cached here, though obscuring the lower end of the price range: http://webcache.googleusercontent.com/search?q=cache:Y7ZEmSC...

I was looking for details on SmartOS. What is it and what does it run?

SmartOS is the OpenSolaris fork.

http://www.joyent.com/products/smartos/

https://github.com/joyent/smartos-live

In addition, from what I know, Joyent has hired many of the original contributors to solaris and opensolaris (and now SmartOS).

Does Joyent count inbound traffic as part of its bandwidth calculation or does it offer free like AWS ?

So the prices are pretty much identical to Amazon EC2. Why would anybody choose an unknown player versus somebody very experienced like Amazon?

Joyent may be unknown to you, but they do a lot of interesting things with Solaris/Illumos, tons with Node.js, and have been around for quite some time. They were an early host of Twitter and have been around for 7 years at this point.

Keep in mind that EC2 has only been out of beta for 3 years next month.

There are some notable names running on Joyent's infrastructure. They could have used AWS instead, but they didn't. I don't know if that means anything though, since there are odd ducks in every crowd.

One thing I like about Joyent is they're not trying to solve hard-to-impossible problems like Amazon (e.g. EBS), but rather realistic and immediate problems (e.g. guaranteed disk latency). For all EBS's promises, the latter is vastly more important to me; I've had few problems EBS would solve, and a truckload that higher-quality I/O would.

Also, those analytics are nothing to sneeze at. That's worth almost as much as the IO.

If you really care about low latency, get a dedicated server. You will even get better specs if you spend the same money. There are amazing deals out there, like this one:

https://robot.your-server.de/order

This is all desktop hardware. Its not the same thing.

From all the hosts I have used, Joyent has been the most rock-solid (by far). So personally it rings a bell :)

Qemu-KVM + SPL + puppet + some monitoring? Yeah, Amazon should start looking for a PANIC-button. ^_^

Update: Oh, come on. I've clicked to the TECHNOLOGY link and what I saw instead of technology review is load of BS. SmartMachines? OK, even I know that one should market any crap with Smart or Easy or Eco or LowFat prefix in it, because, you now, I have a SmartMachine... OK, fine. But what about technology?

Instead of writing that you're building a solution based on fast, light-weight, low-overhead, in-kernel, native virtualization system and RedHat supported libvirt stack, that you're co-sponsoring and actively participating in development process, and here is our contribution and so on, I see loads of BS. KVM is faster than Xen? I know that, thank you. I have a KVM instances running on my Laptop. ^_^

You're using ZFS? Linux native port? FUSE? So, you're active developer and tester? Co-sponsor? You have hired or supporting active developers? Providing a feedback to community? No? You just trying to sell me something you called a SmartMachine, OK, fine. I don't buy it. ^_^

SmartMachines == OpenSolaris / SmartOS (their IllumuOS-based distribution).

Also, they are actively developing and co-sponsoring IllumOS (and are IMHO the only reason why it didn't die yet), so next time do some research before bashing on a good company.

Point taken, thank you!

So, if it OpenSolaris based, then nothing to see here. Community is too small. Who will write and test up-to-date drivers for all new hardware that vendors are pushing to the market each half-of-year?

Why should that matter for cloud hosting? If you wanted to be able to install OpenSolaris on random hardware it might matter. But this is a closed environment where you never see the underlying hardware.

If it will be a closed platform, it doesn't matter.

I had a lot of experience with Solaris (x86 only - people tried to run Informix/Oracle on a cheap hardware) starting from Solaris 7 and onwards. It was always a problem even to install it. And lack of working compiler makes things even worse.

Starting from Solaris 10 they did a lot of work to improve the overall experience, but too late - everyone migrated to Linux to run the same crap.

And I must say, that after it was installed and tuned it was running quite stable as a database server and it can deal with heavy loads, while similar Linux instances failed now and then. But it was 5 years ago. Modern Linux kernels can handle everything quite well.

That's why their new cloud runs Linux and Windows VMs.

Oracle, just like Sun before it. Each time they do a code dump, the new and changed drivers can be integrated into Illumos by Joyent and other companies that depend on Illumos. Sure, it won't support "all the new hardware", but it will support enough. If they decide they need to, Joyent has the resources to do their own driver development.

Tomorrow they (Oracle) will decide to "slash the costs, do restructuring and refocus on their core businesses" as Cisco recently did, and guess what? ^_^

Considering that many of the most talented Solaris hackers have left Oracle and now mostly work at other companies that depend on illumos, I strongly doubt that they can't write their own device drivers.

Joyent probably has fairly homogeneous racks of servers. They only need a few drivers for their entire system, and they have people capable of writing them. If they can port KVM from Linux to illumos, what makes you think they can't port or write a driver?

No, I have no doubts that well-funded commercial enterprise can keep its code up to date, I considered from the Open Source, community-driven point of view.

My blog entry explains some of the technical background of this launch, with links to much more in depth material from both me and other Joyent engineers:

  http://dtrace.org/blogs/bmc/2011/09/15/standing-up-smartdatacenter/

Reply on news.ycombinator.com