Heroku XL | Hacker News Reader

Heroku XL

heroku.com

88 pointszrail64 comments

A quick check on the documentation reveals instead of routing the requests to the first available or least loaded dynos, the dyno routing algorithm now routes requests to random dynos(Reference: https://devcenter.heroku.com/articles/how-heroku-works#dyno-...)

Routing requests to random dyno is definitely not the smartest routing algorithm because it kind of defeats the purpose of having a routing layer. If you are not aware of the issue between Heroku and Rap Genius, I will think it is important enough for anybody who considers Heroku to be aware of. Here is the URL with relevant information: http://news.rapgenius.com/James-somers-herokus-ugly-secret-a...

While good for some dynos, it can be poor for others. Any system that handles singe requests serially is an irresponsible choice for anyone who might bother to look into queuing theory enough to blame the router but not be willing to blame their application server.

There are systems that do benefit from other kinds of routing but there is very little margin to favor non-concurrent request handling in mostly state-less components like web servers. There are lots of responses on the web that take a much deeper look at the heroku router if you care to search. The conclusion is not so clear that Heroku is doing it wrong.

the dyno routing algorithm now routes requests to random dynos

That's not a change; that's how routing on Cedar has always worked.

Random routing is more or less industry standard. NGINX and ELB (what I would consider two gold standards for load balancing) both use Round Robin (non-inteligent, pseudo-random) routing variants:

- http://wiki.nginx.org/LoadBalanceExample

- http://serverfault.com/questions/537450/what-algorithm-does-...

Round Robbin is certainly not equivalent to random, and is much much better for most applications. If all the requests take about the same amount of time, then round Robbin will give the request to the one that hasnt received a request in the longest amount of time, which is clearly the best choice in this situation. Randomized has a 1/n chance of handing the request to this less busy node, which is why the problem becomes worse as you scale.

The issue with round robin routing is that you can't do it in a distributed routing mesh. You need to share state between routers and that coordination overhead is non-trivial.

This has been talked to death, but for the general case, for applications that can handle concurrent requests (like Rails with a rack server just a tiny bit smarter than Webrick), random routing works just as well as intelligent routing with far less coupling in the routing later, making it more robust and easier to manage and scale.

Rails servers like Unicorn can still only support ~4 concurrent requests on regular dynos (roughly 1 per virtualized core), lessening the problem but not solving it if you have many requests per second.

Actually that makes a tremendous difference. It is a bit difficult to explain but even 2 concurrent requests per dyno is a lot better than 1. As you go up from 2 to 3 to 4 the difference quickly becomes minimal. I'll try to explain why, but warning: hand waving ahead!

The problem with only 1 concurrent request is that if that request takes long to process then it can block other requests that are queued up in that dyno. Lets say that 1% of the requests take long. If a dyno can process only 1 request concurrently then at each request there is a 0.01 chance that that will block other requests. If however it can process 4 concurrent requests, then the dyno will only be blocked if it gets 4 long requests at the same time. So there is only a 0.01^4 = 0.00000001 chance that a dyno will be blocked.

Not quite. If you keep routing to a blocked backend, any requests that hit that backend will have unacceptable latencies. Since the routing is essentially random, you'll still be fielding about 1/4 of your requests to that backend, so one long request winds up making 1/4 of all your requests have unacceptable latencies.

I don't think that's how it works. Within a single dyno with concurrency of 4 a single request can not block block any requests. It can only block 1 thread, but then the other requests will be processed on the other threads. Routing is only random to dynos, within dynos it is not random.

At the end of the day, there must be some applications that cannot be fixed by increasing concurrency to the limit in a standard dyno. They would not bother implementing XL dynos if this was not the case.

It's most beneficial for application that cannot fit more than 1 on a single dyno.

I run http://codetriage.com/ on Rails 4, Ruby 2.1.0 and have 5 unicorn workers per (regular 1x) dyno. With a little GC tweaking I could likely get 6, but i'm being cautious. With puma hybid server I can get even more concurrency, but don't need it for now.

This is assuming the application developers understand the problem and write the application accordingly.

There are workarounds, which require some legwork and forsight.

Reducing the pitfalls inherit in deploying a web app is generally what Heroku is selling. It is why they are making the big bucks. It would have been nice if Heroku would have been able to find a solution and implement smarter routing so that this just became a non-issue.

Heroku made the bold decision to design its routing technology to be optimized for things like Node and Go rather than Ruby. So if you are using Ruby on Rails (like RapGenius) Heroku's platform will simply fall short, both in terms of efficiency and flexibility.

XL Dynos let you throw money at the random routing problem which helps routing efficiency at the expense of spend efficiency (b/c they are a lot more expensive and scaling down to 1 XL dyno is still pretty expensive).

In my opinion, if you're using anything other than 1X dynos on Heroku you are compensating for some other part of your stack being broken.

We're probably months away from Heroku having some serious competition in the super easy PAAS space. Heroku's high prices have summoned competitors en masse, and by some stroke of luck Amazon has not really made a serious effort to encroach on Heroku's profits.

Running modern Rails on a threading server like Puma, especially on JRuby or Rubinius, would probably get an application within the same ballpark of concurrency. At the very least it'd be better than Unicorn or Webrick or whatever the heck Rapgenius was using to serve their app.

Running rails on JRuby and Torquebox (especially the upcoming version 4) would provide MUCH better performance.

Am I correct in assuming that the relevant characteristic of Node and Go is high concurrency? If I'm not mistaken, Ruby can have that too, if you use something like Puma or Thin rather than Unicorn.

Python may be on the wrong side of this, though, as long as Gunicorn is the de-facto standard WSGI server for Python apps on Heroku.

Not just concurrency but also boot time.

On my Production Rails app `time rails s` takes about 0m2.273s for me (Rails 4, Ruby 2.1.0). Not an insanely fast boot time, but nothing to lose sleep over.

Roughly how many models and controllers are in the app?

In my opinion, if you're using anything other than 1X dynos on Heroku you are compensating for some other part of your stack being broken.

You may want to read this section: https://blog.heroku.com/archives/2014/2/3/heroku-xl#understa...

Performance dynos give you much higher performance and much better consistency, resulting in dramatically reduced tail latencies over 1X and 2X dynos.

Choosing performance dynos for a high-traffic or performance-sensitive app is definitely not "compensating for some other part of your stack being broken"; in fact it's a very smart choice.

I tend to agree with you. For some large interconnected systems, long tail latencies have a high complexity cost because you need to code around the 1% cases when (for example) web service calls take much longer than normal to return results. Client code becomes more complex.

> Amazon has not really made a serious effort to encroach on Heroku's profits

Amazon hasn't, but there are a reasonable number of simple PaaS competitors these days:

Microsoft's Azure (http://www.windowsazure.com/en-us/pricing/details/web-sites/): ASP.NET, PHP, Node.js, Python

Red Hat's OpenShift (https://www.openshift.com/): Java, Ruby, PHP, Node.js, Python and Perl

Nodejitsu (https://www.nodejitsu.com/): Node.js

amazon has elastic beanstalk, but it's still not as easy as heroku.

Ninefold.com has a wider choice of server sizes and is paas similar to heroku

It seems to me this move is to compensate for the well understood issues in their routing layer.

You can call these issues "bold decisions." In my opinion, it reads more like, "We don't know how to fix the problem. If you really want to pay for it, we will let you skip our routing layer and solve the problem yourself."

It is a bit disappointing. However, I've never had to write a routing layer to handle 60,000 requests a second, so I can't really fault them.

XL dynos use the same router as everyone else.

Yes, but your process(es) on each dyno can do smart routing for all requests hitting that dyno.

Dyno Sleeping: Another interesting fact that hasn't been called out is that even with the Heroku XL Dyno, which costs ($580/mo), it will be spun down if not used for an hour. I don't understand how that matches the promise of 'performance' https://devcenter.heroku.com/articles/dynos#dyno-sleeping

There's also a more informative blog post here:

https://blog.heroku.com/archives/2014/2/3/heroku-xl

Thanks - this is a much better resource than the marketing page!

Heroku's $0.05/hr base pricing hasn't changed in a very long time, while Amazon has both dropped price and increased power.

I love Heroku's service, but it's starting to get very expensive. This $576/month plan proves it. :/

And it still goes down when Amazon does (or has this changed?). Sure you get something extra, but paying more just to add another layer that can possibly fail? I don't know...

edit: Oh, and you are getting vendor-locked and they can go Google-App-Engine on you.

There is no vendor lock-in with any aspect of Heroku. https://www.heroku.com/policy/promise

They don't actively lock you in which is great but you will almost certainly end up with bits of your workflow which are extremely dependant on Heroku. IMO pure AWS has exactly the same problem if not more.

If you compare this to deploying to "choose a generic VPS" the lock in is substantial. If any of my apps fail, I'm comfortable I can provision an Ubuntu VPS from any provider with a one line chef command and then deploy to it with one more.

So although Heroku isn't locking you in through any bad practices on their part, by using them you're choosing to make your workflow very specific to them. To me and quite a few of my clients, while Heroku is awesome for getting started, there isn't enough benefit for the increased single provider dependency and costs.

Not hating on them at all, I use them quite regularly, but I can understand where people are coming from when they talk about Vendor lock in. Perhaps a more accurate description would be single vendor dependency.

Why can't you write a Chef cookbook, Ansible playbook, or the like to deploy a 12-factor-compliant app on a generic VPS? Especially since all the Heroku buildpacks are open source?

I think it's less that you can't and more that most people won't. The reason I know I can provision a new server for app x without hassle is because the chef setup to do it gets used regularly, it's part of the workflow of developing the app. If it breaks I know it's broken because next time me or someone in the team spins up a staging enviro, it won't work.

If you're using Heroku sure you could write that cookbook but in 90% of cases it's going to end up being unmaintained because it's there as a contingency that no-one uses in their everyday workflow. What's more for it to have real value it needs to be tested regularly to make sure it actually functions as expected.

So it's definitely possible to avoid the dependency as long as you don't mind adding a big chunk of devops work. At that point you've got to ask why are you paying the premium for Heroku if you're maintaining the devops expertise and codebase to deploy to a VPS alongside?

Yeah, I don't buy that. The build packs are open source. Clone them and use them to build your app. It's no different than Heroku. Yeah, you have rebuild some of the stuff they were giving you for free, but that's the whole premise behind their business. They handle things like deployments, etc. so that you don't have to.

Well, you can't just take your app as is and move it elsewhere, can you?

Kind of. I've actually found that bundling my apps for heroku actually make them easier to move onto different platforms. The fact that I can run foreman start and have my dev environment running locally is extremely useful to our team.

https://blog.heroku.com/archives/2009/4/24/commercial_launch

It was .055 / hour when it launched in 2009. Back then, EC2 small instances cost .10 / hour (http://aws.amazon.com/about-aws/whats-new/2009/10/27/announc...).

Now? Heroku is .05 / hour, and small EC2 instances are .06 / hour. This is why I stopped using Heroku a year or so ago for anything I plan on paying for. They just let Amazon keep driving down their bottom line without passing any of those savings onto their customers.

shrug

I kinda feel like the service is either worth it to you at the price they're charging... or it isn't. I don't particularly care what their wholesale costs are.

The price they're charging is their price minus the Amazon price. You are paying for a set of tools that sit on top of Amazon. You choose to either invest in those tools yourself (and save the the difference between Heroku and Amazon) or you don't. Every time Amazon reduces prices, this means the cost of those tools increases. Since Heroku has launched, the price of those tools has steadily increased by virtue of not staying in line with Amazon's price reductions.

I get what you're saying... but the price they're charging is the price they're charging. The whole point is that I don't have to care how they make it work, just that it does work.

Would the price for Heroku magically become "better" if they decided to switch from Amazon to a more expensive or less efficient backend? That can't be right.

Heroku has many alternatives -- hosting directly on Amazon among them -- but (IMHO) there are not many direct competitors. Doing it any other way involves different tradeoffs of time, money, skill, and maybe lock-in risk. I sometimes buy a burger at a restaurant even though I know I can make one at home. That doesn't automatically mean it was a bad call even if raw beef is on sale at the supermarket.

Doing it any other way involves different tradeoffs of time, money, skill, and maybe lock-in risk.

This is what you're paying for. The cost of that is the difference between hosting yourself on Amazon and what Heroku charges. That's the cost to your business of using Heroku. This cost has steadily increased. This isn't rocket science, and it certainly isn't burgers at a restaurant. I get that for some people that cost is a worthwhile expenditure, but as that divide widens, the number of people it makes sense for will decrease.

To try and equate this to your burger analogy, if raw beef kept dropping in price, but restaurants all charged the same, more people would cook at home, no? Now, imagine you're feeding 100 people a day. At some point the cost savings of cooking the beef far outweighs the luxury of sending 100 people to a restaurant.

Heroku isn't just a transparent proxy to EC2. You are getting several other services that are not easy to put together on your own - routing infrastructure, simple git push deployment, world-class postgres tooling. To put this kind of stuff together on your own is costly, and you will likely make a lot of mistakes on the way (mistakes Heroku has already made and learned from).

Another thought: When you use Basecamp, are you calculating how much they are spending on hosting your data?

This plan is $576 a month, the actual hardware you get for that is mediocre at best. A postgres instance and git push deploy isn't worth that much and isn't that hard to set up yourself.

sure, you could run into some issues while doing it yourself but that risk isn't worth nearly $600 p/m IMO

How many hours do you spend a month on devops tasks? And on top of that, how diligent are you with applying patches, testing backups, and all other sorts of drudgery necessary to keep a production site running smoothly? Take that number of hours, multiply it by your consulting rate, and there's a good chance Heroku may save you money.

For most production apps, you'd need at least a part time ops person to do things right. That's the kind of cost Heroku saves you.

Amazon itself is already very expensive compared to a dedicated server.

I think Heroku is still a fantastic option for when you just need to get a beta system online. It's certainly a lot less intimidating than navigating the galaxy of AWS offerings. Trying to scale on Heroku doesn't seem like a great idea though.

Heroku is the simplest Paas out there.And I like the fact that they built their deployement workflow on simple stuff like git and bash script,but yeah, it's quite expensive.

I wonder if i could reproduce this kind of workflow on my own server, since i dont really need load balancing or auto-scaling for most of my apis.

Dokku does just that:

https://github.com/progrium/dokku

It's even based on heroku buildpacks so it's very compatible.

Dokku is far from stable and it's still going to be a bit more intensive than flipping a switch when you have to scale. The beautiful thing about heroku as a product is it removes an entire domain of skills/worries from building your app. Like Heroku is your linux/ops person.

How's your bandwidth bill?

It's made a massive difference to us; we've gone from 10 x 1x dynos to 2 x XL - it's done the following to our latency graph: http://i.imgur.com/BGKwGfp.png. Orange is perc99.

Thanks for sharing! Can you share your before and after web server configuration? Also what web server are you using, Unicorn?

Rails 4, Grape, Postgres. We use unicorn; went from 3 workers to 30, though I've not had time to tune this since.

So you changed from $0.50 / hour to $1.60 / hour. Did you first try with, say, 20 x 1x dynos (that would cost $1.00 / hour)?

Yep, but it didn't help like this did. I don't have graphs for you I'm afraid.

Someone explain me, why I would need a 6GB RAM machine over a dedicated server that I could have with 10 times more machine at the same price..

Maybe you don't have a full time sysadmin in your employ. Maybe you are a one-man team that would rather spend your time working on user-facing features instead of fiddling with infrastructure.

Heroku also folds bandwidth and electrical costs into their price model.

Having built a couple of cloudy services, I can confirm that these far dominate your costs at scale.

Reply on news.ycombinator.com