How We Moved from Heroku to Containers with No Docker Experience | Hacker News Reader

How We Moved from Heroku to Containers with No Docker Experience

stackshare.io

67 pointssergiotapia33 comments

If you're only containerizing a Ruby application and nginx, I don't really see how containers are much better than old school Capistrano-style rsync deployment. (How often does your nginx version change, after all?) At the cost of an awful lot of additional tooling and somewhat unstable infrastructure.

For our simple Ruby workers, we just use Puppet to set up app server VMs with Ruby and nginx. Deployment consists of updating the Ruby code directory. It's pretty much cloud provider independent, as long as your provider can make a Linux VM, and it works fine, just like it did in 2010. :)

There's also the version of Ruby to consider. Switching to a tool like rbenv can require changing existing scripts (and iirc it felt like it was always more for development), and Ubuntu has removed tools like ruby-switch in Ubuntu 14.

I find this attitude interesting. By the same reasoning why are .debs better than building from source, and the ultimate - horse drawn buggies vs automobiles.

I've seen containerized applications described as "statically linked apps", it's one blob with dependencies included. I'm moving in that direction (vs cap-style deployments) because it guarantees that all machines are identical (from QA to staging, to production). Spinning up additional machines becomes trivial.

Containers add their own considerable complexity to the stack. If the dependencies are complicated enough, or you need fast scaling/reconfiguration on the fly, then containers can certainly be a net positive. But if you're adding Docker, Compose, DCHQ, and whatever else to your stack just so you can put Nginx and some Ruby code in a container, I'm questioning whether that ends up being a net benefit over Puppet/Chef + rsync.

I hear you. I like the idea of "deploying" a pre-built image. Capistrano "just-in-time" deployments (particularly deployment time "bundle") seem a little non-deterministic. Docker is certainly non-trivial though, but I think (I hope), it's a net win. Time will show.

In my mind, it certainly parallels the binary package (debian) vs build-on-demand package (arch? OS X brew) approach. Both seem to work, but I certainly prefer the debs.

That's where Ansible + Capistrano can work together nicely.

No need to change your application or deviate significantly from a "vanilla" Ubuntu install - just add your nginx, Passenger configs to Ansible, build the server, and then deploy with Capistrano.

What about people who don't want Heroku (or Heroku-style) dynos as application servers and workers, because you pay excessively for RAM and CPU and wrestle with configuration constraints, but want to keep the rest of the PaaS ecosystem?

I'm comfortable maintaining a fleet of stateless application servers, but don't want to maintain my own database servers, or queue servers, or SSL endpoint, or logging aggregator, etc. etc.

Is this a feasible setup?

Isn't this exactly what AWS offers?

Yes and no. Amazon is mostly IaaS in its outlook -- the closest they come to a "true" PaaS would be Lambda.

What you get from Amazon is a box full of building blocks, which you have to assemble yourself. For many people that is compelling. For others it is daunting. PaaSes are there for the second group -- an architecture-in-a-box.

I'd argue that Elastic Beanstalk, too, is Amazon's explicit attempt to package and present AWS as a PaaS.

Elastic Beanstalk is the Amazon PaaS offering. Lambda is much more like a BaaS provider - it abstracts away the concept of a server entirely, and instead just gives you an API to execute code on some cloud service.

You're right. I overlooked Beanstalk.

> Is this a feasible setup?

Yes, there are open source PaaSes you can install privately or on an IaaS provider.

I like Cloud Foundry, some people like OpenShift. There are others emerging as well.

It's not a matter of if you could, but of if you should. In OP's case, clearly not.

Right. In another time, we had racks of 1U servers, a NAS cluster with fast backplane, redundant switching fabric, a tape library, and fiber to the meetup room.

This time around, if we were worrying about anything like that, it would be a distraction from the product!

to a degree, i guess. On AWS there are a lot of services, which would qualify as PaaS offerings. Even on Rackspace, you get Databases (and HA Groups), Load-Balancers (w/ SSL termination), Queues (albeit the obscure, OpenStack one) and ElasticSearch for Logging (the latter is pretty hefty priced IMO, though).

So we're able to run quite a lot of stateless 12factor-like services on Rackspace (using kubernetes), RabbitMQ being the exception we have to manage ourselves (which sucks).

It's certainly feasible. Your containers run on a public cloud, just as Heroku does, which enables access to things like logging-as-a-service. But you'll soon find that such services are just as horrifically marked up as Heroku, because they too are middlemen who need to make money. (I'm referring to services like Loggly or Logentries, to continue with the logging example.)

We've been really enjoying AWS's hosted ElasticSearch, PostgreSQL, Elastic Load Balancer, etc. Most of the benefits of a hosted service at the cost of an EC2 instance.

Definitely agree that the other services are marked-up too! The issue is one of leveraging developer time at a dynamic startup. At some scale, it becomes cost-effective to manage your own application cluster. At some (usually larger?) scale, it becomes a good idea to administrate your own database cluster. We're definitely not big enough to consider managing our own RabbitMQ or log aggregator or other ancillary services from a developer-time opportunity cost/benefit perspective, though!

FWIW Cloudwatch has logs service.

Database servers - AWS RDS

Queue servers - AWS SQS

SSL termination - AWS ELB

Logging aggregator - depends on how you're deploying. We're deploying our containers onto elastic container service and adding one of these: https://github.com/gliderlabs/logspout to each instance. Elastic beanstalk will do the log aggregation for you

Take a look at Convox, convox.com.

It's an open source "PaaS" that runs in your own AWS account.

It's a thin configuration layer for AWS. So you get easy access to ECS for running your app servers at resource cost, then simple tooling for adding an RDS database, adding a Heroku Postgres URL to you app env, or adding to Papertrail.

Full disclosure: I work on Convox full time.

Google or Amazon will both give you PaaS systems where you can run pretty much anything. I have more experience with kubernetes than AWS Container Engine.

Openshift and Cloudfoundry should give you alternative PaaS type systems which in arguably are less versatile but more complete.

Read up on MSaaS or CaaSi: http://venturebeat.com/2015/05/09/goodbye-saas-hello-contain....

The promise of this stuff is that you run the servers and control what gets deployed, but the vendors manage the software for you, like they do with SaaS. Is that what you are asking for?

At a previous company, we initially set everything up at Heroku. As things got cost prohibitive, we moved them one by one to AWS. Since Heroku is setup on AWS, it was easy enough to connect to our Heroku postgres instance from AWS web servers and it doesn't add any extra latency vs. running direct on Heroku. Over time, we slowly migrated everything off to our own infrastructure on AWS, but we were able to leverage Heroku to avoid all of that development effort until it was cost effective to do so. You'll need to deal with your own SSL endpoint if you run your web servers on AWS, but it's pretty trivial to do so with ELB.

Out of curiosity: does Heroku give you a better deal as you grow? I know for a fact Salesforce will discount their licenses more and more and you buy bigger quantities and sign up for longer contracts. It'd be surprising if Heroku doesn't do this at all...!

From my experience, no.

They have the opposite model: give it away to small players to hook them up, then rake it in from heavy users.

That's exactly the roadmap I had in mind. Glad to hear someone had success with this route!

This is a submarine for DCHQ which looks to be some proprietary docker orchestration service that doesn't list prices and doesn't have an open source version. Pro tip - customers don't want to be forced to talk to your sales team before they can try your product.

It does look like an advertisement.

I guess it's changed since you last looked -- DCHQ does have a pricing link on their front page now.

I'd be really curious to know how many servers they're running. You can get a lot of iron for $3200/month.

Also, if I read that correctly it looked like they maintained two sets of infrastructure and one was always spare so they could do green/blue deployment. That sounds like a great opportunity to use on demand instances so you only pay for what you use.

For those who want a less all-or-nothing migration path from Heroku, might I suggest looking into Cloud Foundry? Several of the buildpacks (most notably Ruby) are derived from Heroku's upstream repos.

If you decide later that you want to build your own containers, Cloud Foundry has you covered there too, with a pathway for Docker images to be placed, launched and monitored on an equal basis with buildpack droplets.

I'm biased, of course. I work for Pivotal, which donates the majority of engineering time to Cloud Foundry.

The cloud environment is becoming a "wild wild west" just like the app stores were a few years ago. It all has to do with how much money is being made in those markets. Maybe this is the result of AWS finally breaking out their revenue numbers.

There is so much marketing and plugging of alternatives to [INSERT CLOUD PROVIDER/LAYER HERE] that there really isn't any substance to all this other than reading like an advertisement.

Reply on news.ycombinator.com