Goodbye Heroku
reefpoints.dockyard.com
reefpoints.dockyard.com
That being said, it seems like the model itself is not flawed, just the execution. I've met many developers, especially in the Node community, who fundamentally believe they as software engineers should not have to spend any time learning devops.
My position is that a good software engineer today should be familiar with the full stack from frontend (SPA, javascript, html, css) to backend (sql, basic db admin, db design) and devops. Obviously, you can't become an expert in everything, but the knowledge level required to become "proficient" in these areas seems pretty achievable if you're well-versed in software engineering fundamentals.
FWIW, it took me about 4 weeks to get a "startup-ready" fully open source 3-tier stack running in EC2 with full automation using Chef (which is a whole other topic) plus continuous deployment using Jenkins. Ultimately, we'll pay more for EC2 per month upfront, and I had to absorb the upfront time hit, but now that we're setup, it's extremely easy to tweak things, and we have complete transparency.
I also took advantage of AWS IAM roles so that the server is pre-authenticated for certain S3 buckets.
PM me at josh dot padnick at gmail /period/ com with some background on what you're trying to do and I'll see what I can share. For many reasons, I can't make our Chef code repo public, but maybe I can share some code samples.
Edit: Regarding Chef, for more info you might check out http://www.slideshare.net/JoshPadnick/introduction-to-chef-a....
As far as online examples, yeah, the learning curve does kind of suck. Also, if you're starting from scratch, SaltStack may very well be a superior technology (don't know enough about it). I think the key with Chef is to learn how to read the docs, using the application-cookbook pattern (where you never touch the cookbooks you download online and instead customize them by "wrapping" them with your own custom-defined cookbooks.
It's also helpful to use very thinly designed roles, and define special cookbooks as your actual roles. This way you can version-control them.
I basically learned by reading the same material in multiple places, especially in books on Safari Books Online, and IRC was also a huge help b/c the community was VERY helpful.
I think Chef is a classic case of where the technology is somewhat inelegant and perhaps bloated, but a mature community is there and it's battle-tested so once you suffer the pain of ramp-up you get a huge benefit.
HTH
I'd like quote Adrian Cockcroft: "the undifferentiated heave lifting". However, the heavy lifting is the daily operation work, not the knowledge to scale your app, HA/DR, CD/CI, etc.
Compared with the black-box model of Heroku, I vote for infra-as-code+desired state, e.g. you order, we cook.
Disclosure: I am the founder of visualops.io, a white-box devops automation service for AWS.
Heroku is great for working on your own project or if your data scale is small. Otherwise, the cost is just too high.
Of course that has yet to be empirically proved.
Significant overhead costs to go from nothing to steady-state instantaneously would be enough to scare me.
Here's a tip. You are a black box. Your customers don't care if it's your fault, Heroku's fault, the power grids fault or a meteorite's fault. They just care that their service is up. If anyone from you all the way down to the silicon breaks, its your problem. Not Intel, not RealTek, not Western Digital, not ComCast.
Stop playing the blame game.
I've seen a fair amount of people who loved Heroku for ramping up apps only to feel trapped once they started scaling their service. PaaS definitely has its use case, but I wouldn't recommend it for anything beyond prototyping.
In fact (shameless plug), this is why I've been working on devo.ps, to lower the barriers of entries to managing your own servers (on AWS, Rackspace, Digital Ocean, Linode...) using tools familiar to developers (Git + YAML).
@bcardarella happy to add you to our beta testers; we'd love to get feedback from a heavy Heroku user. Email is in my profile.
If there are other Heroku users affected, drop me a line and I'll see if we can help you get set up on your own infrastructure with devo.ps.
What do you do when one of your droplets disappears or crashes?
Where do you host your database?
Not quizzing, real questions from someone also looking to drop heroku.
They already are the same price/performance [roughly, unless you are buying DO's $5 nodes]. They also have stuff like load balancers built out and are reasonably reliable [although only at the datacenter level, they don't have multi-DC load balancers]:
https://www.linode.com/pricing
Its the same 1 Core / 1GB == $10 pricing DO uses. [roughly]
edit: years
Not complaining, that was exactly what I was paying for, but it's certainly something to take into consideration if you're not comfortable with command-line server admin.
Both the parent and the OP both suggested switching to DO which is the point of comparison.
I've never had better support/more comprehensive with DO than Linode. Then again, I've had a Linode account for like 6 years and I think I've opened as many tickets. ;)
I believe its all of them:
You'd likely be able to do the same with chef, ansible, puppet or whatever else. I don't have deep experience with those options, so I can't really speak to them.
[1] -- http://rubber.io/
Similar value with Heroku: autoscaling, auto-healing, push-to-deploy, but not blackbox. Instances runs in your own account. Even if we go down, your app will not be impacted.
But, I think we fill that niche between hand-holding and trusting devops/sysadmins to do what they think is best. We try to handle the monotonous tasks for you (configuring DNS, setting up backups, configuring load balancers, etc.) out of the box, but give you a great deal of flexibility to influence that. Almost everything is bash and ruby with a healthy set of utility functions.
I admit, it occupies kind of a weird middleground. If you use Heroku because you're not well-versed in sysadmin tasks, there will be something of an uphill battle (but a great learning opportunity). If you want to outsource everything, it's not a great fit. But, if you want a lot of flexibility, have unique app contstraints, want the abilitity to switch betweeen cloud providers and not deal with everything from scratch, it's a pretty good fit. I've personally used it with EC2, DO, vSphere, Vagrant, and leased hardware. It's nice being able to treat them basically uniformly.
So far, I like it; Dokku is pretty slick. Deployments are more or less the same as Heroku - "git push dokku master", in the case of my Rails app. But that said, assuming Heroku's security is super tight, I have a bit of work to do to secure my droplet.
I'll probably still have to scale myself (or just switch to Linode and use their load balancer), and with Dokku I've got a PostgreSQL DB in my Dokku container.
Droplets disappearing or crashing... not sure, but I sure hope DigitalOcean's backups are good :)
This sounds painful.
Dokku is only a single VM solution though, right? Flynn (http://flynn.io), the all grown up Dokku, isn't production ready yet.
I'd love to hear that I'm wrong and that scaling up with Dokku is a piece of cake. If so, please correct me!
Even if you get a hundred thousand hits in an hour, that's only 28 per second. I am pretty sure even my toaster can handle that.
I've been fireballed, HN'd, reddited, etc. All on my little 256MB Linode instance (now 1024). It has never been anything but zippy.
Set up your own HAProxy instance?
https://www.digitalocean.com/community/tutorials/how-to-use-...
Setting it up doesn't look too hard, and you're not going to need it until you reach significant scale - so probably 90% of the people reading this without a dedicated ops team aren't ever going to need it. Startups often seem to over-engineer servers at an early stage or assume they'll need a massive cluster when a single server with multiple processes could serve perfectly well till you actually hit some sort of scaling problem (i.e. millions of users a month). As an example, the HN website ran for a long time on one server (not sure what they use now, but it's not heroku :). Very large VPS instances often cost less than scaling with heroku.
Also, other providers like Linode offer load balancers without setup for a low cost.
What do you do when one of your droplets disappears or crashes?
ansible myserverplaybook.yml
to set up another one in a few minutes. But you'll find this just isn't necessary unless the server config changes - uptime will be measured in months on most providers (not sure about DO in particular, haven't hosted anything serious there).
Where do you host your database?
On another DO droplet, with remote backups, or even on the same one if you have a sizeable VPS and low to moderate traffic.
It looks like the best of both worlds -- higher level than Dokku, but choices when it comes to the underlying infrastructure provider (Linode, DigitalOcean, AWS, or some combination). Small overhead charge for the management which seems very reasonable, otherwise you're just paying for the (virtual) metal from the IaaS guys.
https://github.com/cyrusstoller/gardenbed
It sets up postgres/rbenv/nginx/ s3 db backups etc ...
You design your systems to degrade gracefully, hopefully transparently. Screwing up your engineers' sleep schedules is a terrible idea, so you deploy during the day when people are awake, clear-headed, and on the clock.
I'll take a 1PM outage over a 1AM outage any day. If that's bad for business, business should pay for the engineering time to build a more resilient system.
Also, 1pm and 1am aren't the only options. 8am EDT would have been within the work day but still avoided peak usage in the US.
I'm sure you can find engineers willing to work garbage-man hours, but it's gonna cost you.
I'm not sure if you're just being sarcastic, but talent in North America isn't limited to the Pacific Timezone, and such a claim borders on hilarious delusion. I don't say hilarious emotionally or pejoratively, but rather it is actually ha ha funny that anyone would actually believe that.
However yes, there are endless loads of top-skill talent that will happily do "maintenance" in the middle of the night. I've done it on occasion, and slept in the next day. Big deal. Half the time I simply took a timeout from some online gaming I'd been doing.
The pool of top-skill-talent-with-a-self-esteem-problem is rapidly diminishing.
People do night shifts in so many industries. There's nothing wrong with that, it's just part of the job. My building security guard does a night shift and what he does is way more important and needs more attention than an engineer. Unless you think we should not have security guards in the night.
You need to build a resilient system, and that's gonna cost you. It's a lot less fun than adding features, which is why business people tend to put it off and instead manipulate the tech people into working crazy hours.
The 24/7 web culture is dying. Adjust your business plan accordingly.
If your working environment is good, it's pretty easy to just say "I'll take tomorrow off and come in to do it on Saturday" without feeling like you're getting taken advantage of.
For professional examples of "night shifts", we have ER doctors. They do work 12-24 hour shifts -- but they're allotted a room they can sleep in between patients, and they often have 2-7 days between their shifts.
You going to pay infrastructure staff what you pay ER doctors for as much on-task work? Are those people going to actually know what to do when you put them on-task?
What happens when the IT staff needs to get a developer on-hand to resolve an issue? Wake them up?
>What happens when the IT staff needs to get a developer on-hand to resolve an issue? Wake them up?
Yes. Have you never worked at a place with an on-call dev team?
10am - 10pm [on call] for US West.
7am - 7pm [on call] for EU [UTC +1]
4 days on / 3 days off. [48 hr on call, 40 hours of work]
Its doable. Just expensive to cover 2 jurisdictions and maintain 4 shifts worth of people.
No system is ever going to be 100% reliable and cost effective.
Of course people do night shifts, but at the same time, we've learned that you're better off fixing things during the day shift. The night shift is an emergency crew.
And the idea that a building security guard needs to pay more attention than an engineer is beyond ludicrous.
You've made an argument to authority with your lead-off sentence of "that's not how businessses are run" and a no true scotsman argument with "no sane person running a business", and then tied it up with an absurdo ad reductum analogy about security guards.
There are plenty of shops that I've personally worked with, and many more that I've read about here on HN that do daytime maintenance / make potentially breaking changes to production. I'd be willing to bet Heroku does a mix of day and night time scheduled maintenances, with scheduling determined by a reasoned analysis of the risks and potential for an outage.
If you think a 1 AM maintenance window helps anything, I'd bet you've never seen the sunlight come up as you staggered out of a datacenter 10 hours after your "midnight maintenance" went south, and no one with an optical light meter to help you trace down a dodgy fiber run was awakee at your transit provider, so you had to wait until the first shops opened to buy your own...
...ok, maybe I'm a little bitter. What I'm getting at is, a 1 PM outage - although more customer facing than a 1 AM outage - will almost definitely be resolved faster, because resources outside of the engineers performing the maintenance are in abundance at 1 PM and not 1 AM.
Bonus points: your engineers will not be zombies for the next three days, and that's important both for post-outage work and to make sure everyone stays sharp for the potential "bounce outage" (e.g. your first outage was the core router dying, the second will hit the same week when your new replacement core router's slightly upgraded IOS version causes a BGP flap under certain conditions that don't arise until a good 48 hours after you mail out your customer outage report....)
That might sound trivial, but at scale - if you have tons of engineers and are doing tons of maintenance all the time - it's actually really important, at least IMHO.
(Quick, take a guess - how many Heroku daytime maintenances do you think went off without a hitch that you've never heard of?)
P.S. what time zone are you in? I assume your time zone's 1 AM is the important one, so someone else on the other side of the country is going to either have a late evening or early morning maintenance window..... ;)
Hey what about us people in Asia who use Heroku ... a early US AM outage is likely to be in our early PMs.
As a business owner who's customers are actively engaged with my staff who need servers and resources to be available during the day when, you know, my CUSTOMERS are awake, your recommendation for outage windows would be completely ignored.
You schedule outages around the people who pay the bills, not the people who don't.
Building a system for graceful degradation costs time and money.
Hope for the best, and plan for the worst: Build resilient systems when possible, but why risk (or guarantee) outages during the day? OS upgrades, router swapouts, and so on can NOT take place during the day. That would be foolish in most industries.
Also realize that "just make systems resilient" is not an easy thing when multi-million dollar transactions are occurring on 30 year old code. If everything were greenfield, it'd be different. But it's not.
When 24/7 means millions of dollars, hire a four-shift redundant geo-distributed team.
Australia needs to step up its subsidies to IT outsourcing firms.
Do everything you can to reduce the risk, including building a resilient system, AND not taking a major swath of it offline during a high traffic period.
And even if they did build their service in a way which could tolerate a complete hosting platform failure - degrading a core service during your customer's business hours is enough to make heads roll. It's just stupid.
The benefit of having the engineering team around to troubleshoot smaller-scale issues before they turn into large-scale outages vastly outweighs the small benefit of potentially moving the extremely rare massive outage to a less busy time of day.
12:30am on EST is 3:30pm AEST (Australian Eastern Standard Time) and 7:30am in London.
That way you have skilled people awake all hours should a major issue develop. These people can fix the problems while US based clients are sleeping and non-US clients aren't stuck with waisting a day with less severe faults because the engineers only work 9 to 5 PST.
Not used it yet but www.cloud66.com always interested me.
http://www.stefanwrobel.com/heroku-to-opsworks
http://artsy.github.io/blog/2013/08/27/introduction-to-aws-o...
I think there's a reason, but it's not outright unemployment - yet.
On the enterprise side what I'm seeing are traditional sysadmin positions being eaten up by aggressive VARs and software vendors. They all have XaaS plays now and they're targeting their existing customers first.
The pitch is literally...
"Hey, I can save you $350K/yr. Just fire your ________ team and use our PaaS. We'll perform the migration for free."
Assuming the organization doesn't want to pursue their own internal or lower-level external service it leaves the sysadmins to either somehow make the leap to administering a provider system or diversify into other roles where they don't exactly fit.
Twitter is looking to fill five open SRE positions in San Francisco[1] and two in Seattle[2].
[1] https://about.twitter.com/careers/locations/san-francisco
[2] https://about.twitter.com/careers/locations/seattle
If you'd like to know more about the SRE org at Twitter (Seattle in particular) drop me a line. My email address is in my profile.
Elaborate please? CF uses a variant of them, and they're ultimately just a simple three-script harness to build and run your app.
There are pros/cons vs. dockerfiles , chef/puppet/ansible, or just doing a Netflix style Aminator, I suppose, but "terrible" seems exagerrated.
Dockerfiles are a bit better by allowing you to pick the base image (and thus distro), though there are tradeoffs with that too.
Now say I want to use python with R. It is a huge pain. You have to use heroku-buildpack-multi [1]. The problem is that environment variables and installed code from the first buildpack are invisible to the second. So you end up having to hack individual buildpacks, which is just gross. It worked in the end, but I wasted days on it.
A concept like docker, where you fire up a container, install stuff with apt-get, and then save the results is greatly superior.
We use the the slugbuilder from the flynn project[1] which gives us a finished version of the app in a .tgz which will than be extracted into a docker container that already has all the other necessary stuff setup (logging architecture etc.) which is needed in different parts of the application (webserver, background workers...)
From the docs it looks like Deis reuqires an advanced Ops person to manage a cluster of Deis machines.
It took a few hours, but I was able to get my Heroku-based Rails app up and running on a Dokku droplet (to be fair, the most time-consuming part was getting the SSH keys right and figuring out that I needed a domain name - e.g., couldn't just set it up with a droplet's IP address - at least not with the DigitalOcean tutorial's instructions: https://www.digitalocean.com/community/tutorials/how-to-use-...).
I use it for Node deployment, but they also host ruby, php, go and java apps.
Learning how to secure your servers and scale your app is not an overhead, it is a competence compared with those who don't.
Like some people said before, Heroku is great for prototyping but just not a good deal in the long run. Too expensive, way to risky/unreliable for serious stuff
I've not seen it mentioned so far: RedHat has OpenShift (https://www.openshift.com/) out now, v3 will see Docker support too. With only a quick play, it seems to be a mix of Heroku and dedicated host.
On sysadmins: talking to a guy who managed AWS nodes for a popular iOS game you've probably played, he made the comment that it was great to have a new team member onboard now, dedicated to looking after all the VMs. I figure he's rediscovered sysadmins, just using different kit.
The risk I see with managing your own Dokku, etc is handling issues/maintaining uptime if it's not your only job/skill. When another HeartBleed comes along, can you react quickly and competently enough?
In short, Heroku thinks that firewalls encourage people to be lazy about security, and leaves people crippled if the firewall is exposed. Due to this philosophical difference, they would rather force everyone to rely on SSL for all secure communication, and leave every machine accessible to the internet, including (for example) your database server.
[1] http://deis.io [2] http://coreos.com [3] http://openstack.org
Of course, scheduled downtime should be done in off-hours, but modern services like Heroku don't have scheduled downtime.
You can't build a company of developers selling products to developers while fleeing the working hours of developers.
During this time, Heroku command line tools and web interface actions to create and restart dynos can be delayed.
It's really the complete opposite case, I think. For the value (time is money!) they offer, it's amazing they haven't increased their prices as they scale.
fwiw, here's some resources i collected around docker.
http://wayfinder.co/pathways/53619274d95a741100e7f203/docker...
It's a few weeks old, but of those the one that came closest to our needs was Deis.
1) Downtime.
This is such a huge fallacy, if you think doing this yourself means no downtimes think again. If you had a top notch ops team you are still going to have outages in the middle of the business day. This team of ops guys you hire, guess what? They are not going to be deploying at 1am every day, that is not sustainable. They are going to deploy in the middle of the day, and guess what, they will make a mistake. And guess who notices outages first most times? Customers.
Heroku's uptime is typically in the 99.9%'s, I don't think you can do the same with one ops guy, maybe you will get lucky, however it is more likely that you won't.
5) Buildpacks
Build packs are an AMAZING abstraction. Just think about how you would solve the following without buildpacks:
5.1) Install libgeos on all the hosts running a specific application?
5.2) Create and scale new worker instances on demand?
5.3) How do you setup logging for all these servers?
5.4) Health checks for these services (tcp, http)
5.5) Alerting and automatic failover
* pssh? That will work fine until you spin up a second server and forget all the incatations to get your software up and running. * Puppet? I am sorry but Puppet, chef and their ilk are horrible, who knows what state any machine is ever in at any point in time. humans will inevitably forget to ensure absent, or make an assumption about state in puppet land. Nevermind the slow feedback loop in puppet land. Oh hey you want to change all the servers running X to now run Y instead, good luck with that. * Mesos? Mesos is the best open source scheduler out there, Marathon makes it easy to deploy to and there are scripts to run easily on ec2. However if you want to install native dependencies (libgeos etc) you will still need to do one of the above things. * Docker? Docker is an amazing way to package your app up and ship it to run on Mesos, however how do you package docker apps and ship them? Guess what? Buildpacks.
Now consider this on Heroku (5.1) Add the https://github.com/ddollar/heroku-buildpack-multi (5.2) Change your procfile (5.3) There are plugin's for that
Oh hey you can also write your own language, write a buildpack for it, and deploy it to production without ever having to get an ops person anywhere to setup a server for you. I am sorry but that is nothing short of amazing.
2 and 4)
I agree 100%, it sucks to feel powerless and have no insight as to when things will be resolved, however if you work in any decent size org ops things will go down, and you will also have to wait to see any resolution. I agree it feels better to see people working busily in your office because you can see the sense of urgency. I
3) Pricing:
In regards to pricing: yes Heroku is very expensive. However for small and medium teams it is much cheaper than hiring an ops person. You also get an economy of scale when you use Heroku; Think of all the things heroku takes care of for you: automatic failover on hardware/software(segfaults)/vm issues. Disks filling up on some host (it is embarrassing how many times I have seen something as simple as this cause issues). Agility; You need to switch MRI out for JRuby or rubinius, no problem, just update your ruby "version" in your Gemfile. The Heroku platform is superior in almost every way for running small to medium size applications then any in house cooked alternative.
I am very familiar with the alternatives out there, I have contributed to dokku's buildpack repo https://github.com/progrium/buildstep, and have patched docker.
There is no good behind the firewall/on my vm/hardware alternative for heroku atm. People are making progress but you won't really get anything as full featured as heroku.
Best of luck to you Brian.
I always know what's on my servers all the time; I build immutable servers and throw away the old. And I'm not particularly knowledgeable or skilled at this; I just use a little good sense. I mean, your problems with devops may actually exist, but this is silly.
At Localytics, adding a new Play 2.x (our current standard) app into our new deployment infrastructure takes one command and about 120 seconds. The only part for which a developer needs devops input is the first production deploy, which require a signoff and some (automated, but privileged) IAM permissioning. Adding a new app on a framework we haven't incorporated yet (Rails is probably next) would take a few days, but needs only be done once for all developers at the company. And we know it works with our entire ecosystem.
You might have had bad experiences in the past, but you're extrapolating from them to the state of the world. It's not what you think.
Especially here: > Puppet? I am sorry but Puppet, chef and their ilk are horrible, who knows what state any machine is ever in at any point in time. humans will inevitably forget to ensure absent, or make an assumption about state in puppet land.
You setup the new cluster, switch the load balancer, tear down the old cluster.
I suggest you look into blog posts like this [if this is a pain point for you]: http://blog.codeship.io/2013/09/12/the-codeship-workflow-par...
If you run at near-max load:
If nothing else you should have a staging environment and that can be your second cluster too.
We are talking about Heroku here, they are more expensive than EC2 so you can get away with this on similar budgets too. ;)
1.) Downtime - You seemed to gloss over my primary point. The downtime today could have been completely avoided if Heroku did not opt to do a 2pm EST scheduled maintenance. This was a very large risk they took and it blew up in everyone's face.
As far as resolving downtime. I stand by my original statement that a PaaS like Heroku cannot recover as fast on average as another solution. They have too many customers that they have to get up and running.
5) Buildpacks - I've been fielding this one on Twitter all day and am considering writing another blog post. I didn't really give much weight to this one and that opens it up to attack pretty easily. Here is my point:
Buildpacks in general do not suck. Heroku's implementation of Buildpacks sucks. Customizing Heroku buildpacks is a nightmare. Maintaining said fork is nearly impossible. There is no versioning of the buildpacks. There is no way to know if/when slugs are updated and my fork is no longer useable or needs to merge in upstream changes. Getting anything into the official buildpacks is a political nightmare. Adding CLI tools not in the heavily stripped down Ubuntu AMI is a gigantic pain in the ass.
But if that is not enough for you I'll end this point with David Dollar who did a ton of work while at Heroku with its buildpacks https://twitter.com/ddollar/status/481201978391801857
re: puppet - I'm not sure why you're hammering me for Puppet as I never mentioned it. You seemed to assume that this is what we will use.
3) Pricing - I dismiss most of your argument unless we're talking about Heroku's database hosting. Their DB hosting is awesome and top-notch. Their app hosting has become a less than stellar. I don't even know what I'm paying for anymore. To me managed devops means I don't ever have to deal with pager duty or I can just hand customers over to someone after we launch their product. That's not Heroku, not by a long shot. At best we're getting security fixes and system updates applied. At worst we are paying a premium on resold EC2 instances for a deploy shell script that has been freely available for a while now.
I don't know about you but I don't really spend that much time swapping the underlying language. At least not without some heavy consideration, what we go with is usually thought out from the start of the project. To me, I'm not interested in the least in being able to swap out MRI for JRuby with a change of a line. In nearly 10 years of professional Rails development I have maybe had to do this twice. Upgrading Ruby is a different story, but that is also not something that should be done lightly.
Today's post was a culmination of years of frustration with the platform. I've had it, we're done with it. Moving on. Good riddance.
For heroku it is tougher, they are the app that runs other apps, so bugs have a much larger impact. I would hope they have post mortem's and take actions to ensure this doesn't happen again. IE why wasn't this caught in a staging environment?
I agree that heroku totally fucked up, I would feel terrible if I was the engineer that fucked this up (as they should). However I think you are throwing the baby out with the bathwater.
I have seen outages last 1 hour + at big startups in the valley, it happens. I am not saying that it is right, I am saying that it is very difficult to do right, and very few companies are able to do it. It generally takes quite a few people to do it and far too often I have seen it come at the cost of being able to move quickly. Not excusing heroku for making these mistakes, I think it blows pretty hard. I just think there is no good alternative.
I believe that you are seriously underestimating the amount of effort required to deliver 99.9% reliability. I also believe you are underestimating how much heroku does for you on a git push if you think it is a bash script that has already been written. If that were true we would all have Heroku behind the firewall.
5. Heroku's implementation of buildpacks on their stack sucks? They have the only real implemenation of buildpacks? I agree forking/maintaining the fork sucks, I think there should be an easier way to install apt-get dependencies (ENV var).
Alternatively you can use: https://github.com/ddollar/heroku-buildpack-multi which supports versioning and chaining. You just make a buildpack for your custom bits, and install it before your ruby buildpack
I agree I don't often switch between ruby interpreters, however I do frequently upgrade MRI ruby versions, and the fact that it as easy as changing one line has saved me a lot of time. IMO this is much easier than any alternative I am aware of.
I have written a few buildpacks and I have found buildpack multi to be pretty composable.
I agree Dockerfiles are a decent alternative, however the part missing is a scheduling/discovery layer. Mesos docker integration is coming along via the folks at mesosphere but you still need to wire up the bits and pieces yourself.
3. I agree it is expensive, I think it is cheaper than a full time ops person to a pretty decent scale, and I think you need a lot more than 1 ops person to get close in terms of SLA, and featureset.
Honestly I think you just traded one set of problems for another. Who is going to get paged when both the vm's hosting a service die? Who is going to get paged when you run out of disk space? Co-location of apps? Hosting multiple ruby versions on all your hosts?
I don't fully appreciate your model but IMO it sounds like you have a client expectations problem.You should set the deliverable as a deployable app with the target as heroku, if they want to pay more for a VM they can pay more but they should understand the tradeoffs. If they want their own custom maintained servers or pay for you to maintain them, but it ain't free. It will cost you time and hopefully their money.
I don't swap language often if ever, however I do like to use more than one language depending on the problem I am solving.
I've worked for 2 previous startups doing this. US -> Australia -> UK makes a pretty good global coverage strategy. Once you can afford 3 people this can become a very effective way of ensuring 24 hour coverage.
Between all the countries in australasia and Europe 24 hour coverage is not that difficult. Not to mention rates for a lot of those countries are cheaper then FTE's in the US and if you bring them on as contractors, you still get to avoid having a business presence in those countries (for Tax and what have you).
How can you run 24/7 with 3 people ? In Australia that would mean you would have to be above the allowed 38 hour working week as it would be a 56 hour working week, overtime and possibly penalty rates would apply. Even if it is on call a lot of companies in Australia pay extra for that.
> Not to mention rates for a lot of those countries are cheaper then FTE's in the US
I do not know who you are employing in Australia but I would suggest that there is only about 5000 - 10000 USD p/a price difference for mid-level guys.
You wouldn't do 24x7 with 3 people, but you could do 24 hours x ~5.5 days with the rest on call. You could cover all Business hours within the week. Secondly you don't bring engineers on as employees (that requires all sorts of legal & Tax liability). You hire them as self employed contractors. This works out great for them (they can claim all sorts of things as tax write-offs for their "business") and you get to avoid dealing with the bureaucracy.
> do not know who you are employing in Australia but I would suggest that there is only about 5000 - 10000 USD p/a price difference for mid-level guys.
Avoid Sydney & Melbourne.
A lot of people will take the opportunity to do this kind of work for a decent wage ($65k in Brisbane, $55k in regional centres). I know people who do this work, move to Thailand and live in luxury on similar dollars. Brisbane is one of the Red Hat support centres so there are plenty of engineers moving on from lowish salaries offered.
Not to mention once US interest rates start going up the AUD is set to start dropping again.
Thanks, I hope I didn't sound too harsh as I just wanted to understand how you were doing this.
> You hire them as self employed contractors.
If you are an Australian company you may want to be careful with this. A contractor is not necessarily a contractor in Australia, there are some strange guidelines as to when a contractor may actually become an employee.
> A lot of people will take the opportunity to do this kind of work for a decent wage ($65k in Brisbane, $55k in regional centres).
I worked in Brisbane for 12 years and that sounds about right. Although the need for Linux admins seems to be increasing so there is some interesting salaries being offered.
> I know people who do this work, move to Thailand and live in luxury on similar dollars.
One can only dream, although I now live and travel around Asia .... I am just missing the $$$ :)
Well actually I am not doing it anymore. It turns out that the when the GFC hit the startup I was working for went under. So did a lot of places and I had trouble finding work. Now I have more family commitments I am a working peon for a small digital marketing company.... but there are days I wish I was back doing that kind of work again.
> If you are an Australian company you may want to be careful with this. A contractor is not necessarily a contractor in Australia, there are some strange guidelines as to when a contractor may actually become an employee.
Yeah this really only works if the contractor is working for a non-australian company. Then they are liable for their own super / tax / etc. Its a bit of a hassle, but a good tax accountant will get it sorted and reduce your liability at the same time. Contractor gets to write off their home (where they do the work), their tech, and more. Government hasn't really cracked down on it, because its an export business and your helping the economy.
>I worked in Brisbane for 12 years and that sounds about right. Although the need for Linux admins seems to be increasing so there is some interesting salaries being offered.
Unfortunately not as much as I would like... seems between a lot of "cloud first" outsourcing and the fact that linux / unix roles tend to be in HQ (i.e. in Sydney or Melbourne for aussie companies) the linux roles tend to be limited.
On the other hand PHP/Web/.Net developers with linux knowledge is in demand. (Quite strange the .Net developer with linux server experience, but just check seek.com.au and you will find them).
Brisbane is and it appears always will be a MS town. 80% of systems roles are Windows/Cisco/VMware focused. Smaller guys tend to be more Linux / Mac OS based, but then you end up with hybrid roles where your also 15 other roles.
It's not that complicated... And it is far cheaper !
This is the bit which confuses me. It is easy to think that there should be correlation between Heroku pricing and AWS pricing.
However just because AWS drops their pricing does not mean the capex for Heroku is dropping.
Very interesting point