Heroku is down again
status.heroku.com
status.heroku.com
According to this document [1]:
"The North Carolina Utilities Commission studied the cost of placing Duke Power’s distribution facilities underground and found it would cost more than $41 billion, resulting in a 125 percent increase in customer rates."
[1] http://www.sceg.com/NR/rdonlyres/465E6534-2FFB-4069-BF84-814...,
I've also never heard stories about issues with rats.
Power outages still happen, but they are quite rare - in 30 years I can only remember twoish.
Apparently it's the insulation on the wires that they like.
Like it's gonna be some unprotected plastic cables 1 foot under the ground?
All it means is that humans are not yet powerful enough to make the environment work as it should (ie serve humans).
http://www.washingtonpost.com/blogs/capital-weather-gang/pos...
8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone.
8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. We are actively working to restore power.
e.g. because region-to-region data transfer is not free, and trans-region latency is ugly, you can't just relaunch half your instance farm in another region and expect happiness. There are also routing issues: Internal IPs don't work across regions, elastic IPs don't transfer across regions...
Extremely nerve wracking for new startups like ours.
But anyone deploying a critical application to AWS makes a point of cross-region data replication. Heroku have long known that they lose potential customers to, say, Engine Yard as a result of only hosting at US-East.
One can only conclude that this is a clear business decision on their part. I can hardly believe that Heroku's engineers are incapable of it. Indeed I would be very surprised to learn that they haven't brought up an instance of their platform at, say, US-West, for testing or proof-of-concept purposes.
Of course, productising that is a different matter. Extending the control plane, front end, and pricing/billing systems might have considerable associated project cost. Perhaps they have concluded that the costs outweigh the additional revenue. Or, just haven't got around to it yet.
> uninformative titles left intact
hn 2012
I've learned my lesson.
It's underhanded to call it a "performance issue," if not an outright lie, albeit a small one.
(fyi: our customers are still being alerted)
Their status site is running fine altho it's not reporting errors: https://status.heroku.com/
Their Helpdesk is down: https://api.heroku.com/helpdesk/login?timestamp=1341025835&#...
Devcenter is down: http://devcenter.heroku.com
AWS isn't reporting any errors: http://status.aws.amazon.com
8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region.
However, I have 20 instances on us-east. And haven't seen any problems, even during yesterday's outage on AWS.
Edit: that doesn't mean this isn't an AWS outage.... It almost certainly is.
Rackspace (and many, many other co's) tend to have functional UPS units & generators. Amazon tends to choose the cheapest datacenter facility imaginable and then these sort of failures occur.
Given their size they'll inevitably fix the power issues though -- they've got the finances & they're capable to add a few levels of redundancy.
Looks like Amazon wasn't the only one with inadequate testing of their continuity plan. And I don't think Rackspace offered alternate Availability Zones at that point.
I think Netflix are expecting another cloud to offer the same model and API as Amazon though, which isn't likely to happen - everyone else is learning from AWS's mistakes!
Even if it did, many of the features they're waiting for (like auto-scaling groups) probably wouldn't be as useful in a multi-cloud environment, and would therefore have to be built into Asgard.
- One AZ is down
- API commands are spotty and may return incorrect results
- ELB looks screwed
- IP reassignments don't seem to be working
- Who knows what the fuck else is broken(fandalism is down)
Beginning to feel pretty lucky though -- this is at least the 4th AWS-East outage that has made enough of a splash to notice but missed my instances. Upgrading to multiple availability zones was scheduled for Monday anyway.
Had to move off Heroku for my latest app. That amount of downtime would put me out of business.
To bad. I really like the Heroku platform
They rely ONLY on virginia's instances because its the cheapest, without caring about customers.. or thinking of replicating their services in multiple locations for such issues!