josephruscio

I'm finding a lot of these articles about "surviving" the outage fairly frustrating. They generally boil down to a combination of the following:

1. "We use a multi-AZ strategy!" - This outage affected multiple AZ's concurrently. If you did not see downtime, this means you were fortunate to have at least one unaffected AZ. This is pure luck however, many sites with the same level of preparation had significant downtime. (Note: A multi-AZ strategy is sage and would have minimized your downtime, but does not warrant a survival claim in this case.)

2. "We aren't using EBS!" - Not a single article I've seen has claimed that they weren't using EBS because they feared a multi-day/multi-AZ outage. They weren't using it because it lacks predictable I/O performance in comparison to S3. You can't retroactively claim wisdom in the category of availability for this choice.

3. "We don't host component <X> on AWS!" - Taking this argument to it's logical end, any service that doesn't host on AWS could write one of these articles e.g. "We host on Rackspace so we didn't go down!"

In short, if you don't have a completely multi-region strategy (including your relational data-store) implemented purely on AWS, your blog post is decreasing the signal-to-noise ratio on this issue.

I should clarify, if your title was "How we minimized the impact of the cloud-pocalypse", then the content of the article is fine. I think most SaaS operators would define "surviving" as no downtime e.g. Netflix.

This "cloud-pocalypse" simultaneously affected multiple availability zones in US-East. Had you had the misfortune to have your multi-AZ pair in two affected zones you would have had significant downtime. The only architectures that were truly safe from this outage were those with a completely multi-region strategy, and I suspect those are very far and few between.

This is a really good introduction to the different levels of infrastructure monitoring and their various pros/cons.

http://librato.com (full disclosure: I hack there) is a startup with a new kind of entry in the process-level monitoring/management space. Would love any feedback the infrastructure-minded part of the community here might have.

http://librato.com is another hosted product (disclosure: I work on this) for systematically monitoring/managing applications.

Senior RoR/Sinatra Dev - San Francisco

http://librato.com is a small, venture-backed company delivering application monitoring and workload management as a service. We've got a great team, high profile initial customers, and a new office in San Francisco. Strongly prefer local candidates. If your interested shoot me your resume to joe _@_ librato.com.

I appreciate the author acknowledging some of the hurdles and warning signs that could portend against OpenStack's viability. There are currently an impressive array of organizations pledging support, but talk is cheap. (Very cool that patches are flowing in from some of them already.)

I think the most important signal is another serious provider stepping up alongside of Rackspace with a compatible OpenStack offering. Until then, this is just Rackspace's cloud implementation. Albeit a completely open one that you could in theory replicate on your own hardware, but I imagine that option would be small comfort to most people using IaaS clouds.

Still, it's awesome that they're trying to make it happen, and I'll be rooting for them.

GoF wrote about the Strategy Pattern (and the rest of them) in 1994, so I'm pretty sure Java (first public release in 1995) was the least of their concerns at the time. And believe it or not, there are times when languages with closures are not an option.

The "Conditional Inherited Modules" seems to be one (of many) possible implementations of the venerable Strategy pattern from GoF (http://en.wikipedia.org/wiki/Strategy_pattern). It's a really elegant and one of my favorites. Just define a clean interface and pick between different implementations at runtime.

There is absolutely no moral or ethical dilemma in "jingle-mail". These banks handed out huge no-money down loans like candy to people who bought way more house (and in some case multiple houses) than they could legitimately afford. Both parties were complicit in a transaction that should never have occurred. Now that the inevitable has occurred and the music has stopped, the only thing that matters is the letter of the law regarding the contract. If the bank proposed/agreed to a no-recourse loan, just walk away.

Morality aside, jingle-mail is also the healthiest solution from a market perspective. The faster banks are forced to take their lumps, move these foreclosures through the market, and return lending standards to their historical norms (e.g. 20% down), the sooner the housing market can return to normalcy.

The exclusivity deal was arranged when AT&T held most of the leverage. Steve Jobs shopped around the different carriers to get the iPhone off the ground and with the exception of AT&T, they all balked at the terms he was seeking. AT&T was the only one willing to play ball, and as such could insist on the lengthy exclusiveness.

-Wall unfortunately leaves a lot of warnings disabled. We've used:

"-W -Wall -Wcast-align -Wstrict-prototypes -Wmissing -prototypes -Wpointer-arith -Wshadow -Wsign-compare -Wformat=2 -Wno-format-y2k -Wimplicit -Wmissing-braces -Wnested-externs -Wparentheses -Wtrigraphs"

Update: Forgot the most important one ... -Werror. Making warnings equivalent to errors is the only way to ensure that they're always taken care of.

Your compiler may offer a more formal way to indicate that it's acceptable for a parameter to be unused. In gcc it's an attribute:

int foobar(int foo, int bar __attribute__((unused))) { /* Code that does not use bar */}

We're using it in production in service we just launched. "Performance" is a relative term. We measured performance in:

1. Minimal amount of time to setup (minutes) 2. Minimal amount of time to maintain (none) 3. Minimal amount of time to make redundant (none, see Multi-AZ support just announced)

By the time the segment of our data we're keeping in RDS exceeds the capacity of the largest instance size AWS offers ... it'll be a very good problem to have.

IB is typically deployed in a Fat Tree network: http://en.wikipedia.org/wiki/Fat_tree

It's been used in high-performance compute clusters with thousands of individual nodes.

I highly recommend using fresh pasta if you have access to it. In the US most grocery stores carry it in one of their refrigerated aisles. It's more expensive but the texture is night and day. Be sure to read the instructions as you need only boil if for a couple of minutes.

Librato is looking for a lead designer to own UI/UX on http://silverline.librato.com

Brick/Mortar office is in Santa Clara, CA, but we offer part-time telecommuting e.g. I work from home in San Francisco 3 days a week.

It's somewhat annoying when articles of the format "Part N on this topic" do not begin with links to parts 1 -> N-1.

My point wvenable, is that we are absolutely not dealing with two different kinds of screw drivers here.

Yes they both store "data", but there are types of data that one is well suited to deal with, and alternative types of data that the other one is better for. On my team we've already split out data into two sets, one that lives in ACID RDBMS and one that lives in NoSQL.

If you are dead set in using one solution for everything, my pity for you.

Just because you can pound a screw into the wall with a hammer doesn't mean it solves the same problem as a screwdriver.

Scott Chacon has a couple excellent resources: http://progit.org/book/ and http://whygitisbetterthanx.com/

Distributed Replicated Block Device (http://www.drbd.org/) can be used in conjunction with something like Linux HA (http://www.linux-ha.org/) to create a systematic high availability clustering approach for stateful (and transactional) applications such as a database. Lots of sites use it (e.g. github.com IIRC) and up until now, you've had to run a patched kernel to take advantage of it.

Perl's Pod (Plain Old Documentation) is actually a pretty humane format that exports to manpage, HTML, text, etc: http://perldoc.perl.org/perlpod.html

We did all of our command line tool man pages in it, and it only took a few minutes to get up to speed on the syntax.

I couldn't agree more with DHH here. Although addiction to "work's highs" is one explanation, I think its often simply a noble-sounding excuse to not expend the effort to exercise regularly or maintain a significant other. Hacking is fun, working out and maintaining relationships are hard. I fell into this trap when I was an undergraduate, sleeping 4 hours a night and putting on a significant amount of weight. After I finally got my priorities straight a few years ago, I lost the weight, started sleeping 8 hours a night, and my quality of life has improved dramatically.

There are always going to be periodic crunch times when you need to pull out all the stops to hit a deadline. However, if this is the norm rather than the exception, I think you're doing it wrong. Note that I still agree that you shouldn't be expecting to take vacations and enjoy lazy weekends, its just that 8 hours of sleep a night and regular exercise should be non-negotiable.

Not trying to troll or anything, but in the age of services like EC2, App Engine, and Heroku, why would anyone get locked into a long term hosting contract without prior proof of "legs"?

So there is a level of dilution doesn't make sense, but I don't know that a good VC would proffer them. Does a VC no good to close a round and have the whole team walk away. I think this article is talking about situations where dilution is painful, but within the realm of reason.

Assuming even a small/moderately sized team, i.e. other people than just the founders are also employed, a good VC will grant new options to employees. The founders get hurt the most. If the VC's want to keep the founders around, then they need to ensure they're properly incentivized as well.

In some cases, you can get the deal done to take care of your people. Then unwind yourself and move on.

It's a concern, but any amount dilution is better than infinite dilution (i.e. turning off the lights).

"Google manages its kernel code with Perforce."

Did not see that one coming.

Agree that most people who skip CD are doing for lack of trying ;-). I think the ease of gaining access depends largely on the target customer profile. Consumer, even SMB targeted stuff by definition has a large number of potential customers compared to the Enterprise. Additionally their barrier to purchase is much lower than the Enterprise, therefore the signal from your 30 minute cup of coffee is much stronger. It's a trickier proposition with Enterprise customers, but I guess that's why the deals are bigger.

Missed a 4th reason for a lack of customer development: 4) Hard to get meaningful face time with customers and/or get honest, well-thought out responses from them, particularly hard for a brand-new startup without existing customers on existing products.

Customers are very busy people too, and they are not on your payroll. Totally agree that you should make your decisions as informed as possible, i.e. do as much Customer Development as you can, but ultimately a big part of a early-stage startup is making decisions without the luxury of a complete data set i.e. gut and luck.

Which is precisely why you need to "ship early" (get paying customers ASAP) and "ship often" (start developing them immediately and incorporating their feedback into development).