Downtime
staff.tumblr.com
staff.tumblr.com
Not only is 37signals refunding everyone for the month, DHH is replying to pretty much everyone personally to tell them so.
http://productblog.37signals.com/products/2010/12/explaining...
though, "We didn't earn your money with this service." is a little harsh. actually, lots of it seems harsh. cheer up dhh, we still like you (and i even got a few 500s i think).
Experience with LiveJournal, Friendster, Twitter, etc. has been that problems don't just magically fix themselves; absent someone with enough vision to figure out potential problems and actually solve them in technology and business process, you're kind of fucked. In the case of Twitter, they had enough money to buy some excellent talent. In the case of LJ, Brad stepped up as an amazing hacker (with limited financial resources, and really one of the first to solve the problem). In the case of Friendster, bad executive leadership (CEO/board being ineffectual, VP Eng being a tool) killed them.
They have banking people (who use a largely entirely different technology stack, budget, and way of working), and people who have worked for smaller companies, but they don't have a huge number of veterans of large consumer webapps. Google's NYC office probably is more of a sink than a source for ops talent, and there isn't a large presence of declining big giants to strip for engineers like there is in Silicon Valley.
Do people move from SF/SV to NYC to work for established companies? I could see this if you had personal reasons for wanting to move (being from NYC and going back, or just wanting to go there), but it doesn't seem like a compelling option otherwise.
Still rooting for Tumblr like crazy, but bummed out :(
* I make many comparisons to Twitter here. They are extremely different. But they are both social and function like places, so I reference Twitter to make it easy for outsiders to grok.
First off, Tumblr is not primarily a blog. It's what we call a "tumblelog", which is a stupid word; I like to call it "vomit" or "spew". The fact that it's the sleekest blog platform out there is secondary to its larger function as a spewing platform.
Blogs are for creating. Spew is for recycling, for breaking things down into small teeny pieces, for streams and streams of things which are essentially meaningless but contribute to a larger whole. Now, not all blogs are strictly bloglike either; the Linked List format is somewhat in between the two, though Linked Lists are usually more disciplined in nature. Another way to draw the distinction is to say that blogs are for building things, while tumblelogs are for shaping them. Or you could call blogs classical music and tumblelogs jazz. One is looser, more freeform, more about the movement than about the individual notes. The other is about the finished product, or about realizing a vision.
A defining characteristic of a true geek is that he builds. Doesn't matter what he builds; what matters is that he appreciates structures. So when we look at a blogging platform, our needs traditionally tend to be focused on constructing more elaborate things. They also tend to be relatively solitary in nature. (Wordpress is; I never had a friend that used Posterous and so I don't know how they handle following things.)
I like slagging on Posterous because I'm still bothered that they get compared to Tumblr when they're really entirely different. Tumblr's breakthrough was its deconstruction of the blog format. You could post an image without a title. You could post a quote without something to frame it. You could post ANYTHING without a datestamp, or a "posted by" attribute, because nobody cares about them, they just care about the flow of content. Posterous has lines of datestamping, and they don't handle title-less posts. They're all about traditional title-body posting. They're all about "blogs", about these elaborate constructions. They aren't broken down like Tumblr.
Tumblr's a fucking awesome engine because it removes everything bloggy about blogs. It's designed for a steady vomitstream of thoughts and ideas. Not your thoughts or ideas. Anybody's. And it strips away everything we associate with blogs, and with everything it strips away it becomes sleeker, lither, more powerful. No comments means no conversations other than reblogs — and reblogs are great because first off, they let you improvise off anybody else's stream, and second off, they make it IMPOSSIBLE to participate without being a "primary creater" with a flow that other people are following, versus blog comments where every commenter is attached to one site at a time.
That means it attracts people who don't build things. People who just want to push out content without worrying about being judged for value. But they want to push it out, because it is a creative and cathartic act just to release these ideas out into the world, or to change their flow. Lower barrier to entry, more ability to interact meaningfully. You can participate without any skill but still find that people are interested in you.
Now, I use Tumblr primarily as a building tool. I find that its theming system makes it very easy for me to design custom interfaces for complex publishing sites, and yet still push through the entire site as a streaming feed to any Tumblr user who wants it. And it's used by lots of serious designers who appreciate its versatility. But I'm an edge user. Every HN user here who uses Tumblr is an edge users. The real users don't post here, because they're not out to build things, they're out to just express themselves loudly and with fury.
Without more information, this can either make people trust more tumbler in the future (we went from 3 to 12 engineers) or less (we went from 200 to 800). I'd worry in the later case, because it might show that Tumblr is not aware of Brook's law (http://en.wikipedia.org/wiki/Brooks%27s_law), which is always unreassuring for me.
How do you create and keep a good culture when you're growing that fast? Anyone have experience with this sort of scenario?
I think in this case the point is to reassure people that they are taking it seriously.
People would argue that places like facebook and/or google might have retained their culture etc etc. I doubt that there was a point when someone grew the ENTIRE engineering team 4x in a 30 day span at those companies (unless it was from small number to small still small number).
That number is a bit arbitrary, but makes sense from what I've seen and experienced at other companies. A 4x number is is hard to overcome when you have more than 4-8 core original engineers. Think about what that means. If you start with 5 devs, 30 days later you have 20? Are they seniors that can live on their own (where did you find 15 senior people in 30 days?)? Are the original people all leads who are now responsible for shepparding these new devs onward into the company (who is doing the real work for the first couple of weeks these people are coming upto speed...5 people generally don't have fully automated systems, builds, deployments, automated tests....)
Imagine if this number was 8->32 in 30 days? 10->40? 20-> 80? 100->400?
You have to lose something. I think what you lose becomes the key thing that defines what your culture is going forward...this would look to be one of those moments in a company when someone really, really needs to decide what is important to them. This is when the REAL culture will be defined for the company.
Move the blog/status site outside your network (linode.com)
Work on a process to try and follow if you have another outage.
- One person to handle communication (blog post / respond to users)
Work on a faster way to recover from such a failure.
Maybe have a read only version you can switch to "maintenance mode" ?Done:
Probably the biggest outage you'll face.
20+ outages don't usually happen.
Learning from it. . .
We've all had too many examples of bad pr-attracting events and their fallout over the years. A wise man wrote 'One shouldn't be forming a(n incident-response) team during a crisis', and I don't expect different here. Do I hope for better as someone who uses and admires the Tumblr platform, and expect more communication next time? Yes.
The quantcast graph pointing to the 1 _billion_ increase over the past _two_months_? Easy to play armchair qb to that, but by jeebus, it's almost surprising they made it this long.
Of course, once you have enough 20h outages, you dont have to worry about the problem anymore.
Thats not to say its a good excuse, and you certainly lern from it - but it is understandable.
I understand this is aimed at all users, but I’m still disapointed.
On a side note - anyone know what DB they are using? The cynic in me is thinking "Hey, another MongoDB + FourSquare 'success' story of webscale awesomeness."