Kevin Rose on the sources of Digg's problems
dl.dropbox.com
dl.dropbox.com
It seems Kevin wasn't blaming any problems on his team, but rather on their lack of attention in selecting that team. Whether he's able to select better will soon be seen, but I take his comment as more of a mea culpa than anything.
There was no such thing as a performance evaluation.
The founder and CEO mostly worked on side projects or spent time "taking money off the table."
There were lots of parties and social events.
They could have evaluated mediocre developers, put them on performance improvement plans, and if they failed to improve, fire them. They were too busy.
In the podcast, I didn't hear any mention of what I'm talking about. I heard "we hired too many B and C people."
Instead of "we need to scale to a 100 million,", there should have been incremental improvement.
"we need to serve the next 5% (growth) better than the last 5%." To do that we have to be a better company (incrementally).
And repeat.
I think while finding big successes can be hard, dealing with them once you have them can be even harder.
Kevin was pretty quick to realize that Twitter/Facebook were going to eat Digg's chances of become a huge company, but it wasn't trivial to figure a response to that. It's a valid question: should Digg have even tried to grow grow grow or just been happy owning their area?
Engineering gripes I think were mostly focused on acquisition and investment attractiveness. There probably is also some elitism in the Valley: if you don't have the right pedigree and look the way that fits what people expect you don't deserve success
Then why Digg´s main competitor Reddit.com is growing like crazy these days ?
its because Digg was managed badly ..
The ship is sinking really fast, after all...
He was simply telling it as he saw it.
They hired people that turned out to be not as good as he would have liked, and that was a major issue that cascaded.
I don't see anything wrong with that.
If at any point you find that they are significantly underperforming, go and find out why. E.g. maybe for the first 3 weeks they were on database stuff, which they love, but they've been reassigned to UI which they hate. Maybe their dog died.
If they underperform at any two of the reviews stages, get rid of them. If they are bad for morale, get rid of them. A great coder that is an energy vampire for your good coders is a net loss.
One place I worked we had a guy who was supposed to be a good programmer. The first week there he was really depressed about Michael Jackson's death. I tried to cheer him up and he perked up for a couple of days in the second week. Then in the third week I discovered he was actively looking for work elsewhere (and telling other devs about it). I pulled him aside and told him to pull his head in. He came right for a while but the next week he was down again. He broke the build every second day, and nobody ever chastised him for it (there might have been some playful ribbing along the lines of 'you broke it, you bought it') but the one time I broke the build he lost his shit and started screaming at me, telling me (in front of managers) that I was incompetent. He had epic mood swings, when he was up he could be the most charming guy in the world, when he was down he was evil up to and including physically attacking and bullying other devs (but never in front of the managers).
Naturally, the managers loved him, so I left.
Performance reviews are horrible, and consist of all sorts of minefields and HR BS. E.g. rate yourself from 1 to 5 in these areas, but nobody gets a 5.
When I said mentally review their performance, I meant mentally ask yourself if the person is up to scratch. E.g. is their actual performance good. Now, most managers wouldn't be able to tell you this, so this shouldn't be something that is done by a manager. It should be done by a senior dev who has some kind of knowledge of the area the 'intern' is working in.
Alternately, you could have a tribal council of say three senior devs, and they vote at the 2w,4w,2m,3m periods. Anybody that gets too many downvotes gets to leave the island :D
But note also that if somebody is temporarily under-performing then you need to find out why and fix it (Note: not the same as 'fix them').
The goal is to as fast as possible identify the people who are consistently underperforming, or show enough signs of flakiness, so that you can utilise the built in 'thank you for playing, you are the weakest link, goodbye' clause built in to most contracts.
However, consider the reverse, is there anybody who is consistently good at picking good developers? Is there any interview process that consistently works? If so, I am not aware of it. Hence I don't consider the hiring process to be 'solved'.
I can see the reasons behind Joel Spolsky's "make sure they write code during the interview" mantra, but you look at some of the other stuff he wants and it's all pointless pointer twiddling. Then he went and broke all the rules anyway by working with Jeff Atwood who by his own admission wouldn't know a C pointer if it bit him in the face.
It makes me wonder, how do they do it in other professions? E.g. is there a process by which you can weed out the good Civil Engineers from the bad ones? Or good Lawyers (leaving aside the obvious and droll joke), or good Doctors? What about a people oriented profession like Priest? Surely they would be able to weed out the good from the bad.
But no, it turns out that for Doctors and Priests we're always getting bad ones in the news (Doctor's lying about their qualifications, practicising without a licence etc).
Okay then, the 'fluffy' professions haven't got this sussed, what about the technical ones?
Yes, this the process of licensing. In those professions you mentioned, all people who (legally) work in the field must undergo a process of training and subsequent certification. This includes both work experience and exam requirements over the course of several years prior to certification. Yes, there are people that cheat the system, but the licensing exists to weed them out and protect the consumer.
Software "engineers" undergo no such process or requirements. I can learn ruby in my basement in a year, call myself a software engineer, and get hired building any number of apps. Is my work critical enough that someone may die if I am not certified? Usually not, so the need for certification isn't as strong. Nor have we had the time to develop such continuously relevant certifications in a field that changes several times a year.
I recall a story about a flight instructor who said to a freshly licensed pilot something like, "Now you've got your license to go out and really learn how to fly".
The point is that true competence and most certainly brilliance requires individually driven effort and expertise that is unlikely ever to be measured by government or professional organization administered exams.
Put it another way: Do you feel that licensing assures us that all drivers on the road are "good"?
(Sorry if my argument sounds abrasive; the subject's a pet peeve of mine.)
And getting a pilot's license is a little bit different from being licensed to practice medicine. It also depends on whether you're referring to a private or commercial pilot. Private pilots only need 40 hours[1] of time in the air. And even once a commercial pilot is licensed, there are other regulations in place. And a driver's license is an even lower bar, of course.
So no, licensing (in any form) does not ensure that the licensee is "good" or "brilliant," but it does allow us to ensure a minimum bar that does not currently exist in the software industry today.
(And no worries, it's a pet peeve of mine as well. At least at it relates to the software industry.)
1. http://en.wikipedia.org/wiki/Pilot_certification_in_the_Unit...
The historical pattern in the US seems to be that professional licensing, introduced as protection from incompetency, inevitably evolves into a way of limiting competition in a field by restricting the number of practitioners. At least on architectural exams, to the best of my knowledge, the maximum number of those who "pass" each year is set in advance.
It's a slippery slope once you start regulating entry into a field. The regulating body comes under political pressure from anyone with a financial stake, especially practitioners, educators, and major consumers. I'd wager that soon after you need a license to call yourself a software engineer, you'll need an accredited degree and years of internships before you can sit for the licensing exam. It's played out in similar fashion in the other professions.
There are other ways of ensuring competency without the use of government or quasi-government institutions. In software development, incompetency generally wastes time and money, not lives. With stakes this low, perhaps something as simple as a standardized testing service that could be used by HR departments to screen applicants would be viable... (Please pardon my designer's ignorance if something like this already exists in the development world :)
I know people who themselves cannot program but know what the hell loops are and stuff that you learn when you open a basic programming books. The stuff that tells you what the hell complexity of a program. If I were to tell them write a simple for loop they would not end up doing it, but when the programmers tell them that some task is taking time, they break down each task so logically that they would make a damn great programmer themselves.
Every reason is analysed and possible directions put infront of the developers. If they say the servers we are using are not fast enough, they are quick to ask back, if it is the code or the servers themselves, since they know of people who are doing just fine using the same servers.
Another great thing in a company is that there will always be smarter developers who will do the same task effectively and faster. Then of course it becomes quite evident if the other guy was at fault or had a genuine issue.
Why then is it so hard to view managing people in the same light?
Btw, there is a giant flaw in your logic. You assumed I couldn't judge a programmer, but you then assumed I would successfully judge a technical cofounder.
Even more so, if you end up being an acquisition target for companies like Google and Yahoo (who want to read your code), then it becomes "this will cost us hundreds of millions of dollars this week if its not fixed now."
Regarding technical debt, I think we're conflating a few things under this umbrella. There can be code that is easy to read and maintain, but won't scale to millions of users. Then there is spaghetti code which is brittle, hard to read and modify, but may or may not scale. I think it's ok to have the first before you know if you need to scale.
When you guys got to the point that you realized it needed to scale was it fixed immediately? If not, would it have been that hard to fix it then.
Btw, thank you for talking about this stuff. It's great to hear insights from someone that's been through such things.
As the saying goes "Always assume that the next guy to maintain your code is a psychopath who knows your address."
That is not something I knew in the early days of digg. I know it now.
The only people who know who are the good devs are the other devs who have worked with those devs. Reputation outside of the dev circle is largely orthogonal to dev ability, and rather closely tied to social skills e.g. schmoosing the managers.
I was talking to an older guy the other week, he didn't seem particularly "switched on". Turns out he makes well over twice what I do, because he predates COBOL, the people that he works for a terrified he'll leave/get hit by a bus etc and nobody will have the skills and knowledge to maintain the systems their business runs on, because all his compatriots got wiped out by an asteroid.
I wouldn't say he was bad at his job. If I was to guess I'd say he was 'okay' at his job, but he didn't seem to have that spark that I associate with greatness. But as his take home packet proves, sometimes 'good enough' is great too :D
Also really shitty programmers, when confronted with better ones, often do lots of crappy shit to sabotage the process and cause infighting. I did little if any of that (probably because I had significant equity). I stayed in the background, learned as much as I could, and contributed as much as I could.
I don't think Kevin learned anything there.
If you're here, you've probably heard that Ycombinator favors determination over talent. I kept the site up and made things work with extremely limited resources.
Sadly I can't see this going well. Reddit generally makes a big joke of how much Digg [supposedly] sucks. I don't think it would be generally a fulfilling experience for the AMA subject or most onlookers. YMMV.
So with that, I say Owen is a good coder.
Kevin never understood tech. As Owen says, he's just a guy who looks good on TV. Now he's throwing all his good coders under the bus because he fucked Digg up with his grand V4 vision, and he's trying to point the blame at others besides himself.
"It's a low move, Kevin. Low move. We supported you. We took the blame when we fucked up. We never pointed the finger at you. So take the blame when you fuck up. Asshole."
Sincerely, The guy who kept DBs running for Digg for 4 years.
[1] http://www.deserettechnology.com/journal/reddit-the-open-sou...
There is a diagram explaining this in the mythical man month, but I don't have my copy handy to give a reference.
I watched a documentary a few weeks ago (damned if I can remember what it's called) about Netscape's transition to Mozilla. The amount of effort that went into that transition was staggering. For his part, Jedberg said [1] that preparing and maintaing redditOSS will likely be one of the first projects for their new hires, as it will make development and maintenance on their current codebase that much easier.
That documentry also reaffirmed my commitment to not do pure sitting-at-desk software development. Because my god netscape looked like a horrible place to be, no matter how many smart people you had around you. (These days facebook looks similar, but worse not even cubicles for privacy)
You can expand inwards and look at how to improve your solutions to existing problems, or you can expand outwards and solve new and harder problems. I highly recommend the former when you're just starting out, because it builds your basic skills and gives you a solid foundation. But eventually you have to stop polishing turds and start hunting for gemstones. There's a limit to what you can accomplish if you solve the same problems over and over again.
I wouldn't dare make excuses for a developer sabotaging work, but there are some bad situations many developers can be put into which are more likely to produce lazy and shoddy output.
Revision3
Pownce
Wefollow
Blaming this on the B&C level programming talent is a bullshit way to deflect mistakes he made.
A few things I've learned over the years:
- Realize you're only as good as the people you hire.
- As a non-coding founder, it's important you perform deep reference checks and have a trusted senior developer review code samples prior to hiring. (We rewrote ALL of our early code base because we didn't take this step)
- Make time to interview every potential hire and vet them with proven experts.
- Ensure employees are a good cultural fit.
- If someone is doing a poor job, step-up and fire them.
- Look for employees that are up for the technical challenges. I remember several occasions when the answer was "MySQL can't do that" vs "we can do that, just not in MySQL".
Of course, that's over-simplifying things. Sometimes you asked for things that were physically impossible. When you need 1Bn operations to complete a task, you can't do that with 1M worth of resources, no matter if you try to use MySQL, Cassandra, or text files to get that result. You see, some of us at Digg actually had real computer science degrees, where we learned algorithm worst-case complexity, and we understood you were asking too much of too few computers. And if you think back hard enough, you'll remember the answer was actually: "MySQL can't do that unless you get 10x more computers."
Thanks for talking about your dissatisfaction with the theoretical limits of computing here, in this forum, where I had to find out about it third-hand, instead of actually talking to me face-to-face back when it could have made a difference.
Signed lovingly, Your MySQL DBA from Digg.
That said, once the company grows, there are places where a B or C player in one role could be an A player in another role, and vice versa. I think specifically about people who are promoted to their level of incompetence based on excellent performance and become miserable managers who make everyone around them miserable as well.
The important thing in that situation is to respond decisively and either support the person and train them to be competent, or respectfully return them to the level where they can excel.
I figure I'm probably a "C" coder, I get stuff done, and I'm always improving, but I know there are better coders out there than me.
Does management take any responsibility in driving C's to be A's??
Apparently not from Rose's point of view. What does HN think?
A "B" coder is one who for the most part writes good code, but doesn't design a good long-term architecture, or makes many small mistakes that add up in the long run, or leaves the code riddled with small bugs that eventually cause frequent problems. The "B" coders are the dangerous ones because really you should fire them or fix them, but it's too easy to ignore them.
An "A" coder writes solid code and is dependable, but doesn't hit the high notes, as Joel Spolsky says.
A star coder is an "A" coder who hits the high notes.
(I don't know what a "D" coder is, but perhaps our scales are offset by one letter grade.)
ooowww I've seen some of that.
An "A" coder's code is simple to spot: It's boring. There's nothing heroic or clever (that's hard to debug) it's well documented so even junior devs can understand it, and thought has been taken so that it can be extended in the future. To put it simply it shows taste*. That is a rare thing indeed.
Sometimes you can't avoid hiring B players. But if you do (either by mistake or because you explicitly decided to) you need to be disciplined about either turning them in a A players or ensuring they are not responsible for hiring. If you find you have C or D players get them to move on IMMEDIATELY.
Kevin says something here that is really important, and I think it is part of the lesson he's learned: "It was really hard to get rid of the C & D guys".
This is why I now try to work on the principle that I am far better off not hiring someone than compromising on quality. I have learned the hard way (multiple times) if I don't live by this principle, I WILL pay later.
B. I completely agree with the idea of making B players into A players. It's called "management" or "employee development." It did not exist at digg.
"You just fire them" is the right ATTITUDE, but in practice it's rarely so simple.
Related to all this is another lesson I've learned: In almost all cases when it comes to removing people from jobs, or re-organizing teams, it should be done quickly and decisively. Rip the band-aid! As a leader, you are far better off dealing with a mess that exists ON YOUR TERMs than one that exists on other's terms.
This, of course, assumes you have a set of core principles you both believe in and live by. But that's a whole 'nother diatribe.
* Do evaluations
* If someone isn't performing, start a process to make them improve. Usually called a PIP. Manage them more closely, set standards but help them improve.
* Give feedback often, rather than all at once on a podcast several years after the problem has manifested itself.
Based on that anecdote, I'm amazed that Digg got this far.
I suspect the one making the analogy is such a developer. Related: http://en.wikipedia.org/wiki/Dunning-Kruger_effect
Every company is inundated by poor applicants. One of the most important--maybe the most important--responsibilities of a founder is hiring good people. He failed at that responsibility.
There would be a folder.
You'd put your stuff in it.
It would sync.
Why didn't anyone else build that? I have no idea.
"But," you may ask, "so much more you could do! What about serving public files over the Internet. It could be an easy-to-use Content Delivery Network!"
No, shut up. People don't use that crap. They just want a folder. A folder that syncs.
https://www.quora.com/Dropbox/Why-is-Dropbox-more-popular-th...
Replies need not be antagonistic, you know. I'm more than a little amazed how quickly people assume disagreement. Aren't we all one big happy family?
In any event, (a) I was coat tailing on your excellent and insightful joke, (b) I quoted someone else's brilliant wit, contributing nothing original, so (c) no matter how people arrived at the idea that my comment wasn't contributing to the conversation, (d) they may be right :-)
The problem with Digg is they stumbled onto a formula that drove them to success overnight, but the company never had a true vision that they believed in and followed as a company.
I hope this hasn't done a disservice to other "about to be" funded entrepreneurs who might not be so cavalier about their ventures.
if the answer is positive, that would be an argument to defend more careful development over quick business returns and short lasted customer happiness.
there are plenty arguments against over-engineering, there must be the balance :)
The Dropbox bandwidth limit for non-paid users is reportedly 10GB/day. If the file's 20MB, that's only 500 views. Definitely not a CDN.
they can hire as many of the worlds best developers as they want - it does not hide the fact that diggs management people are yesterdays stars