Unfortunately, I do not not anyone well enough to conduct this experiment.
We expect and accept a high false-negative rate. Our interview process is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant rate of rejections of people who are clearly qualified.
The sort of people that we want to hire are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.
Of course everyone strives for better processes —at the end of the day, that means more revenue— is just that those problems seem easier to solve from the outside.
Consider these stats : the last three versions of the program — each
420,000 lines long — had just one error each. The last 11 versions
of this software had a total of 17 errors. Commercial programs of
equivalent complexity would have 5,000 errors.
(Also, it's just an analogy. I think perfect hiring would be much harder than bug free software).
Anyone who tells you differently is deluded or lying.
I don't know if it's possible for anyone to answer this question but: does any organisation have a hiring process that's 5000:1 better than the average equivalent, like NASA's 5000:1 software defect rate?
But I would like to point out that of the two of us, only the one who doesn't know is suggesting that we have "terrible hiring figures" or a "terrible interview process".
What exactly is your objection to this post?
Kind of reminds me of this HN comment and rejoinder: https://news.ycombinator.com/item?id=35079
Did you win the Putnam?
Yes, I did.
(I didn't downvote you)
And you therefore need to focus on things you can clearly demonstrate and/or prove. Which is good advice for all discussion, really, but especially when people are going to be evaluating your commentary with more skepticism.
If you know someone is wrong due to confidential data, there's a couple ways of tackling that. Saying you know they're wrong because of info they can't see is generally not a productive way to correct someone, because nobody can really be sure you're being honest, or see what you mean. You will change no opinions with that approach. However, you may be able to use public data to at least demonstrate that someone is probably wrong.
Quality of candidates and success of hiring practices tends to be pretty subjective, mind you, and it's going to be hard to provide hard evidence of it. Long term success of a company would probably be the best metric. (Of course, which Google has in spades.) Though the counter to that would be many of their more recent stumbles that show that trend may be coming to an end.
As an additional question, apart from the rest of my comment: You've said you're an SRE. Do you have a hand in hiring at Google, perhaps for a team under your management?
> Of course, which Google has in spades.
http://www.investopedia.com/articles/investing/020515/busine...
That includes gmail, youtube, etc, not just search.
A further breakdown would be interesting though, AFAIK the still pay mozilla a bucket load to use google as the default search, so I'd say search alone was a significant portion of this.
The Mozilla deal ended some time ago. That is how we got here: http://arstechnica.com/business/2016/07/recode-mozilla-could...
So 23% + some portion of 77%. Still not the vast majority.
From the article:
- 75 billion revenue total
- 52 billion "AdWords and Search Advertising"
- 15 billion AdSense
So yes - the vast majority came from the ad business:
15+52=67 billion (90%) from the ad business, that leaves just 8 billion from all other businesses.
"came from Google’s own websites" - that was traffic, it still was ad revenue! With that ad revenue they created free content websites like Youtube. But what pays for Youtube are Google's ads. It is ad revenue. Creating a huge free video site over the years when you have billions to subsidize it is not a sign of superior business intelligence in my opinion.
I never disagreed with that. Originally you said:
> Or do they? The vast majority of their income still is from the search business alone.
This is very different, their search business is a subset of their add business.
The sort of people that we want to comment are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.
echo "Your Manager Name Here" > file; shasum file b8de53741d9e64711d4f47dd3a409230fc242fac
I made 30+ interviews before i got bored, for the reasons explained by ludable@. That's a personal choice, but i didn't buy much into asking useless and weird data structure questions. The attitude interviewers had wrt the "false negative" was also really artificial, and also felt elitist. You could really feel that when interviewers were thinking of/discussing new questions, and you could feel how "proud" they were when they found something not even remotely relevant to their day to day job.
I have been at google more than you have, that might be a factor in my thoughts, too.
I was hired when getting into Google was arguably more difficult than it is now (2007). During my years at Google, I conducted hundreds of interviews and managed dozens of engineers at Google. I left of my own accord, in case there's any temptation to question that.
In my personal experience, the high false-negative rate in exchange for hiring only the best people is a myth that perdures from the early times. It's also a great morale booster for those who need to feel part of a select elite.
I had the pleasure to work with many exceptional people at Google and I learned more from them that I could have ever dreamt. As the organization grew to tens of thousands of engineers, I've also worked with many who are definitely not the cream of the crop. Same for management (proof: they hired me). And sadly saw very talented people rejected for stupid reasons because they just couldn't, or didn't want to fit into a mold.
Of course, you'll point out, this is just anecdotal data, and you have internal, non-shareable data that proves you right and we just have to believe you because you're currently an SRE at Google.
My point is that bringing a tired, cold argument about false negatives sounds elitist and inconsiderate. It's obvious that discrimination has taken place that has affected a fellow engineer and human being. I would be amazed to be helped at an Apple Store by such a talented individual. I would love to chat with him about his past work and share war stories of he had some time.
Your comment and posterior response smack me of elitism and lack of sympathy and tact.
It's a ruthless industry but we need not be.
Yes, there are many things I just can't share, and the only data point you can get in that area is my own opinion. How much value you place on that is up to you; I'm giving you the only thing I can. If that is of no value to you, you're free to discount it. I freely acknowledge that I can't prove you wrong. The only alternative I have is to say nothing at all, which is what I usually do. If you would prefer to have no input from people like me at all, by all means say so.
> It's obvious that discrimination has taken place that has affected a fellow engineer and human being.
I would like to make it clear that I am responding only to the comment I responded to, which raised a very specific question that I could answer. I do not feel that I have any basis to comment on the original article; please do not associate what I am saying with that.
What he said was fully reasonable, why do you doubt his ex-affiliation?
I just didn't find it necessary to comment on anything until I saw your comment yesterday. And, honestly, I don't see any reason why I should disclose my identity to you.
I don't know how long you've been at Google, and I don't know what culture you're drinking from. At my Google we tried to listen carefully and respect opinions and arguments without regard to who issued the opinion. Doubting a person's background just because they are presenting conflicting arguments wasn't part of the Google I worked at.
It's ok to be proud of one's company, but it's dangerous to bask on reflected glory and put oneself above others based on that. Ego is a reason killer.
A huge amount of time/energy/money wasted in interviewing way too many people in a way too deep recruiting process.
Now, imagine that google's interview process, optimised to reduce the false positive rate [1] to 0% as it purportedly is, rejects 10% of applicants that should be hired (i.e. it has a 10% "false negative rate").
How would you guarantee that this rejected 10% does not include the 1% that are "the best"? You can't find out because you've already ditched them, so you can't exactly compare them to the ones you hired. You can find out which of the ones you hired are "the best" but only compared to your other hires. There's no guarantee that you don't end up hiring mediocre people, just by consistently failing to hire the actual best every time.
How likely is it that you'll ditch the 1% by chance? If you consistently reject 10% of candidates you actually should hire, then it's one out of ten, I'd say.
So it depends on how high is google's "false negative rate". If it's as high as 50% they may well end up rejecting half of the people they're trying to hire. The google SRE user above mentions "many false-negatives". That sounds like worse than 50%.
So, to answer your question with another question: what happens if you consistently miss most of the group you are trying to hire, week after week?
_______________________
[1] Normally people look at true positive rate and true negative rate, the former being the proportion of all positive results that are correct, and accordingly for the latter. "False positive" is just the complement of "true positive".
Also, note that a process may have a high TPR and high TNR at the same time, so a high TNR on its own is no guarantee of a good-quality process.
You never go into an encounter with a new person or team being unsure of whether they're going to be difficult. You never have to avoid dealing with "that guy". You get to trust everybody that you meet.
It doesn't just improve overall quality, it makes it a better place to work. Good engineers are happier in this environment, and there is pretty near universal agreement from people who have experienced the results that this is a thing worth preserving.
There are plenty of things about the hiring process which get enthusiastic internal debate, criticism, and data-driven analysis. This is not one of them. This is a thing which we really like.
(Full disclosure: I have gone through the interview process twice, failed the first time, passed the second.)
I do some work on this personally. I'm just not allowed to discuss the details at present.
(The reasons why I'm not allowed to discuss this are due to tedious bureaucracy, not anything interesting. I have tried to get that changed, but it would require more effort than I am willing to expend on a problem that will go away in time.)
Even when large organization do look at the data, they are reluctant to share it, even internally. I applaud your efforts and hope are tracking the results.
For instance, say you notice your workforce is strongly biased against one population group, like female software engineers [1].
In that case:
a) Set a target for the proportion of female software engineers you want to hire.
b) Stop hiring male engineers at the point where hiring more would make you miss your female-engineer target.
c) Keep hiring only female engineers until you hit your target.
In principle, that might cause some concern among male engineers who could feel discriminated against.
In practice however, google and all other tech companies are already employing that process, except they do so informally (one hopes) and the groups they hire for are not the ones usually included in "diversity"- for instance, according to [1] 72% of tech workers at google are male, vs 50% ish in the general population.
Additionally, when it comes to google specifically, recruiters are supposed to actively go after "the best", so it shouldn't be a problem for that company in particular to go after the best female people.
In fact, that google targets its hires and yet it ends up with a strong bias towards a specific kind of engineer is a very good example of how not to do diversity.
[1] http://www.theverge.com/2015/8/20/9179853/tech-diversity-sco...
The problem with your idea is that it's illegal and a law suit waiting to happen.
You simply can't hire someone because of their sex.
Yes, deliberately. The fact there's inequality in the distribution, when there is no good reason for it is evidence of some sort of bias in the selection process.
>> For whatever reason, the majority is of a particular gender.
For the reason that there is inequality between the two genders in society. You don't need to grasp for metaphysical explanations when you already know the two genders are treated differently in employment (as in many other matters).
If it's unconscious bias, it's not deliberate. And it's much more than just selection bias, it's also a cultural/social issue. It happens way before someone enters the workforce.
>>> For the reason that there is inequality between the two genders in society.
I'm not denying that.
The bias might be unconscious, but the tendency to avoid hiring people from a certain group is entirely conscious. The person subject to the bias will find a way to rationalise and therefore justify their tendency to hire fewer of that group- "in my experience, most A candidates are not fit for the role" or "maybe group A is just not very good at this role because of human evolution" etc.
>> It happens way before someone enters the workforce.
It's 2016. That excuse -the whole "it's society's fault"- died sometime last century, when society's attention was drawn to the issues of gender inequality for the first time. We all know there are issues, we all know to keep an eye out for them- those of use who care about that sort of thing anyway.
As to the tech companies in particular, they don't have much of an excuse because the lack of diversity in their workplaces has often been pointed out.
That statement doesn't make sense. Avoiding to hire people from a certain group is bias by definition. How can it be unconscious and conscious at the same time?
>>> It's 2016. That excuse -the whole "it's society's fault"- died sometime last century
What are you talking about, there's absolutely a cultural and societal element here.
>>> those of use who care about that sort of thing anyway
Oh get off your high horse. Some of us just don't recommend breaking the law to fix this.
Your solution to tech diversity is overly simplistic and unhelpful. Not to mention sexist and ILLEGAL.
At best, it would solve equality of outcome, but not equality of opportunity. Women might still be discriminated against during interviews, but "that's OK because we filled our quota for this year" ?! How is that fixing the problem?
Just curious, would you ask other industries to do the same and implement these quota requirements? Would you impose that sewage workers reach 50% gender parity?
Also btw- "my idea" is not mine at all. It's the method of gender (and also race etc) quotas, already utilised in some countries.
Those are also countries that have very strict regulations about gender (etc) discrimination so it's perfectly possible to have legislation that hinders discrimination and supports equal treatment at the same time.
Think about it, if it was that easy, why aren't these tech giants already doing it?
Why are they even publishing these numbers then? Even when it's not helping their image.
Besides, "diversity of ideas" is not the point. There is no such thing in the industry anyway (see "Uber for X" or "Facebook for Y" etc). The point of what I suggest above, essentialy imposing a quota to "help" companies hire more of the kind of people they currently don't, is to give more opportunities to those groups, not to help companies increase their bottom line.
Because increasing companies' bottom line is a goal of companies, not a goal of society. Society must ensure that all its members are treated fairly and have equal opportunities (an opinion, of course and you may disagree). Whether company Z makes a few more bucks or not is of no consequence, as long as the whole community prospers together.
And there's no way for the whole community to prosper together when half of it (at best) is traditionally excluded from the best paid jobs.
Edit: also, from my POV as a soft eng, the idea of using only half of your resources (in this case, people) because you are unwilling to challenge some centuries-old tradition that says the other half is not as good, is bloody stupid and very, very inefficient. It's up there with forcing left-handed kids to learn to write with their right hand because the left one is "bad". Or something.
Starry-eyed kids with no sense of ownership to root them anywhere else, always ready to study and jump ship every 6 months? :p
I wanted to run that small experiment as a way of showing that the shortage of engineers is overstated (a common refrain in the H1-B visa debates). I am not sure if Google is one of the companies that says they can't find qualified engineers. Many peer companies to Google do say that. Having a very high false negative rate is something that should be taken into account with the current visa debates.
Why are false positives worse than false negatives? Optimising for either exclusively sounds like a very good way to fill up with mediocre people: either you miss too many of the best (what you do), or you hit too many of the worst (what you try to avoid doing).
Of course, there's no hard-and-fast rule about the quality of a hire. You can hope to have zero false-positives, but you can't really count on it.
What you should aim to optimise is the ratio of the people you really wanted to hire over the people you actually hired. This would allow you to improve the quality of your hires over time so that it approaches the high point of some measure of goodness.
Realistically speaking, that's the best result you can expect to achieve. Any tactic that purports to give a better outcome (zero false negatives? Really?) should be regarded with suspicion.
If Google is so worried about false positives, why don't they just ask much harder questions in interviews ?
If the hard questions are solved, Google should proceed for recruitment and reject otherwise. No candidate feels puzzled by a rejection, since they know they failed too.
I'd say the questions used by Google can be solved (or expect to be solved) by most of the CS grads coming out of top 50 world's universities and also various coding competitions/ top coder etc. The fact is that the output of those universities far exceeds what Google wants to recruit in a year. Yet Google complains on not finding enough talent !
Please learn to convert the false negatives to clear cut negatives while also preventing false positives.
Why does Google set up multi-layered committees post interview to gloss over the interview discussions (actual interviewers not included) and then take decisions based on some fuzzy undisclosed logic ?
Srinivasa Ramanujan was dismissed as unqualified many times, as was Michael Faraday, and many others.