Interview with Nick Chamandy, statistician at Google
simplystatistics.org
simplystatistics.org
"...my PhD research was on Gaussian random fields, with particular application to brain imaging data. The bulk of my work at Google is in other areas, since I work for the Ads Quality Team..."
I wonder how much innovation in health and other important areas are hampered by the draw of bright people like Nick to the quest for ads and likes. Follow the money I guess.
"The biggest tragedy of our time is that so many brilliant minds go work in the finance field"
and also, still Buffet: "Finance is the crumbs of capitalism".
Now I'd say that the reality is quite complicated: nearly anything that helps grow the GDP has a positive effect on technological progress.
Even finance, by providing liquidity, plays an important role. Now when a country like the UK has 8% of its GDP coming out of the city obviously there's something inherently rotten: you can't just have an economy made of nothing. 8% of finance is way, way, way too much and it doesn't take a genius to see were we're heading: major crisis after major crisis. These "brilliant minds" in finance managed to create crazy derived products totally unrelated to the fundamentals of the economy and it cannot possibly be good.
So it's all about finding a % that works, just as with everything. Let's take the "geniuses" that go into politics and raise public spending in a good Keynesian fashion to 67% of the GDP like Sweden in 1993 (recent The Economist article)... Government deficit running out of control, public debt accumulating fast and, eventually, lowering the % of public spending to saner level (49% in 2012) and reducing the public debt.
There's a place for ads. There's a place for a disrupting company like Google who brought us so many things (including a free OS for smartphones, cutting the grass under would be monopolist' feets like Apple or Microsoft). There's a place for finance. There's a place for reasonable public spendings (what there probably should be no place for are crazy socialists who are going to use intellectual terrorism to get elected and then constantly raise the power of the state and raise public spendings, constantly strangling the public sectors and destroying the entrepreneur mindset and, inevitably, leading to state default and authoritarian states).
The question is: "Which % of the economy should it represent so that we work efficiently?".
Just as with nearly everything: "Which % of the GDP should public spendings represent before there are diminishing returns?" (hint: it's way lower than what socialists think it should be but I'm still a statist: I'm not an anarchist and I do believe there should be a reasonable state).
Mostly, I'm siding with Buffet and with GP here: there are two many great minds working on too many bullshits.
The reality is that people do it for the money and prefer not to think about deeper matters. Some of the work is also really interesting and cutting-edge. But if you start rationally thinking about your contribution to society, the only answer is that there is none. If you want to do something useful, you leave finance.
It's not simply a matter of throwing money and brains at healthcare. I used to work on data analysis at a healthcare startup. There is no shortage of driven, talented, intelligent people making businesses in that space. There are giant firehoses of money in healthcare as well, both effective uses and of the boondoggle variety.
So it's not solely the lure of money that draws bright people out of healthcare. It could also be the realization that nothing you ever did would be more effective than getting doctors to wash their hands, and doctors still don't do that.
Many people are brilliant enough to make wonderful tools, but it seems that nobody is brilliant enough to overcome a system that doesn't want to use them. Whereas if you make a tool that helps businesses sell advertisements, or whatever the source of revenue, there is a higher chance people will actually use it.
If you could invent some sort of mechanism that told doctors to wash hands, I think they would be more likely to. Here's a few ways to do that:
1. Google Glasses with OpenCV - touching blood, bodily fluids etc. sets an alarm that doctors must either snooze till later or walk up to a sink to silence.
2. Special gloves that instantly turn green when coming in contact with the most common bacteria.
3. Wrist watch / mobile device that beeps loudly every time a doctor goes from one patient to the next without stopping by a sink.
I agree that hospitals and doctors don't want to change because that takes time. However, there is no point in giving up today because your work isn't going to be popular for two more decades.
First, people are pigeonholed into certain roles. Statisticians analyze data and design experiments. They are expected to know classical statistical fields, theorems, and their standard usage and caveats. Programmers should be able to answer standard programming questions, such as dynamic programming, balanced tree, etc., and of course write programs. In large company where specialization is the mantra, expertise in both fields is not an assert in interviews.
Second, multidisciplinary experience can be a liability. Since my experience in statistics is nonstandard, more of signal processing than classical statistical inference, I am not as conversant in classical statistical theory as a statistical Ph.D. does, especially because genomic research often prefers most basic methods, as in industry. Interviewers rightly ask about things they know, and are not impressed if one cannot answer questions they learned in graduate class. It is similar with computer engineering interviewers. They will ask me to implement an interval tree or dynamic programming algorithm in 30 minutes. Most of my heavy programming is in numerical analysis and optimization, where dynamic programming is very different from what a programmer thinks it is.
Third, depth is not required in industry, and certainly not in interview. Interviews now-days feel very much like college entrance exam in China or one of those East Asian countries, where people are expected to regurgitate set answers and the most important trick is to meet expectation. It is not important to master materials but to have right answers. And the right answer depends on the person asking question. One may considers SVM to be mainly a kernel trick that molds nonlinear relation into linear function, while another considers SVM as finite approximation of dynamic optimization with a breakthrough in quadratic programming that efficiently solves the two-points boundary problem coming out of dynamic optimization. This gets back to pigeonholing roles. A professional statistician will prefer one while a control/dynamic system expert will like the other. The killer is that some interviewers ask questions with their preferred answer in mind, and the questions can baffling to people with different background.
Forth, different companies demand different capabilities for supposedly same roles. Data scientists can be as mundane as denormalization or as sophisticated as inventing a way of causal inference. It is not always easy to tell from ads. It is even less so when the company wants jack of all trades and experts in all possible tasks. Ask very pointed questions.
Fifth, there is no advantage in being both good programmers and good statisticians, at least in interviews. I have already noted several disadvantages. People much prefer build inter-disciplinary teams each member of which is tasked with one special area and let them talk. It works well. I cannot think of anything that requires expertise in both programming and statistics in one head. It may be a little slower, but not noticeable.
I am having doubts about a position in a large company because of too much specialization. I like to derive an algorithm and implement it efficiently. Even if I could get a position, and I could if I cram for interviews, I would be at a disadvantage when others can concentrate on one area. Sadly, academia is becoming very much like industry, except they count papers or grants instead of make money. I am still looking for my niche.
I guess the gist of my rant is that today's job market demands specialization and people better conform.
In general you are right about pigeonholing/specialization in large companies. But it usually happens after you are hired, not before. Right now, without industry experience your skillset is very broad and it is hard to pigeonhole you.
As for the interviews, if they are like an exam where you know questions in advance you certainly should prepare. Sadly, this is usually not the case. As you mentioned in your point (4), sometimes you have no idea what the role is about. And there is no way to know because your recruiter doesn't want to get specific or maybe has no idea herself.
Which brings me to an important point which you might be missing. If you are applying for jobs through websites or by any other well-publicized way, you will inevitably subjected to a vetting process. The interviewers do not know anything about you and are basically looking for reasons not to hire you. If you are very good technically, it increases your chances but it is no guarantee of passing. Good communication skills are just as valuable.
It is different when you know someone inside the company, say your friend recommends you. You will have more meaningful interviews from the start. Most startup/small company hiring is done this way.
That said, with your background you should have little problems finding a job, if current Big Data hype is to believe. But of course it depends on your school, where you are looking etc.