Techniques like deep learning were out of fashion for a long time. Google was making tons of money on non-AI techniques; I remember Sergey Brin saying he was surprised by the deep learning breakthroughs in 2012 because the conventional wisdom was that "AI doesn't work".
And people who did Ph.D's under Hinton and others 10+ years ago also were not following trends.
You have to take risk to get reward. The way to do that now would also be to labor on an obscure subfield of AI that isn't certain to even work, let alone become commercially viable. (And there are plenty of people who did that on fields that looked more promising than deep learning 10+ years ago.)
There's definitely a need for new AI techniques, see this other current story:
Also, having a few NEURIPS or ICML papers is not a hiring guarantee any more. It's decidedly not 2015-17 any longer. In particular, I feel bad for the people who started their PhDs with $$$ in their eyes around that time.
Don't get me wrong, they will all be employable, but the field moves so fast, I have my doubts there will be cushy FAANG jobs for everyone capable of playing around with network architectures in a few years. It's a terrible idea to start one based on the hype few years ago.
- Would you say that you'd have got an AI researcher position without a Ph.D.?
- Also, why is NEURIPS or ICML papers is not a hiring guarantee? I thought they're highly sought after.
It's difficult to get any true researcher position without a PhD. It doesn't mean that PhD has to be in AI. Research involves a lot of reading and writing papers, which a PhD is supposedly training you how to do.
That said most places will say "equivalent practical experience" and it's entirely possible to be competent in AI/ML without a PhD.
I did a PhD in space science, I now do machine learning in ecology and spent the summer working on machine learning for disaster management. The interesting jobs (to me) are where domains cross, and it's also (hint hint) much easier to get a job doing AI for X than it is doing "fundamental AI". In any case, you're often doing stuff that nobody has done before anyway, but you don't need to spend your life hunting for the new ResNet.
> Also, why is NEURIPS or ICML papers is not a hiring guarantee?
What the OP probably might be implying is that everyone has a publication in NeurIPS nowadays.
I think it goes deeper than that though, publishing in machine learning is broken. Having 10k people at one conference is not an efficient way to distribute research. You have to submit a full paper in November for a conference next Summer - pretty much only computer science does this madness.
What's interesting is how unique this attitude is. In astronomy, for example, conferences are a fun place to catch up with folks in your niche. There might be a few hundred people and probably it'll be single-track. We publish whatever journal is the most relevant and they're generally all considered equivalent. Nobody cares if you publish in ApJ vs A&A vs MNRAS, if your research is good.
There are also concerns that the quality of these venues is decreasing because the pressure to publish in them is so high.
Do you think it is possible to that without any background in anything? I mean could someone apply black box frameworks without understanding them. How would they be caught?
To do machine learning research? Or work in some random domain?
> I mean could someone apply black box frameworks without understanding them. How would they be caught?
Machine learning is rapidly becoming commoditised, but lots of people still don't understand just how much effort it is to get a good dataset and to prep
Domain experts scoff at machine learning people who are trying to solve Big Problems using unrepresentative toy datasets, but also tend to have much higher expectations of what ML can do. Machine learning people scoff at domain experts for using outdated techniques and bad data science, but then propose ridiculous solutions that would never work in the real world (e.g. use our model, it takes a week on 8xV100s to train and you can only run it on a computer the size of a bus).
There are also a lot of people (and companies) touting machine learning as a solution to problems that don't exist.
Overfitting models is probably the most rampant crime that researchers commit.
From the second half of your comment it seems that the answer is yes?
Maybe a comparison would help: someone pretending to be an experienced iOS/Android developer without any qualifications or ability would quickly be caught. Since they couldn't produce any working app or use a compiler, and anyone can judge an app for themselves. You can't really just make it up out of whole cloth, people judge the results. You would have to start actually doing that, and if you couldn't or didn't want to, then unless you outsourced your own job or something the jig would be up pretty much instantly. (Unless you caught up.)
So, how about machine learning? Do you think a fraud could land and keep such a job, without any knowledge, qualifications, ability, or even interest in getting up to speed? Just, a pure, simple fraud.
What's your guess?
Fake it til you make it isn't a terrible strategy. But pure fraud? If you didn't even make an attempt to learn on the job? You'd get caught pretty fast as soon as someone started asking any kind of in depth questions about the models you were supposed to be training.
I'm not sure you could land a job knowing nothing. Maybe. Depends how hard you get interviewed and whether they know about machine learning. If you could fake a portfolio and nobody questioned it perhaps? I can see that happening in academia for sure.
There are a few problem classes where you could throw stuff into a black box and get great results out. Image classification for example. Fast.ai have made that three lines of code.
So maybe there are a bunch of applications where you could fake it, especially if you were willing to Google your way round the answers.
Would be harder in industry I think, but you find incompetent people everywhere.
>But pure fraud? If you didn't even make an attempt to learn on the job? You'd get caught pretty fast as soon as someone started asking any kind of in depth questions about the models you were supposed to be training.
That's just what I mean. It would depend on someone asking you about it, right? (As opposed to being an iOS or Android developer or running microservices on the backend: in those domains nobody has to ask you anything, it's instantly obvious if you're not building and can't build anything.)
For machine learning, who is asking these questions?
If you throw data into a black box (3 lines of code) and are incompetent, can you please tell me a bit more about where you would get found out?
Let's use your example, ecology.
I show up, I get a dataset, and I put it into tensorflow using three lines of code I copy from stackoverflow.
I lie and bullshit about the details of what I'm doing, by referencing papers from arxiv.org that I don't read, understand, or actually apply. It's just the same 3 lines of code I copied on day 1. I don't do anything on the job.
How long could I last? An hour? A day? A week? A month?
Assuming I am outputting 0 useful work. I'm not doing any machine learning. Just 0 competence, or I make something up by hand or in excel.
I am trying to understand how people are judged.
If you really wanted to you could fabricate results and in lots of cases nobody would be any the wiser unless you were supposed to be releasing software. Despite emphasis on peer review and repeatability, science relies heavily on etiquette. If you don't release code or a dataset a lot of times it's extremely difficult to repeat paper results, and that also means it's hard to disprove the work.
It's quite hard to get rid of incompetent people in academia, so I imagine you could get away with at least a year or two.
They're sought after, but the conferences have also grown huge. NeurIPS 2018 accepted around 1,000 papers! Based on a query of the DBLP [1] dataset, there were 4,409 distinct authors who had a paper at either NeurIPS 2018 or ICML 2018 (or both). If you add in a few of the other big AI and ML conferences (AAAI, IJCAI, ICLR), the number grows to 10,995 distinct authors, again solely for the year 2018. The field is hot, but is it hot enough for ten thousand people to be automatically hired because of one paper?
There's also decreasing confidence in the big conferences' review processes I think. NeurIPS 2014 actually did a study to estimate how random acceptance was by assigning some papers to two different sets of reviewers and checking how similar the decisions were [2], and found there was a much higher degree of luck in acceptance/rejection decisions than they had expected. I personally have more confidence in the review processes of smaller and more focused conferences (and journals!), though they don't have the same level of name recognition.
[1] https://dblp.uni-trier.de/
[2] http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
Also, NeurIPS reviewing has gone to absolute hell. I mean, peer review everywhere has problems. But I've never seen something quite this bad. At this point I think it's safe to say that most reviewers wouldn't even make it to an on-site interview for a faculty position at a research university. That's definitely nowhere near normal. You can't really blame anyone, I guess; the community is growing way too quickly for any real quality control.
Frankly, I think those conferences have outlived their usefulness as anything except marquee marketing events. I'm now mostly attending smaller and more specialized conferences.
Besides just "quality" in the general sense, one thing this has really hurt, I think, is any sense of history or continuity. There are a ton of reviewers who have basically no familiarity with the pre-2010 ML literature, and it kind of shows in both the reviews and the papers that get published. I mean I get that deep learning beats a lot of older methods on major benchmarks, but it's still not the case that literally every problem, controversy, and technique was first studied post-2010.
- "containing nothing; not filled or occupied"
- "lacking meaning or sincerity"
- "having no value or purpose"
If one lacks that primary motivation, they certainly "lack" it and are "not filled" with the academic ethos. So "empty" is a fair term.
Now, I understand the realities as well -- seeking publications, recognition, tenure, and funding are also political activities. But this does not contradict the underlying community norm that I mention above. In fact, it supports it -- it explains why so many people endure a tough, grueling, political process despite it not being their wheelhouse.
No idea why you think this. A phd has a contrat with a salary, starting and end dates.
My comment above makes an argument. You don't have to agree with it, but it should give you some idea about how I came to my conclusion.
So basically your conclusion doesn't go well with my experience. I only saw a contract with low salary and a lot of work on my end. It was worth it for me (I think?), so I accepted.
To what degree did you enjoy the process of learning, collaborating, teaching, writing, experimenting, and so on? I'd wager you did enjoy some or many of these... otherwise, it might have been a long slog. :/
Maybe the following story can convey part of my message. You might have seen movies about a wily protagonist villain (or a flawed, tenuous partnership between several) who meticulously plan to steal some priceless artifact from some nearly-impregnable facility. What drives such people? I don't think it is purely money -- there would be alternatives that would, rationally speaking, generate more income, on a risk-adjusted basis. In the case of the ninja-suit wearing infiltrator(s), I'd argue they fundamentally enjoy the process (the preparation, the planning, the deferred gratification, the meaning). Perhaps the same is true for people that pursue and complete a Ph.D. -- some get a decent financial payout, but on average, I don't think the degree made them better off financially compared to other alternatives (e.g. holding together some rotting infrastructure with bailing wire). They value the title, the activities, the identity, the community, the kind of work they do.
Seeking a job only for money doesn't really endow much meaning -- (Please, don't take this as an endorsement to go off and work for some harebrained startup when you have better options. :P) -- though I think there is plenty of meaning even in the mundane (e.g. rearranging JSON) to be found if you open yourself to experience (e.g. books with dragons about parsers).
I would like to share my views around fairness and judgement. My apologies if the numbering makes them seem formal; my intention is only to give them a rough ordering.
1. One should not be eager to criticize others.
2. One should seek to understand others.
3. However, one should be willing, intellectually, to differentiate between aspects and assess those differences.
4. It requires some care to balance 1, 2, and 3.
5. One should be honest with oneself, at least, about your conclusions.
6. One should be comfortable with your assessments, particularly if you've thought them through.
7. One should be willing to share these thoughts with others, because debate will improve your thinking, scope, and articulation.
8. One should accept the consequences of what you say.
9. One should learn from what you say.
10. One should not refrain from making assessments only out of a fear that someone will label you as "judgmental".
11. Some people criticize others because they dislike the other person judging others. This is somewhat ironic, because in some cases this criticism is premature. If one judges another without engaging to develop an understanding, I think that is unfortunate. Doing so would be acting in a way inconsistent with one's own values.
All of these "should" statements should be adjusted to the situation. For example, repeated experience, if reflected on fairly, may warrant that some particular people do not deserve the same degree, say, of "benefit of the doubt".
Presumably because technical needs of FAANG might be moving in other directions. Could someone comment why this might be the case, and what other fields might look relevant
I don't think this will happen, because typing "import tensorflow.keras.*" isn't the skill that an ML Ph.D develops, and it is the part of the skill set that is (and will be) commodity along with the automl stuff.
Constructing a problem, handling the data and running a proper process is harder, and it's the value that will put processes that use ML at risk, and deliver differentiating value for the ones where it works.
I also suspect that other fields have similar patterns where a few experts make a ton of money, they're just not as publicized because the media is so hyped on AI. How much you think Jim Keller is making?
TL;DR do the PhD because you love the research. If you want the money do some internships, land a FAANG job, and try and get to senior as fast as possible.
The problem that most technical people seem to have is that they think a qualification is a licence to print money. It isn't. The majority of people who obtain qualifications have no idea how to make money (that is why they are doing the qualification rather than making money, and this effect increases with the complexity of the course of study).
So if you want to make money and your interest is AI, sure do a PhD...but at some point, you are going to have to work out how you can use those skills to sell something.
I think, at some point, there is going to be a rude awakening. Right now, we this situation where you can tap up some VC idiot to pay your salary for a few years whilst you fuck about on some bullshit. You can get acquired by some megacorp with a ton of cash and have no accountability from shareholders while you light huge stacks of their cash on fire...this always ends. Always.
You are either a bureaucrat chasing qualifications to get no-accountability jobs or you are someone that can create value for other people. I am not knocking any kind of advanced degree (I have one). But the point of it is not money.
So even posing the question is totally misguided.
Frankly, I think this is a bubble, and don't expect to ever make anything like that much money, in academia or industry. Yes, deep learning has a lot of successes and industrial applications, but deep learning wizards are not nearly so impossible to replace or replicate. As the field settles more towards engineering than wizardry, and as expectations are recalibrated towards the realistic, it's going to stop raining cash.