My story as a self-taught AI researcher
blog.floydhub.com
blog.floydhub.com
The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups).
The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.
Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reali...
http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T...
And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText
That isn't even counting our hardwired animal intelligence.
The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself.
It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).
Yes, and they do. They aren't choosing completely arbitrary algorithms when they attempt to solve a ML problem, they are typically using approaches that have already been proven to work well on related problems, or at least are variants of proven approaches.
The question is, how much information is encoded in those algos (to me, low-order logical truths about a few elementary variables, low degree of freedom for the system overall), compared to how much information is encoded in the "algos of the human brain" (and actually the whole body, if we admit that intelligence has little motivation to emerge if there's no signal to process and no action to ever be taken).
I was merely pointing out this outstanding asymmetry, as I see it, and the unfairness of judging our AI progress (or setting goals for it) relatively to anything even remotely close to evolved species, in terms of end-result behavior, emergent high-level observations.
Think of it this way: a tiny neural net (equivalent to the brain of what, not even an insect?) "generationally evolved" enough by us to be able to recognize cats and license numbers and process human speech and suggest songs and whatnot is really not too shabby. I'd call it monumental successs to be able to focus a NN so well on a vertical skill. But that's also low-order low-freedom, in the grander scheme of things, and "focus" (verticality) is just one aspect of intelligence (e.g. the raging battle is for "context" as we speak, horizontality and sequentiality of knowledge; and you can see how the concept of "awareness", even just mechanical, lies behind that). So, many more steps to go. So vastly much more to encode in our models before they're able to take a lesson in one standing and a few examples.
It really took big-big-big data for evolution to do it, anyway, and we're speeding that up thanks to focus in design, and electronics to hasten information processing, but not fundamentally changing the law of neural evolution, it seems.
If you ask me, the next step is to encode structural information in the neuron itself, as a machine or even network thereof, because that's how biology does it (the "dumb" logic gate transistor model is definitely wrong on all accounts, too simplistic). Seems like the next obvious move, architecturally.
Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)
I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.
As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.
while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal),
fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough.
Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.
I succeeded one time in convincing the guy behind a desk in an internet cafe, so I could bring my HDD and download a dataset in a calmer time of day, and throttled so it wouldn't disturb other customers. This went without any problems for the other customers in the internet cafe. When I asked again a few months later for a new dataset, they no longer wanted me to do so...
There seems to be no download by mail service (and I only get people forwarding me to google cloud products etc, which as a European is so financially out there with automatic balance deductions and non transparent pricing schemes, I would have no qualms using GCP or others if they ran a prepaid alternative for people who refuse to take on risk)
There is still plenty you can do with a reasonable personal budget, however.
You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.
Also, Emil's approach to learning will create a flawed sense of expertise. Look at how the article presents him as if he has a deep-domain expertise which might not be true.
One important thing to consider is to look at the article more like content marketing tactic, that FloydHub is using promote its brand which might not serve well for engineers as it lacks some aspect of truth.
Is that really the case? Apart from the obvious "maybe that's what they're intrinsically interested in" - if you start with a problem, try to solve it, and "pure mathematics" (whatever exactly that is supposed to be, anyway) is required to arrive at a solution, it becomes part of the intrinsic motivation. And if you keep solving problems without it the question that eventually comes up is "is it really useful at that point?"
I do however agree that if you're looking at someone who's qualification is primarily his "portfolio", you do actually need to check whether it includes interesting problems, or at least projects that are related to what you need them to do at your company. But if that is the case, I really don't see a problem.
Few ex:
Like, reading literature is a purely fun and mentally draining activity that might/might not have any goal attached to it.
Like travelling, is a purely fun activity and might/might not have any intrinsic goal attached to it.
I started learning Algebra out of the random, without any intrinsic goal. Because, it was purely out of fun. Playing chess is an activity without any intrinsic goal.
For some people it does. Everyone has a different approach to learning and I think outside of the heavily regimented strata of traditional schools it would help everyone if we could celebrate that. Maybe that means not everyone learns for the sake of learning, but then again not everyone has to do the same things.
You seem to use goals and "capitalistic value" interchangeably, I would just like to add that projects in people's portfolios are often not initially created to make money, but simply to satisfy the creator's curiosity or because they thought it would be neat.
We both can agree on this. But that's not how Emil looks at it, his entire point is that projects are better credentials than even degrees.
And since projects have higher value in portfolios, they will implicitly derive capital? One of the reasons why they have capitalistic value.
You and I can both agree, that projects and learning do have fun elements attached to them and might not necessarily be part of a larger goal. However, that's not what his views are!
I'm not sure how you got to that conclusion. Certainly I don't see how
>projects are better credentials than even degrees.
would necessarily conflict with
>that projects and learning do have fun elements attached to them and might not necessarily be part of a larger goal.
Also finance... really? I'm not sure I would agree that is a subject with a less capitalistic motivation.
I don't get a lot of the bitterness here. I mean not everyone here creates their own programming language, or would know how to write a database, yet we are fine using them as tools. Why must we understand all the code and concepts in a neural network to apply it?
> Why must we understand all the code and concepts to apply it?
That is one of the reason why our engineers are so sub-par because we were told to just shut up and write shit. We could have become a force to be reckoned with because of our expertise, because of our ability to solve complex problems. Yet we are living in an industry were there is huge disparity in salary, structure, and principles.
I don't necessarily agree with why one shouldn't understand the concepts, I'm more a guy who says why not?
Because, if you are standing on the shoulders of pioneers and claiming to be improving their work at least do it with compassion and honesty.
Sorry about the rant.. would love to hear alternative opinions!
I wonder:
— Is math a problem for non-academic researchers?
Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but math nonetheless.
Not that you can't learn these on your own, but how big is the gap in practice, on the job, compared with actual PhDs in ML/math? (how much of a hinderance, a problem it is for the self-taught researcher)
— "Contracting" in the field of AI sounds great but, how exactly? Especially solo: what type of clients and how/where to find them, what type of 'business proposition' as a freelancer do you offer, what's the pricing structure of such gigs?
I mean, I can sell you websites and visuals and stuff, but AI? I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny and demand scripting skills to sort out (extract business value), not AI, so the value proposition is low for both parties; it's still early adoption so 90% don't even consider spending 1 cent on "AI" unless as a SaaS (they actually don't need to know if it's AI or programming).
I can imagine tons of fantastic research to do with SMBs, as partners or 'interested sponsors' (should they reap benefits on a low investment), but really not much yet in the way of "freelancer products" to market and sell for a living. I'm eagerly anticipating those days, but it's more like 2025-2030 as I see it.
I would love to hear first hand takes on this.
It takes a while to figure out how to read academic papers, but it's largely about learning the notation. In the end, it maps back to the code you write anyway in most cases, so it's just another way of writing stuff you already know.
It's not so much linear algebra you need, since much of that is not relevant to AI. It's really matrix calculus. Which is largely about multiplying things together and adding them up. Terence Parr and I tried to create a "all you need to know" tutorial here: https://explained.ai/matrix-calculus/ .
You certainly don't need topology (unless you happen to be interested in that particular sub-field).
It might be wrong but I tend to see vectors and matrices as two notations for the same mathematical object[1]. So I indeed meant matrices! However I didn't see calculus itself as such a big requirement, as it all felt pretty "linear" to me (regressions etc). Are we talking things that e.g. "Calc 2"[2] should cover?
I feel reassured by your first paragraph. This can be done.
I'll definitely work on your tutorial; I assume it's a good benchmark for math pre-requisites in the field. Thanks a lot for the work, and advice.
[1]: That was particularly reinforced with Geometric Algebra, which I'm currently diving in. https://en.wikipedia.org/wiki/Geometric_algebra
> Most papers strike me as requiring a non-trivial knowledge of linear algebra
I think this is correct, if you consider college level linear algebra and an intuition for applying it to novel problems to be non-trivial knowledge
Yes, in the context of "a self-taught researcher", I think I intuitively meant anything that precisely requires a degree, typical academic knowledge. E.g. you can become a great business person who won't feel hindered by lack of academic knowledge, you definitely can't do that as a surgeon or lawyer.
I guess I was wondering where math fit in this picture for AI research. (which I should explicitely relate to "#2" in user ineedasername's post, i.e. "AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1 [AI Engineering Research]. What might be termed "basic" or "pure" research.")
Well you can't do #1 or #2 without having a level of maths proficiency that most college grads do not have.
FWIW, I don't understand the difference between #1 and #2 above. Most academic/industrial research is incremental (i.e. #1), and a tiiiiny % will have any impact in the way something like XGBoost would (the example he gave in another comment). That doesn't mean that the non-impactful research isn't 'basic'. You could alternatively just call #2 "groundbreaking research" and #1 "non groundbreaking research, but you need mathematics knowledge for both imo.
Things like topology (e.g. TDA, persistent homology, etc.) aren't really mainstream yet, but even then most of it isn't really "hardcore" math in the sense that you can get away with a basic understanding, e.g. what a Vietoris-Rips complex is and why we use it instead of a Cech complex in TDA. Plus most DL research nowadays is pretty (advanced) math-light. That being said, taking the time to understand the math is absolutely worthwhile in my experience.
It should also be noted that a lot of real world ML/AI projects in industry aren't really about brand new algorithms using advanced math, but rather more about applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
Basically, i want a book like Statistical Rethinking or Blitzstein's Introduction to Probaiblity, but for linear algebra. And i havent been able to find it.
I think it's exactly that kind of "intuition of what Linear algebra is and is for".
There are also some video tutorials on the first chapters here: https://github.com/minireference/noBSLAnotebooks#no-bullshit... (lots of hands-on examples using the computer algebra system SymPy)
I won't lie to you and tell you linear algebra is "easy" by any means—there are a lot of things to pick up, so it takes some time, but it is totally worth it since LA is like the swiss-army knife of science: lots of features and super useful.
Thanks for the pointer! (link[1] for those interested)
> you can get away with a basic understanding
Great news to me!
> taking the time to understand the math is absolutely worthwhile in my experience.
Strongly agree — for any topic, any field. My concerns are practical indeed, and less about the 10-year horizon (well enough to become skilled at anything) than the early stages of that, the best way to propel oneself far/fast enough on year 1, then 2, etc.
> applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
I hear that. I actually do like the sound of that, hence concerns that I was biased.
[1]: https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra...
Thanks to transfer learning, tiny datasets are not a major issue to developing AI solutions. Fast.ai makes it super easy to overcome that hurdle.
Will investigate. Thanks a lot!
So don't feel bad about your life just because someone on the internet pretends to have a more interesting one. Those people are usually just attention seekers and for some reason need the outside validation to feel good about their achievements. And remember that not needing that validation can be a strength too!
I realize that these articles suffer from the connecting the dots thing, where people make connection in the present which they would never have in the past. But that is besides my point, even if he failed at all those things, I am still jealous he had the chance to try all these things.
I submit to you that you don't sound jealous, because it really doesn't fit the rest of what you express; I suggest that you are maybe "envious¹" in the sense that jealousy means envy + depriving the other of what they have ("it should be me and not them", it's a matter of exclusivity, like being jealous of #1 if you finished second, or jealous of the one dating someone you love, or whoever took your job/offer). You don't sound hostile to them, merely wishing more for yourself (which in itself is a positive feeling?)
I really don't know what the word you meant to use in your own language actually meant (I'm French personally, so English is just our medium translation layer). But I'm curious, culturally you know, about these nuances².
I'd love it if you could just introspect that feeling a bit and share what it really feels like, the complex emotion and what it "touches" in you (does it bring despair, or motivation, or resolve, etc). [if it's too personal, I got email, just ask]
Personally, I can feel "aspiring to" or "inspiration from" people who achieved more than me — I don't want to deprive them of anything, I don't wish they failed, nor do I wish to belittle their accomplishments; however I'd love to eat everything they know, steal like the greatest of artists (the good ones merely copy!), that is at best become friends with these people and let them influence me (the true deeper meaning of that book³, if you ask me). And I know, somehow deep down, that the more I'd be rooting for these friends, helping them go further, the more I'd be moving forward/up as well, taken in by the positive storm.
[1]: envy is “a feeling of discontented or resentful longing aroused by someone else's possessions, qualities, or luck.”
[2]: Part of my (personal) research on human nature, motivations, "what makes us do what we do".
[3]: How to win friends and influence people, by Dale Carnegie
I would definitely say this lack of even the opportunity to do this makes me feel despair. I can try as hard as I want, but some things are just out of my control, some truths about my life were written even before I was born. I honestly cannot muster up the strength to derive inspiration or motivation from these, because those things are relevant for things which are possible.
> The word in my language would literally translate as desire but it's more than that, it would mean more like I would like to have that and it feels bad that I don't even have the opportunity to have it.
Yeah, OK, I get it. That's a very good word (the one in your language). I think it's a rather universal feeling, this "invisible ceiling". Many people feel that for various reasons.
There's a certain school of thought, somewhere between philosophy and spirituality, that speaks of "abundance", and beyond (or perhaps before, on the way) of "inner peace" or "inner happiness". The oldest forms I know are Stoicism (western cultures) and Zen (eastern cultures), and you'll find it nowadays in e.g. Tony Robbins, that kind of field. I think there's truth in it that just works, at least it did for me (took me about 35 years to figure it out though, as it's just totally outside the realm of "education" nowadays¹, unfortunately IMHO).
One mechanism that I've always found to be true, is that from the depth of our biggest despair comes our symmetrical potential for joy, and vice-versa. It takes knowing how good/bad it gets to really feel how worse/better it is, or rather goes.
It's certainly trying on one side, but invaluably rewarding on the other.
[1]: at least in most of the western world, afaik.
We're about the age you describe, late-20s, early-30s. But our friends of around the same age with kids seem to have a lot less of this 'freedom'. Responsibilities take over.
Not saying you can't do those things with kids - but it does seem harder.
Here in Northern Europe you're expected to do most of your traveling in your 20's, but we also have pretty decent vacations - so the solo / friend / backpacking type of traveling gets replaced with more family friendly stuff when you start getting kids.
I have lots of friends in their late 20's / early 30's that still travel the world, many times a year. But they don't have kids, and their travels (outside summer vacation) tend to be shorter, as in long-weekends etc.
As I write this, my (admittedly limited) understanding of how Western society works makes me think these would not be problems but my biggest assets.
He was hardly a qualified teacher. Lots of young people up here go on voluntourism trips to countries in Africa or Asia to teach a couple of months. If you want to try teaching, you can do so locally; There are lots of options.
Musician, sure, it's fun (I've toured in 4 different countries myself playing in band, though 15 years ago now), but it's hard to do full-time.
DL expert would be to push it, there's a long way between being productive and being an expert.
But I'm gonna be frank with you, doing all those kinds of things is possible here because we have a great welfare system. You can work, save aggressively, and do whatever you want.
The unfortunate fact of life here though, is that a lot of entrepreneurs and indie developers (just to pick a few) are funded by welfare checks.
If you think I can do that, I have definitely failed in explaining what the problem is. I have almost my post tax yearly salary as savings (apart from the investments and stuff), but due to the reasons explained in my other post I can't do shit.
Don't believe it. Like him being king of the village??? LOL
I think it's good that companies are willing to look into non-trad candidates, that may not have found their "calling", so to speak, until their late 20's / 30's or whatever. But it does start to sound contrived when a bunch of 'em have the same type of alternate-route stories, which involves traveling to Africa / India / SE Asia to help out kids, create some startup aimed at climate / poverty / equality / etc. I guess it makes you sound passionate and legit - no-one can say that you wasted your time on chasing those things.
Nothing on the quirks list is actually a quirk. They're interesting things he's done that other people wrote books about, received praise for, and then he followed their newer, well-traveled path.
It's not a non-traditional background. He's not a refugee who managed to learn coding. He's not volunteering at a needle exchange clinic. I think that's what's bothering me; he's pretending to be interesting, and taking the room which could be going to someone else.
Thank you for helping me get to why something felt off. Appreciated, internet stranger.
Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.
That is, for example, why it is possible to find people presumably seriously suggesting to:
3. Flashcard the Deep Learning Book (4-6m)
4. Flashcard ~100 papers in a niche (2m)
I mean, it's clear to me that the language deployed in the article is ostensibly about teaching yourself to do machine learning research when what it's really discussing is how to get hired by one of the companies that are curently paying six-figure salaries for machine learning engineers etc.
Or I'm just old and cynical. Wait, let me find my false teeth so I can chew that over.
However, there is a lot more to AI than high-school maths and I don't just meean -more maths. I mean knowledge, lore if you like. It's a field with a long history, stretching back to the 1930's even (before it was actually named as "AI" in Dartmouth, in the 1950's). A lot of very capable people have worked on AI for a very long time and have actually advanced their respective sub-fields each with leaps and bounds and it's not very sensible to expect new leaps while being completely clueless of what has been achived before. You can't stand on the shoulders of giants if you don't know that there are giants and that they have shoulders you can stand on.
Unfortunately, most people who enter the field today know nothing of all that, or even that there was an "all that" before 2012 (if they even know what happened in 2012; and to be honest, one wouldn't understand what 2012 means if one doesn't know what came before). So on the one hand they are not capable of making leaps and on the other hand they don't even know what a leap would look like. And probably think that a "leap" is a 10% improvement of the state of the art for a standard classification benchmark.
I agree with you though that what is needed to make leaps in AI is curiosity. Lots and lots of curiosity. Vast amounts of curiosity. Curiosity of the kind that you only find in people who are a bit zbouked in the head. Or just people who have a lot of time in their hands, to study whatever their fancy tells them to.
So- not the kind of person who flashcards The Deep Learning Book, if nothing else because that means the person doesn't have the time to, you know, actually read the damn book well enough to grokk it.
I mean seriously, what the fuck is it with the bloody flashcards?
I know I’m just speaking from my own experience and what works for me doesn’t necessarily work for everybody. But my claim isn't that everyone should do as I did, my claim is that you're wrong that a self-taught ML researcher would necessarily only be able to make superficial contributions because they are bad at math.
MIT has takes 3,000 students, Canadian universities take 30,000 students. (Remember Canada has 30 million people and US has 300 million.)
- https://web.mit.edu/facts/enrollment.html
- https://www.univcan.ca/universities/facts-and-stats/enrolmen...
But I'm not sure what that has to do with buying expensive formal education credentials.
That may be the case for small colleges with high tutors-to-student ratios, but it's not the reality in many of the behemoth colleges / universities that feed warm bodies into jobs.
I've seen first hand classes with hundreds and hundreds of students, where everything worked like an assembly line. Standardized tests with zero feedback, mentors were student TAs, a class or two above you, and they had been assigned to tens of students themselves - while correcting hundreds of homework / problem sets on the side.
When you go to school like that, it can quickly feel like you're just another name on a list, with some avg. grade on the side.
And it's only going to get worse with the ever-rising number of enrolled students.
If it's easy to see that a piece of output (a paper, code library, machine learning model, whatever) or a job candidate is great, then the credentials behind it don't matter much. However if it's challenging to evaluate quality, then people will shift to looking at secondary signals such as credentials, price, etc.
As to why a lot of economists go with the signaling model of education, well, it might just say something about their field and how much they got out of their own educations.
https://en.wikipedia.org/wiki/The_Case_Against_Education#Rev...
Bryan Caplan back and forth with Noah Smith on the book: https://www.econlib.org/archives/2015/04/educational_sig_1.h...
Bryan Caplan back and forth with Bill Dickens on the book: https://www.econlib.org/archives/2010/08/education_and_s.htm...
3 months learning FastAI, 3-12 months personal projects and consulting, 2 months flashcards of ~100 papers, 6 months to publish a paper
What does he mean by ‘paper’? A Medium post? NeurIPS?
But yeah, going from 6 months of programming experience with C, to a Deep Learning internship - that sounds a bit far stretched.
I have some friends in Oxford who are DPhil/Postdocs in highly reputable research departments specializing in ML and if they sometimes struggle to get more than a poster session at the leading conferences, with the addition of well known professors names attached, then there's just no way I can believe Joe Bloggs who just learnt python 12 months ago is able to do the same.
I almost want to follow his guide just to check.
One also needs to be skeptical while reading such a PR post and not get swayed away by the hero's journey in it.
I would rather build a small company by solving a real problem than work for a big company spinning my wheels.
I think if you look at history this is also evident: the inventions of the late 18th century were a function of necessity, the invention of semis (not just in the US but how Taiwan developed)...this isn't to say academia is pointless but there is just far more going on (I think if you look at some of the East Asian nations that get great academic results, their progress on actual R&D innovation is far less impressive).
For a thoughtful counterpoint to the necessity argument, see: https://jnd.org/technology_first_needs_last/ (previously discussed on HN)
1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research"
2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1. What might be termed "basic" or "pure" research.
I'm not placing one of these above the other in terms of importance. They are both necessary, and they form a virtuous feedback loop between the two that, one without the other, would see the other wither on the vine.
In the example of this particular person, Emil Wallner, he appears to be doing #1, and perhaps doing so in a way that might help inform more of #2.
But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of overlap. When Leo Breiman [0] did his work that led to gradient boosting and tools like Xgboost, that was firmly #2.
It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications.
To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
But I was asking because I was specifically looking for concrete examples in deep learning.
In the field of ML, a concrete example might be the tool Xgboost (#1) and the original work that led to and developed Gradient Boosting itself (#2), of which Xgboost is an implementation, and probably one that has helped refine the underlying theory as well.
ML has lots of examples where the researcher(s) for #2 were also doing #1. A famous paper in NLP comes to mind as an excellent example of this overlap (PDF: https://www.csie.ntu.edu.tw/~b92b02053/print/good-turing-smo...)
You're confusing the occupation with the role. Just because your job title is professor of structural engineering it doesn't mean that you are not studying "matter, its motion and behavior through space and time, and the related entities of energy and force."
I think this is what we try to capture as “expanding human knowledge”.
IMO the more isolated the result (“technique x gave good results for problem y, the end”), the less like “research” it is. Though plenty such papers get into good conferences every year. A nice story and a little reviewer luck go a long way.
#1 might ask about the performance of a deep neural network in approximating a given model in a specific application. Alphafold, on the front page currently, is an example of #1.
Personally I don't know enough about AlphaFold or the problems of protein folding to be remotely confident in my judgment on it
Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)? I can't help but wonder a bit.
1. Solves previously unsolved problems
2. Publishes papers sharing those solutions
without regard to the kind/spirit/scope of problems solved.
Since conference publications don’t have the same number constraints as journal papers, and are accepting of application-specific results, this explosion of what is considered “research” is somewhat inevitable. Also, there are a lot of people chasing this given the prestige associated with the title.
From his GH profile looks like he's a competitive applicant for ML engineering positions or perhaps a fellowship/residency/PhD program.
So, a junior researcher at the level of a decent second or third year PhD student. A researcher, maybe someone you'd trust to build a prototype or product, lots of potential, but probably not someone you'd trust to run a research program.
> I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to self-fund your research.
Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020? How many are going to place in a Kaggle competition with just half a year of experience? More importantly, just how many people are privileged/secure enough to put their all into learning, with no sense of security or peer support?
> I started working with Google because I reproduced an ML paper, wrote a blog post about it, and promoted it. Google’s brand department was looking for case studies of their products, TensorFlow in this case. They made a video about my project. Someone at Google saw the video, though my skill set could be useful, and pinged me on Twitter.
So what really mattered was self-promotion, good timing, and luck.
> Tl;dr, I spent a few years planning and embarking on personal development adventures. They were loosely modeled after the Jungian hero’s journey with the influences of Buddhism and Stoicism.
Why does the author have to present his life like one would in a fucking college essay?
[0] https://medium.com/@andreas_madsen/becoming-an-independent-r...
Yes. He seems like someone who is good at self-promotion and networking. Well, good for him, but I think he underplays the role these have in his success.
> Why does the author have to present his life like one would in a fucking college essay?
I guess that's the self-promotion. And humble-bragging. Like this bit:
"I started working as a teacher in the countryside, but after invoking the spirit of their dead chief, they later annotated me the king of their village."
Exactly. Good for Emil, but it's always frustrating to hear survivorship bias preaching. Even the interviewer starts off by saying:
"By the way, I really love your CV - the quirks section was especially fun to read."
It's even more frustrating when I hear non-POC's talk about their journey to some non-western country (and subsequent conquering of fantastical goals like gaining the approval of locals) or pursuit of some sense of foreign culture. It's almost a given that they have internalized and appropriated the ideas (i.e. Buddhism or even worse post-retreat Buddhism). Good for the author to receive such positive feedback for such signaling, but it makes me sad to know that I might not receive the same.
If you want to talk about white people say white people.
I don't think the idea is to look for an internship after the course but an additional 4 months of personal projects. After applying state of the art deep learning for 4 months full time you'll have some very cool projects, and you could probably convince some company to take you on as an intern for a certain amount of time.
> Early evidence of practical knowledge often comes from usage metrics on GitHub, or reader metrics from your work blog. Progress in theoretical work starts by having researchers you consider interesting engage with your work.
> Taste has more to do about character development than knowledge. You need taste to form an independent opinion of a field, having the courage to pursue unconventional areas and to not get caught up in self-admiration.
When I study abstract interpretation or lattices, I'm doing so because I find those subjects interesting and beautiful, and studying math relaxes me. I can lie to myself and say that it's improving my problem solving ability and that it's like doing mental yoga and will make me better at my job or some baloney, but that's not why I do it.
I can spend time with a plant in my garden, take a cutting, root it and replant it, and watch it grow, learn the ebbs and flows of its watering needs through the seasons, learn what its seed pods look like, and eventually watch it die through some misstep of my own or otherwise.
And in doing so, I am learning, and building a mental model for this plant and an intuition for it, but I'm not "creating value" in some weird capitalist sense, which I feel always underlies these sorts of opinions about learning and education, and people who self-identify as "makers" in general. It rubs me the wrong way because it encourages a very narrow view of the human experience and what it means to learn and why we should learn.
Seems to me the difficult part is how to support yourself financially while spending your time doing interesting learning and research, or how to get paid to do it
Maybe the most important detail in the story is "He co-founded a seed investment firm that focuses on education technology" but it is not discussed further
FAKE.
They're free to do so of course, but they should not give advice based on it.
Regardless of that, I suppose the bar for being a "researcher" has been stooped so low. According to this guy publishing an ML paper is equivalent to writing a blog post or making a video about "AI".
Either way, this whole focus on "portfolios are everything and credentials are meaningless" spits in the face of all the work I did to get my university education. And it didn't involve "copying assignments". And you come out with one hell of a portfolio if you take your education seriously.
I mean I don't think self-educated people are without merit. I happen to think they're really important. But I only ever see them rag on higher education, despite them having "never been there".
Just another example of wunderkin super genius knows all because he was able to follow a non-standard path and make it. Glad he was smart enough to become a Google employee. But I question whether he should be giving advice on paths to get there when there's always many paths to a position. And especially after reading his brief comments on how credentials imply you're a liar.
Then actually pay attention to the arguments they're making instead of talking about how offended you are because it goes against your self-interest as a degree holder. It's not as if the people bashing modern education are some kind of elusive minority.
I've got a master's degree and I've always though our education system is stupid, and at least in the U.S. not unlike a giant pyramid scheme given the cost of tuition these days. Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
This is categorically false. Face-to-face time with an expert is incredibly valuable and incredibly expensive outside of an academic setting. In fairness, you have to show some initiative in college to get quality face-to-face time with a professor, but it still takes a lot less motivation than self-studying a complex subject for a nontrivial amount of time.
Which brings me to my second point. There's an enormous amount of free stuff you could learn from. But actually doing it is a completely different matter and the overwhelming majority will fail. For instance, the bulk of a university-level education in pure mathematics is over a century old, and free resources are easy to find. With stackexchange, you can even get expert feedback on your work! Yet most people who try (who are already a very self-selected sample) do not in fact succeed in teaching themselves undergraduate level mathematics. Even Ph.D. students taking a few years off for whatever reason find it highly (but not impossibly) difficult to do any significant amount of self-study for a prolonged period of time. And these are precisely the people who are training to become independent researchers!
Just because talking to an expert is valuable doesn't mean you can't learn it for free on the internet. Also most undergrad curriculums are teaching old stuff - not exactly cutting edge knowledge requiring face-to-face one-on-one time with an expert in your field.
It's not as if the alternative to 4 years of undergrad and $100-250k in tuition + living costs is just teaching yourself the same arbitrary curriculum alone in your room for 4 years getting a degree in some random field learning things you never actually use in the real world. One could instead intern or work, and not only potentially learn significantly more relevant and lucrative real-world skills for free, but actually get paid to do it. A business student could instead work directly for entrepreneurs and use that tuition money to start their own ventures.
Most people use very little of anything they learn in school after they graduate. For example I majored in math, and now as a software engineer I don't use any of that. I know some math majors will try to rationalize it by saying they learned "problem solving" skills or whatever but there are a million other more useful things I could've done instead of what I did in school. Everything I learn now as a software engineer I either teach myself or learn on the job. There is no curriculum that could prepare me for what I do now because by the time the curriculum is written, it would be outdated (well perhaps such a curriculum of "fundamentals" could be constructed, but the CS curriculum is not it).
The system is outdated, inefficient, and a downright pyramid scheme scamming the youth into indentured servitude in the U.S. If tuition was reasonable and having a college degree wasn't required for most jobs then I wouldn't be as critical of college.
However, the university I went to, could be classified as "No-Name" and I use what I learned in school almost everyday. In fact, I used K-maps to help a senior engineer struggling with a complex logic problem by simplifying it. The CS fundamentals I learn prevent me from writing ugly code and at least give me a sense for what's slow.
I also went to a university with a built in co-op education program where you got credit and paid for being an intern. And let me tell you, most companies treat interns like shit. They sometimes don't even bother having them do anything besides mediocre grunt work. My intern experience was not the greatest and arguably worse than my college experience. Most the time I was left on my own having no idea what to do and spent most of it reading programming books. Whatever "real-world" skills I picked up, like doing actual projects, was moot.
But again, it's mostly relative. So making categorical statements like "universities are useless" and "credentials are for cheaters" doesn't really help and certainly doesn't speak the truth.
The alternative isn't just being an intern taking on grunt work, there are many who forego college to work full-time jobs in industry.
Also,
> It's not as if the alternative to 4 years of undergrad and $100-250k in tuition + living costs is just teaching yourself the same arbitrary curriculum alone in your room for 4 years getting a degree in some random field learning things you never actually use in the real world. One could instead intern or work, and not only potentially learn significantly more relevant and lucrative real-world skills for free, but actually get paid to do it.
I mean, sure you can not go to school and do different things, and it might even be a good idea, but that's a far cry from the original claim, which was
> Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
Not only that, please tell me how many people can afford things like a mass spectrometer, fume hood, VNA, and other pieces of equipment that cost upward of 6,000+ dollars when they're in high school. If you want a real STEM education, you need to learn how to use test equipment unless you stick with CS or math, there isn't a lot of options to learn for "free". Even online courses cost money.
Sure there may be many people that get lucky and get into actual positions, but they're few and far between and are a direct result of success bias. The media shows you the thousands or so odd people that make it under extreme circumstances and never once mentions the people who never make it because that isn't "cool".
I'm not claiming that the best way to learn is alone in your room, I don't even believe that to be the case. The best way to learn is by working with people who know more than you. That is generally what happens when you work a full-time job. Sure in school you can learn from professors, but (1) what you learn is often divorced from the reality of professional work (2) the format of listening to lectures, completing busywork, and taking multiple-choice exams on random information could be Googled isn't the most efficient way to learn.
Yes I know it's difficult to find work without a degree because companies unfortunately discriminate against applicants without degrees. I wouldn't be opposed to a law that banned employers from discriminating against non-degree holders unless the job clearly requires someone with the expertise gained from the degree (eg. medicine, not law or marketing).
I don't know who "you" is (perhaps you in particular are very gifted) or what you personally learned in college, but on my end in college I specifically took particular classes to learn topics that I had previously tried and failed to learn on my own, so, if it's meant to be generic, I'm pretty confident your claim is false.
I went to a bad No-Name University for undergraduate and then Top Tier University for phd school, so I have an unusually representative view here.
I do agree that, for CS, the no-name university was... bad. Fortunately, I realized this early and did a lot of self-study. I probably learned more reading taocp and going through MIT open courseware courses in the library during the evenings than I learned in my actual undergraduate courses.
The mathematics courses, even at No Name, definitely provided me with a better education than I could have ever gotten on my own. I'm pretty bad at math, it was my worst subject in high school. So I double-majored in it during undergraduate. This dovetails with your advice to use university as a time to learn things you already tried and failed to learn on your own, or which you otherwise know will be difficult to learn on your own.
The CS education that undergraduates get at Top Tier University is far better than what I got, even though I worked through that Top Tier University's online courseware/lecture notes/exercises during undergraduate on my own.
My hot take: college is always worth it, but only if you intentionally invest in "leveling-up" past your previous potential.
That will happen almost by default at Top Tier unless you're a genius (...but you'll pay a lot for it). But not if you're going to university at No Name. So, in that case, students should definitely a) minor or even double-major in something they're not good at, and b) heavily supplement their CS courses during evenings/weekends.
Also, this is all highly specific to very self-motivated learners -- the sort for whom "self-taught" is a reasonable route. I'm one of those people. Over time, I've realized that we're a very small minority. Our perceptions of what others are capable of learning on their own, and prescriptions for how others should learn, are typically quite warped. Most people probably do need something like a 4 year college degree to become a competent programmer.
But then again, how many people majoring in biology or chemistry actually end up using anything they learn? I'd wager it's probably a small minority.
Do you have any idea where the majority of massive breakthroughs in technology come from? They come from university.
AI started in university before it was even a thing. Autonomous vehicles were a university funded DARPA project. Nearly everything, even this guys research, is an important derivative of this.
Why do you think this guy chose to publish his paper in a academic journal? It's because it's peer reviewed. The Internet is not peer reviewed, it's public reviewed, as in anyone with an opinion can say whatever they want and get a million other opinions accepting or rejecting that opinion with little evidence. That's essentially what the entirety of Hacker News is. Very rarely do I see a post, including my own, that's properly sourced.
Let me ask you, do you think it's stupid that I paid money, like most people, to build a solar power management system? Do you think it's stupid that I built a memory management system, and text message system from bare bones hardware? Do you think it's stupid that I built my own shell? I did all that in school with equipment that I could only dream of owning with people that spent more time helping me than writing blog posts trying to get famous.
It's funny, I was actually glad this guy got where he wanted. It must be nice to be a genius. And to be honest, I'm probably not as smart as this guy. It's great that he had a lot of drive and achieved greatness. But he doesn't need to imply that I'm some liar because I chose to go to university. I worked incredibly hard to get my degree, spending hours and hours in the lab doing assignments. Hours thinking I was dumb and that I'd never make it. Months trying to find a job.
And all I see is people that did it a non-standard way and then have these warped views of the traditional way despite the decades of people lifted out of poverty because of it. And despite having never even attending a university! I see all these smart people and how they're the only ones that matter. College did a lot for me. And I try and do my best every single day because of it.
Sure, I may not be able to write an ML paper in a year like this guy, but that doesn't mean I'm not going to defend myself when he essentially implies I cheated my way in because I got a degree.
I'm not sure I have heard anyone claim that "higher education is useless" in general. Education is never useless in general.
Perhaps you are confusing increased income with being higher up on the income ladder? It is true that, statistically, those with higher education do find themselves higher up on the income ladder. They are not making more than they did before, when they did not have higher education, however. Incomes are stagnant.
All we're really observing there is the fact that people range from more to less able, from geniuses who seemingly can do anything to those who have crippling disabilities. At one end of the spectrum you have the people who do well in school and also the workplace due to their natural ability, and at the other, those who struggle in everything they do, be it school or the workplace because of their disabilities. And then everyone else somewhere in between. Statistically, the more able will find themselves higher up on the income ladder, and able to go further in school, thanks to being more able.
How you can reconcile a higher educated person being higher up the income ladder, but not making more than what they made working minimum wage at a lower education level, eludes me.
What stagnant income means is, they aren't making more relative to their productivity. Meaning salaries have not changed all that much for about 2 - 3 decades through raises despite high productivity.
I think I see the issue here. I am talking about the population as a whole, you are talking about an individual. It is true that over time individuals tend to move up the income ladder. And that more capable people move higher up the ladder thanks to being more capable, free of disability that hinders their achievement. However, the steps of the ladder have remained unchanged over decades. Making more than someone else is not the same as making more than you otherwise could have.
If we break that ladder up into percentiles, the person at the top of, say, the 70th percentile made x of number dollars in 1970 and the person at the top of the 70th percentile still makes x number of dollars today, in real dollars. In 1970 that person did not have an education above high school. Today that person does. Despite promises, their income did not increase with increased educational attainment. Incomes are stagnant. There was no advantage to gaining that higher education with respect to income. The person at the top of the 70th percentile, who got there because of the abilities and constraints they were born with, would have ended up there regardless.
> What stagnant income means is, they aren't making more relative to their productivity.
What stagnant income means is that incomes are literally not changing, relative to inflation. On average, if you made $1 last year, you will make $1.02 this year, assuming a 2% inflation rate. A real increase of $0; stagnant. To put it another way, incomes, in nominal dollars, are increasing at the same rate as inflation.
https://cew.georgetown.edu/wp-content/uploads/Exec-Summary-w...
Inflationary dollars are largely irrelevant as everything is more expensive than it was decades ago due to inflation.
That is not the same as incomes increasing because of higher education. Incomes have held stagnant for a number of decades, even as more and more people attain higher and higher levels of education. If 0% of the population had a higher education or 100% of the population had higher education, your income would remain the same as it is now.
McDonalds isn't going to pay someone flipping burgers a million dollars more over the span of their burger flipping career simply because they attained a physics degree. That is not how the economy works at all. If 100% of the population had a physics degree, someone is going to be left flipping burgers. 100% of the population are not going to be working on solving string theory. The economy cannot function in that manner.
Inflationary dollars are relevant as that is how we measure stagnant incomes. An income that tracks inflation is stagnant. Incomes are stagnant. We are not talking about spending.
Are people with higher education in low wage jobs? Yes. Is this common? No. And the only reason it would be common is if there was no income incentive to gain higher education, which there isn't.
I'm beginning to think you really don't understand what I mean.
What choice would they have if 0% of the population had a degree? This is ultimately why incomes haven't changed even as more and more people attain higher education. In the past the best and brightest people were hired into those positions as high schoolers. And now the best and brightest still are, except they have higher education now because of the social pressure to attain a higher education.
> And no business will pay a physicist with a degree minimum wage.
They absolutely would. Of course they would. I have no idea where you got the idea they wouldn't?
A person capable of becoming a physicist has little reason to want to work a minimum wage job though. The fact that they can attain a physics degree means that they do not posses the limiting qualities (disabilities, poverty, lack of intelligence, etc.) that leave less able people stuck in minimum wage jobs. People who are burdened with certain disabilities, poverty, lack of intelligence, etc. never had a real chance of completing physics degree. It is not within their ability.
> The result of obtaining higher education categorically results in you earning more income.
No. Those with higher educations, statistically, earn more than those without higher educations, but they are not earning more than people in the past who did not have higher educations. Additionally, this correlation exists because people who failed to attain higher education have qualities that limit their success in both school and the workplace.
> I'm beginning to think you really don't understand what I mean.
No, I fully understand that being able to excel in school is correlated with being able to excel in the workplace. This is obvious. Someone with crippling autism, which did not allow them to graduate from high school, was never going to become CEO of Google. I get it. That does not mean dropping out of high school will cause you to contract crippling autism.
It is well understood that education acts as a filter, leaving the poor performing people, who also perform poorly in the workplace, behind in academic achievement. This is quite different to school causing someone to become a high performer.
nice to be able to work for free and not starve
1) There is the general phenomena or collective project, where hardware, algorithms and human insights are improved to approach the situation of man-made intelligent machines.
2) There are the people who are designing algorithms, using mathematical intuition and knowledge, analogies with physics, etc... Most people would agree these people are doing optimization / machine learning "proper".
3) There are the people working on improving hardware for machine learning / optimization purpouses, by looking at the most performant algorithms, breaking them down into primitive operations and requirements for hardware, there are also people working on the algorithms themselves and finding computational shortcuts (which can end up in software or hardware, can end up as proprietary knowledge or common knowledge, ...). The distinction between hard and software is somewhat blurry, since hardware designers can optimize or implement a section of software into hardware. A lot of this can still be considered ML "proper".
4) Then there are the people who apply the ML frameworks and their exposed choices and settings to a specific problem domain. Many of them don't need to understand the internals if they don't need state of the art results. Many would nevertheless benefit from understanding the internals, and the requisite math. What I propose is to stop calling their activity as Machine Learning, and instead call it Machine Teaching. They are teachers, and just like elite schools they can choose which specific type of available student they will teach, and they can tweak (or filter from a large family of students) which student they select to teach the task at hand. There are bound to be many advantages of having actual human teachers get involved in machine teaching. These people will not be proficient in designing novel families of students unless they also know the requisite math, and identify those ML papers that are ML "proper" instead of ML "teacher". When trying to find important foundational insights in ML "proper" one is typically overwhelmed by a large surplus of ML "teacher" type papers. These are important datapoints, and necessary to advance human insight into ML "proper", but they are data, not knowledge. There are actual ML "proper" knowledge papers out there that explain why a certain phenomena is such and so, and they get very little attention because they necessarily lag the breakthrough ML datapoint paper, and most ML "teachers" don't have the math background to understand them. So the probability that a given ML "proper" researcher fundamentally improves the state of the art is much higher than the probability that a given ML "teacher" will fundamentally improve the state of the art. At the same time the probability that a given fundamental breakthrough was achieved by an ML "teacher" is higher than the probability that a given fundamental breakthrough was achieved by an ML "proper" researcher:
P( Breakthrough | Proper ) > P ( Breakthrough | teacher)
while
P ( Teacher | Breakthrough ) > P ( Proper | Breakthrough )
Since most people don't have the broad math / physics / ... knowledge to draw on, the number of ML "teachers" is much higher than ML "proper" researchers.
[1] well, really, some actors have vested interests in conflating those together...
EDIT: just to be clear, I am not complaining about ML Teachers, we need the ML Teachers, and their breakthrough datapoints. What I am complaining about, is conflating both activities of ML Proper and ML Teaching. This makes it harder for the few ML Proper researchers to find each other's insights.