Lessons from a year of AI research
jetnew.notion.site
jetnew.notion.site
Here's my 2 cents on the topic from a thing I wrote last year ('Lessons Learned the Hard Way in Grad School (so far)'): https://www.andreykurenkov.com/writing/life/lessons-learned-...
I think there may be some who breeze through grad school, likely by being in a strong research environment beforehand, or by having lots of support. I mean, they have to exist?
Have I genuinely met anyone like that? Nah. And, oh boy, can it be a struggle!
During a PhD, it's very easy to put yourself into an increasingly hopeless situation that is mentally tough, for years on end, and it is really hard to even convey to anyone outside of academia why you would do that.
It is especially hard to convey to people outside of academia how 5 (nowadays often 6,7, etc) years of work all come down to a few, often seemingly random decisions of hiring committees and other structures.
You start grad school thinking it's something you build, bit by bit. At then end, it's more of an ever shifting collection of you, put together in the hopes it will be evaluated well. But then you realize, it really has little to do with objective quality, and much of the situation is outside of your control. There is no short project in grad school. It's the entire thing!
I think the worst thing is the uncertainty about literally everything. Is your Professor ever telling the truth? Will you really get that support two years out? Will the funding persists? What are you even researching?
I have seen people had the work of the past five years ripped apart during their thesis defense (or wherever), I have seen people make a few missteps and ruin their chances on the job market. I have seen people invest a lot in work that never makes any impact. I have seen people who thought they'd make Professor, and then found they have zero motivation or talent for research and/or teaching.
You can never be sure that ain't you, until you have your PhD and your job.
It's always on your mind. Projects aren't short anymore, you work years on stuff before it ever becomes a paper. You work every waking hour and often every sleeping hour as well. And you lack the knowledge and experience to assess whether what you are doing will really go anywhere, if the network you are building is right, if the people you write to will be interested in your work. At the very least, you never know who else starts with you and will compete with you on the job market, which is increasingly filled with people willing to work for little money and little job security.
In the past two years, academic positions dropped 70% or more, at least in some fields. And we have 2-3 cohorts of applicants on the market due to Corona, and this will remain for years.
Academic research nowadays requires nerves of steel, and I have seen grad school ruin the health (mental and physical) of quite a few people, all of them very good at what they do.
So after all that doom and gloom, people need to realize that it is a struggle for probably everyone. But it is also rewarding, or it can be.
There you go.
pumpkin 3.14
Normally I prefer apple, but during the holidays you must adjust.
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
“What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
“Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said.
Minsky then shut his eyes.
“Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.”
At that moment, Sussman was enlightened.
Proper methods of bias analysis are network compression or random surgery, and expanding test data set. Proper methods of training are robust to varied initial states.
If you randomize your seeds, you do not deliberately bias your model. This does not mean your model is unbiased.
>If the title contains a gratuitous number or number + adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
"You Can Now Save Money with Y New Strategy”
"You Can Now Travel Abroad Without Having to…"
"The Last … You’ll Ever Need"
"You Won’t Believe… What Y has Found"
"Why You Should…"
"Why You’ve Never Heard of This Top Travel Destination"
"This is why you’re losing money"
Just a few examples.
The irony is that once you solve a problem, it's not a problem, so people's natural reaction is to only call things AI when they're unsolved. Once it's solved, "that's not AI".
On the Measure of Intelligence, François Chollet https://arxiv.org/abs/1911.01547
Why not? Is there a strong payoff later? How did you learn this lesson?
:)
Good listicle!
Let me tell you that I've finally made it to the stressful part of the being a serious "AI" researcher, where I have a real project (as in difficult to achieve goals, not just "turn the crank" stuff) and real deadlines (deliverables on collaborators projects and my own conferences submissions) and the only thing I prioritize above doing the work itself is keeping my advisor (and other collaborators) up to date on what I'm doing so that when he reads my paper draft he's not completely lost. Everything like organizing papers, citations, logging infra, etc is meaningless when you're trying to piece together a solution. Like seriously somedays I barely have time to exercise and eat dinner with my wife (let alone organizing my bookmarks).
For example I'm trying to solve a particular compilers problem using integer programming (note that at a high this isn't that high level because this is a small cottage industry) and so I have like 50 paper tabs open that I bounce between when thinking/experimenting. The way it usually goes is I'll hack, get stuck, go back to the papers, find something, hack, and on. And usually the eureka moment comes some hours later because I connect something.
You might say that I'm a bad researcher but I know for a fact (external validation) that I'm not. And if you look at other highly productive researchers (like TT track profs at my "elite" school) this is indeed how they work. All of this zotero, notion, mlflow stuff is of the ilk of productivity porn for other flavors of knowledge workers (ie a mirage and/or snake oil). Let me put it this way: my advisor is a top 500 h-index person (the exact significance of that metric notwithstanding) and he doesn't have a bibtex of his own papers, let alone zotero for all of the papers he reads/comes across.
The only thing that matters is code/math/etc output (whatever your material output is) and your abilities are also highly correlated with it with the casualty flowing in t opposite direction (make more stuff and you'll get better at making stuff).
But I guess conversely do do some of these things when you're young and have the time (and I don't mean that condescendingly). E.g. reading outside of your area is probably the most valuable (from my own, admittedly a typical, experience, since I jumped domains many times); I very frequently can outpace even my senior colaborators very quickly on understanding a problem and solution simply because when I was younger I dabbled in ... all the things (physics, math, cs).
The other thing that I'll say is there's something obviously missing from this list but only if you've really made it this far: collaborators and interactions with collaborators. The only thing that matters aside from the produce is getting people to make use of it. That means writing, speaking, and getting buyin from your collaborators. If you really truly want to be successful then work on your people skills as it pertains to this area - that means learn to speak the language of your research community, learn to give good (engaging, interesting, useful) presentations, learn to write well (including making nice diagrams), and learn to explain things in ways that smart but busy people will understand. Besides all of this being key to being productive it's also what feeds you (i.e. the real #1 priority) since it gets you jobs, academic and industry.