Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters”
reddit.com
reddit.com
and so on.
I would guess it is possible to go through this list if you make it your full time job for a year or so. That might be worth it depending on your priorities. Regardless, it bears mentioning.
Sounds like a good MSc program
There is no need to direct negativity toward others for making time to learn. Carmack is in this position today because of such work ethic, not in spite of it.
Work isn’t the only responsibility.
What is the correlation between taking a week away from responsibilities - work, family, etc. - and job satisfaction?
I do not mean telling family that I am disappearing for a week starting tomorrow, but that I want to disappear for a week in a couple of months and the family would cover responsibilities for me (I should be willing to reciprocate).
Many people have a lot more flexibility than they let on. They just make excuses instead of acknowledging their own priorities and commitments or lack of actual interest in change and work.
Of course work isn't the only responsibility.
Of course some people have kids, pets, parents, medical conditions, etc.
It's one week away from work. It can still be done. You weren't going to be taking care of any other responsibilities during your normal 9-5 hours, so that should have no bearing on the week you take off; treat it as any other work week, except you're working for yourself.
Have you ever been married, had kids, etc? Pretty much any time I take off of work, it involves catching up on projects, chores, or parental responsibilities during the day. It seems the vast majority of people I've spoken to tell of similar situations.
I was homeless from 16 to my early 20s. I suffer from multiple traumas, both physical and mental. I have ADHD and bipolar disorder. I've had sciatica, degenerative disc disease, gout and other issues since my teens. Most days I can barely walk. When I do have kids, I'm not going to get to be the dad I want to be, running around and playing with them. My parents are abusive drug addicts who were not present in my life, so I was raised by other extremely abusive people instead, who kicked me out into the streets for being atheist. So malnourished as a child that my friends would make fun of being able to see the bones in my face and body. I spent years living homeless on scraps of food.
I have to advance my career. Every single thing is stacked against me, trying to make me fail. I don't get the luxury of deciding if I want to take a week off to do so. It's a matter of survival, and I have very purposefully structured my life in such a way that I have a shot in hell at one day raising a child above the poverty line.
Believe me, this is not a matter of me not empathizing with others or understanding the complexities of life. It's a matter of prioritization and commitment. If I can make time to learn, anyone can.
And on top of all of that, I do in fact have to provide for my sister and her four young, expensive children, because she doesn't have anyone either. I don't get much free time, man.
I'm still learning where I should set boundaries on both sides of the field, and this will probably be something I continue to struggle with for some time.
I think you didn't read the parents carefully. The comment being replied to is not about 9-5 hours, it's about "locking yourself into a hotel for week", and if you have kids, this is not exactly a healthy thing to do in the face of other responsibilities. That's the point being made here, and you are changing the goal posts.
The negative sentiment itself is what I addressed. Everyone fortunate enough to have internet access, a computer and a functional brain/body can make time, if they really want to, to learn new things that benefit their long term goals.
That's what I mean -- you didn't read the parents carefully. The commenter (me) made that reply to this:
> Carmack did it an a week by locking himself in an hotel room
Everyone can and should spend time learning, but the little point being made was that locking yourself in a hotel room is not something most people can do due to responsibilities. Somehow you changed this into your more general statement "spending a week learning something."
It's downright silly to claim that I have no life responsibilities because I prioritize setting aside time for self-growth and encourage others to do the same.
Open your mind a little bit and ponder why you feel compelled to have such a negative and dismissive attitude about this.
> If you can't take a single week off of work during an entire year in order to advance your career, then you're satisfied with your current one.
It’s a pretentious remark, saturated in lofty assumptions about people, and you could learn from these reactions.
Edit: I just noticed your username, “soul of mischief“, and now it makes sense to me. Are you trolling?
> I just noticed your username, “soul of mischief“, and now it makes sense to me. Are you trolling?
It's against the rules of Hacker News to assume bad faith. I certainly am not arguing in bad faith, nor do I intend to make anyone feel bad for the sake of it.
I purposefully wrote an abrasive comment in order to stoke the minds of others and make them think inwardly. There was one time in my life where I also needed to hear those exact words. That no matter what adversity I face, I must make times for the things I really care about.
I must constantly re-evaluate what drives me and check to make sure I'm not letting life pass by without achieving at least some of my goals. I must not make excuses, and I must not make excuses for others. I must not be negative toward those who are in a better station of life.
As for my username, it is derived from a popular hip hop group, Souls of Mischief, who are part of the larger Oakland set, Hieroglyphics. https://www.youtube.com/watch?v=fXJc2NYwHjw
But in general I can do what I want. This month, it's been relearning awk to do financial data analysis and reading the books of a British author, Diana Athill.
I'll say that my personal project satisfaction level is medium. Life satisfication has a major social component that provides meaning, though. Optimizing for both isn't necessarily easy even in retirement.
I'm going to guess you don't have kids.
Tony Stark: Last night.
The Avengers (2012)
Of course not being able to ask someone any questions would be disadvantage (but is that even a 'rule' of this scenario? Surely at least the internet is allowed, i.e. a person to ask who's available to anyone), and it would have been a lot easier while at university than with X years' rust on the background maths etc. - but still, seven full days is a lot of time if you choose to completely dedicate it to some particular thing, we just generally don't do that, so it doesn't sound it.
Maybe someone should ask Chuck Norris how long it took him. Who wants to go first?
Anyone can read an absurd amount of text in a week. To absorb and understand it all is a different matter entirely.
Seems more likely to me that Carmack simply convinced himself he'd understood it all. He does seem to think very highly of himself.
I would not bet against his understanding on technical topics where he's done a serious deep dive. Also it was more than a year, not just a week.
[1] https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe...
[2] https://twitter.com/keshavchan/status/1787861946173186062
https://twitter.com/andrew_n_carr/status/1752526711311507526
https://nlp.seas.harvard.edu/annotated-transformer/
https://scottaaronson.blog/?p=762
https://karpathy.github.io/2015/05/21/rnn-effectiveness/
https://colah.github.io/posts/2015-08-Understanding-LSTMs/
https://arxiv.org/pdf/1409.2329.pdf
https://www.cs.toronto.edu/~hinton/absps/colt93.pdf
https://arxiv.org/pdf/1506.03134.pdf
https://proceedings.neurips.cc/paper_files/paper/2012/file/c...
https://arxiv.org/pdf/1511.06391.pdf
https://arxiv.org/pdf/1811.06965.pdf
https://arxiv.org/pdf/1512.03385.pdf
https://arxiv.org/pdf/1511.07122.pdf
https://arxiv.org/pdf/1704.01212.pdf
https://arxiv.org/pdf/1706.03762.pdf
https://arxiv.org/pdf/1409.0473.pdf
https://arxiv.org/pdf/1603.05027.pdf
https://arxiv.org/pdf/1706.01427.pdf
https://arxiv.org/pdf/1611.02731.pdf
https://arxiv.org/pdf/1806.01822.pdf
https://arxiv.org/pdf/1405.6903.pdf
https://arxiv.org/pdf/1410.5401.pdf
https://arxiv.org/pdf/1512.02595.pdf
https://arxiv.org/pdf/2001.08361.pdf
https://arxiv.org/pdf/math/0406077.pdf
https://www.vetta.org/documents/Machine_Super_Intelligence.p...
wget -A .pdf -r -H -D arxiv.org -nd https://punkx.org/jackdoe/30.html
Keep in mind there is no confirmation that this is the actual list.
Reading through the list, I can barely parse half the titles due to acronyms and terms I've never seen before. Would I be able to figure them out along the way? Or is there a body of knowledge this is assuming I should already have?
(This isn't a criticism of the list, or even the idea of presenting such a list without explaining who the target audience is and isn't. I'm genuinely wondering, if I just start at the top, is there any hope I would figure this out, or do I need to start somewhere else entirely?)
It's been 20+ years since I took calculus in high school, and I haven't taken a math or science class since. I reckon that puts me at least five years behind where I'd need to be to start this course and the listed ones.
Thanks for the recommendation, and helping me assess whether it's the use I want to make of my time!
If one has no relevant background, then I would 5+ years of full time study.
The papers have bibliographies with citations in text. The knowledge you need to read a given paper usually lies within 3 degrees of bibliographical separation.
You can also use reverse citation search on Google scholar to find newer works that cite a given work. This can uncover a review or textbook, which often makes the concept much simpler to digest. And it can be a more advanced approach which gives you an alternative view or lets you abandon the original paper altogether.
2. Back to tha basics
For math, it would help if you have college level calculus, differential equations, linear algebra, and statistics. If you don't have that, be ready to assign yourself math readings and homework. It would also really help if you had analysis, topology, abstract algebra, and category theory under the belt, just so you can see the mathematical forest for the trees... But that's not really so necessary
You can often find the author's preferred reference for a given work in the bibliography, and use that.
For computer science, it helps to be able to read algorithm pseudocodes, follow complexity and correctness proofs, and understand performance experiments and charts. Skimming Knuth's "Art of Computer Programming" should be very helpful.
You'll need some algorithmic/statistical learning theory as well, since authors often use these concepts to prove that a system can learn. Though TBH, I usually skip these proofs and focus on the performance details. I usually find a higher performance paper before I need to care about the learning proofs. Still it helps to expect them and have an idea of how they work, if only to know when you can safely ignore them.
3. Narrative Arcs
Many AI results build on keystone papers that define eras. We can build on these papers by understanding their philosophy and deeper implications, even if we don't really understand their math. There's a story that you will tell yourself, and it may lead to interesting research or business results.
For example, if you read "Attention is all you need", you set the stage for LLMs. then "LLMs are in context learners", followed by the "chain of thought", "reflexion", and "toolformer" paper will give you a great sense of how LLMs are emerging as more than just stochastic parrots, ready to be developed and deployed in a lot of ways that overcome their issues. But looking at how bad the numbers can be even with our best efforts, you also get the sense of how much work these systems will need to actually be good, and the sense that corner cutting will lead to a lot of pretty awful systems.
It's a fun time to be a computer scientist or software engineer! Lots of cool papers to read!
Thanks for taking the time to expose me to your enthusiasm!
Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? https://news.ycombinator.com/item?id=34641359
396 points by alan-stark on Feb 3, 2023 | 137 comments
--
John Carmack’s ‘Different Path’ to Artificial General Intelligence (dallasinnovates.com) https://news.ycombinator.com/item?id=34637650
378 points by cmdr2 on Feb 3, 2023 | 495 comments
--
Ask HN: Where can I find an updated AI reading list? https://news.ycombinator.com/item?id=33166948
3 points by costant on Oct 11, 2022 | 6 comments
This looks like a subtle way to advertise a VC funded web browser.
https://twitter.com/ID_AA_Carmack/status/1622673143469858816
That confirms the opposite, no?
There's no RL in there, nor diffusion. There are many other very interesting papers that I think have lots more to offer — but I just have a master's degree here. I am not a phd let alone a distinguished researcher.
Wouldn't that account for the other 10%? To me it sounds like the quote is not too far off when you consider that he said "of what matters today". There are many other things missing from that list that might become more relevant tomorrow.
Without in context learning, self critique and tool using, for example, it's easy to think that LLMs are stochastic parrots bound by their training distribution, instead of last mile learners that can adapt to out of distribution contexts. And business applications need RAG to overcome hallucinations. Without these concepts, I don't think people will really know 90% of what matters today, or be ready to take on what matters tomorrow.
I could be wrong here, with my focus on so many llm specific topics. Things like in context learning and RAG exist as engineering workarounds for LLM limitations. They are not definite paths to tomorrow's AI, which won't need our cajoling and hand holding. There's a possibility that we'll move past them, so more speculative and theoretical papers are better.
But I think there's also a possibility that the AI scaling is done for this AI bubble, and we're stuck with building larger scale systems out of imperfect buggy LLMs for the foreseeable future. In that case, maybe some next generation Zen of integrating LLMs into a working mind eventually emerges as a way forward.. so we should pay attention to engineering efforts around LLMs!
Interesting facts: published date ranges from 1993 (the only item from the 90s, by his advisor) to 2020, types varies from 1 course, 1 dissertation, 10 preprints, (1 is a preprint of a chapter of a book, some are presented somewhere), 11 conference papers, 1 book, to 3 blog posts.
At least this is relevant information for people who are actually interested.
https://github.com/thomasj02/AiFilter
https://github.com/GoogleChrome/chrome-extensions-samples
https://huggingface.co/docs/transformers/en/tasks/sequence_c...