How Might We Learn?
andymatuschak.org
andymatuschak.org
It is especially frustrating when I recently tested GPT-4o on a factual question and got 1000 words which were all completely wrong, including fake citations.
It is especially frustrating to read this sci-fi daydreaming after talking to a high school science teacher who was forced to use generative AI tutors in their classes this year, even though these tutors are poorly tested and seem to have a ~20% confabulation rate. This particular teacher is technically sophisticated but even they sometimes get confused and misled by the chatbot. Students don't have a chance.
I think Matuschak has valuable insights on learning in general. But it seems incomplete to go through this AI thought experiment without discussing how inadequate current AI is to the task. "Technology will get better" but what if it takes 50 years?
The high school AI tutor probably wasn't using GPT-4, but the district definitely paid a lot of money for the software.
I also hate this entire argument, that AI confabulations don't matter for free products. Unreliable software like GPT-4o shouldn't be widely released to the public as a cool new tech product, and certainly not handed out for free.
I like your term for it.
I have started double checked with web searches more and more. It went from 100% of the time to only 20%
I started building a prototype of this idea that I've been very slowly working on in my free time that indexes and uses my notes in emacs for RAG against a locally running LLM. I do think these kinds of learning LLMs have to be run locally, though I've recently gotten a little frustrated because I cannot run a capable open model without my machine's fans turning on.
Learning music is one of the best areas to learn how to learn. When you start a new instrument or technique, you will not be good relative to experts. That's ok. You just need to focus on what you can do and build from a solid foundation. Listen closely and your weaknesses will become apparent. You can even learn to appreciate your weaknesses as they provide the opportunities for growth and development.
Unfortunately, I worry that ai tools will ultimately hinder learning as much as help (at least in the aggregate). My fear is that it will prevent people from exploring and finding their own path, passively following the track laid out by the ai. Inevitably many will compare themselves to ai and find themselves wanting instead of asking what they can do that an ai can't.
Teachers can be helpful, but you ultimately are responsible for your own education. It is one thing to follow an individual who has a proven track record, but at least at this stage, I feel like ai tools may be more pied piper than wise sage.
My experience learning with LLM-based tools over the past ~18 months has been the opposite. The ability for me to redirect the learning path in a direction that best matches my own interests and curiosity is unparalleled. It's that ability to directly influence the learning path that makes me do optimistic about LLMs as a tool for learning.
Every single one was an extremely life-threatening moment in which I very likely would (and should) have died, but for my very rapid learning. The growth came in not being a corpse.
I realize I am not the target audience of this article.
Necessity and lack of control in a novel and frightening situation transforms our minds. It isn't sustainable, but it's not meant to be, because the goal is to get through it and get out, by becoming a different, better person.
To pick some slightly less dramatic examples than combat or wilderness survival;
A person who learned a new language in 14 days because they fell in
love with someone who spoke almost no English, but simply had to be
with them and make things work.
Someone who became an expert bricklayer when stuck in a remote
village where that was the only skill they could contribute.
The article I just read sadly describes more scaffolding, more mollycoddling, more "learning on rails", but "Now with added AI!"
[0] https://en.wikipedia.org/wiki/John_Taylor_Gatto
[1] https://thesunmagazine.org/issues/186/a-few-lessons-they-won...
So, life.
> It isn't sustainable, but it's not meant to be, because the goal is to get through it and get out, by becoming a different, better person.
So, again, life.
I loved kindergarten and first and second grades which mostly seemed to be play. I think it was effective for me as far as creativity and socialization goes.
From third grade through high school I was bored with most of the material. A lot of it was not interesting and sometimes the pace was too slow.
At university, the work load was too high. I think if I had taken six years to complete the 4 year program, I would have been a lot better off. Too often I didn’t have the time to really dig into the material and explore related ideas (side quests). Instead I settled for memorization which was enough to do well on exams. My GPA at graduation did not reflect my command of the material.
Like Andy, I think an AI-powered course of learning could be great. The strength, I think, would be its adaptability. If while learning topic A I stumble across an interesting idea, it would have no problem with changing course and running down topic B.
This perfectly describes my experience too. My learning style is to 'ride the wave' of curiosity where i am obsessed with something and keep digging into it. Uni learning was antithetical to this learning style with strict schedules and tests. I got mostly As and some Bs by doing what you did but i didn't learn much of anything.
My son is somewhat like me and feel kind of sad to see him at a university to earn a living. I feel disappointed that i didn't provide him enough financial freedom to really gain enjoyment from learning istread of grinding at a uni.
For me, two use cases of "digging" come up a lot: 1. I want to know how the concept I'm learning can connect with other concepts that I'm interested in (i.e., related concepts); 2. I want to know what other materials are available out there that can provide different perspectives (i.e., related resources). So I ended up building a map visualizing concepts (https://afaik.io/) where the proximity indicates relatedness and under each concept there are various resources attached.
In addition to those more "objective" connections, I think what AI could really help is to find a more "subjective" connection that's very user-specific and utilize those connections to build a personalized tutoring experience, hence the adaptability. For now, I think the barrier to realizing that level of adaptiveness is the high hallucination rate.
While I see the value in it, the cost for me personally is also very high. I was surprised to hear that with just 10 minutes of practice he can sustain up to 40 new cards added every day.
I did quantum country a while back and I sometimes needed more than an hour to get through all the questions in a single practice session. Maybe my brain is just slower at recalling things.
[0] https://journals.sagepub.com/doi/abs/10.3102/003465431668930...
A lot of people get frustrated when they have been learning an item for weeks, answered the card over a dozen times, and yet still struggle to remember it. That’s because you need months for the SR algorithm to work properly. You really need to commit to it for the long haul for it to be effective.
Personally I can remember random words in obscure languages off the top of my head (instantaneously), purely because I added a card for them years ago and still keep current.
I have no doubt that spaced repetition is a biological reality in how our memories work but I haven't found a way to make it resonate for me.
The mistake most people make is simply adopting the "boring" flash card format of FRONT-BACK, and not incorporating other more creative types of cards. For example, question-answer, visualization, triggers, or unique images. ChatGPT is pretty useful for this, as you can get it to present the same information in a variety of different formats and contexts.
A concrete example from my deck: word added 2024-02-24, three reviews, now with an interval of 3.03 months. No 'again' answers; nice and easy! Another word, added 2024-02-29, not so good: 14 reviews and an interval of 15 days.
I think there are several factors contributing to forgetfulness with Anki for me, some of which might overlap with your experience:
A: not properly 'learning' the word in the first place. For me, 'learning' means using the word in context, studying its etymology (even if I do not intend to memorise that) and saying it out aloud. If I don't do that at the beginning, it won't really stick until I effectively start all over again.
B: 'learning' words on a bad day, or too late in the day. Even if I 'learn' the words properly, I need to have learnt them in Anki when I'm feeling moderately energetic. If I'm exhausted mentally or physically, my rigorous learning strategy doesn't seem to translate into memory. When I notice myself doing Anki reviews much slower than usual due to tiredness, I generally limit the review count and try to catch up when I'm fresh another day.
C: not being consistent enough with reviews. Both the time spent on each individual review and the time spent in total are important - 4 seconds per word is a good sign for me, and strictly 20 minutes a day in total. That allows me to keep up a pace of 12 new words a day very consistently.
Would love to hear which parts of this sound familiar to you, or what other things you've noticed for yourself!
This should (hopefully) get easier over time. From your dates there, it looks like the cards you have to re-learn often were only originally added 3 months ago? I think this should become easier for you in 6, 9, or 18 months – provided you continue to keep updated on the cards.
A. I definitely agree here. If I don't at least know understand a word to say, 30% confidence, I will never learn it, and will forever repeat it without making any progress. Personally I use images and sound to help make words stick in my mind. There's a lot of research about the effectiveness of imagery (see the "picture superiority effect.")
B. Ditto with #1. In scientific terms, this is called "encoding." Properly encoding things at the beginning has a big effect on your long term retention.
C. I do my reviews every morning while on the exercise bike. I use a gamepad to move through them more quickly, although I would technically be better with typing them out.
Also, as a side note, I have written a few blog posts on an old Substack about using Anki and AI tools:
https://neurotechnicians.substack.com/archive?sort=new
You might find the one about Using Images to Remember Things useful.
Whenever I fail a word, I try hard to find a reason for it. Most of the time it's interference - my answer resembles some similar word that I already know. I make a mental note about it, add this other word to the card. In most stubborn cases, some redundancy is good, I create another card for the same word in another context or just for a derivative of it.
Another thing that works for me is adding images (some of my cards just have a picture on the question side), and example sentences with the word in various contexts.
Matuschak writes:
> In my personal practice, I've accumulated thousands and thousands of questions.
> I spend about ten minutes a day using my memory system. Because these exponential schedules are very efficient, those ten minutes are enough to maintain my memory for thousands of questions, and to allow me to add up to about forty new questions each day.
It takes a while to formulate (and subsequently edit) a single good question (https://www.supermemo.com/en/blog/twenty-rules-of-formulatin...), but suppose we leave that aside and assume that by "using my memory system", Matuschak means answering previously-formulated questions. He says that questions should take only a few seconds each, so suppose it takes 10 seconds on average to answer a question. Then one could answer 40 questions in 400 seconds, which is under 7 minutes. That leaves 3 minutes to review roughly 20 questions of older material.
The gamification of finishing my queue doesn't work when it always has thousands of entries in.
My deeper beef with this method is the complete absence of emphasizing, discovering or forming connections between cohesive things. We're trying to learn, it's a super power to start seeing patterns in what we learn, it forms buckets that we can put new concepts and information in. Without it, the learning is ... shallow.
I found a better way. I map out full concepts to fit on single sheets of printer paper. Frontside has mostly words with lines connecting them or forming groups. The backside is for related drudgery (formulae, dates, numbers, names). I repeat new things everyday till I can reproduce the sheet front and back without any help. And then slowly introduce days of spacing between repetitions.
This is way more satisfying, no tech involved, no algorithms, just hard work and way faster. I do not have any evidence of this working long term. The things I put so much effort in learning to reproduce with such accuracy usually is useful in the short term only. So it works for me.
We've been doing it for a few weeks and she now knows ~100% of all countries and their flags. Just absolutely domination level learning.
I think it's a matter of finding things that fit Anki, and not trying to fit Anki to the thing you want to learn. Geography is a perfect application: we all would be a bit more informed by knowing all countries, seas, etc; and it's something that Anki is very well suited for.
I've also added:
- the numerical value for letters (A=1, B=2, C=3, etc) which I think will give me greater powers of lexical sorting. We'll see.
- NATO phonetic alphabet
- multiplication up to 12x12. I neglected/avoided automating that stuff as a kid and my confidence in doing mental arithmetic is still low. Not sure this is a good case for Anki yet...we'll see.
- A custom deck with the faces and names of everyone at my work. This feels like a slam dunk. I am terrible with names, so I think this can up my game a lot.
In my experience it's hard to find things that feel marginally useful/fun to learn and that works with Anki. But when it does, it's amazing.
Another way of doing it could be generating a deck with questions like:
"Q or P. Which comes first?"
I suppose that which technique will be superior depends on whether you usually sort things relative to each other, or relative to their container. If you have a fixed container of files, you could think, "ah, 'T', that's 20 (out of 26), I should look down 3/4 of the length of the container". But if the container wasn't evenly divided - for instance, your 'I' for 'Insurance' was a much thicker file than your 'T' for 'Taxes' or whatever - you'd no longer be able to use those numbers directly. What do you think?
is this just for fun?
The things I'm trying to learn like past economic decisions and investments and their impacts, logical fallacies, algorithms and data structures for my next coding round, database design patterns, areas where one system design pattern excels and sucks at with examples, all study areas where finding the core patterns and their applications is central to the learning process, to make any bit of real progress. Anki sucks so bad at this. m
my disgust at atomic spaced repetition, of which Anki is the cheerleader, comes from how gullible I was reading salesy pitches of "remember anything", "remember forever", "how i could memorize x in y days" kind of articles floating around suggesting it. It left a bad taste, like those As-seen-on-tv home exercise equipments and non stick pans with grifty promises.
Anki maybe useful, to some, but it falls apart for everyone as soon as you add any meaningful complexity beyond mapping 2 lists word to word.
So why do it? Why not learn things the wholesome way? With pen and paper ?
If we accept that all learning involves some memorization, I believe there's no harm in using the best tool for that specific job. I've seen a good amount of literature showing that SRS-like systems are indeed the best.
You can make connections that give you really deep intuition; it just takes practice making cards. I wrote about it here: https://jacobgw.com/blog/tft/2024/05/12/srs-intuit.html
I'm confused why you'd expect spaced repetition to serve this purpose. Did someone claim it would?
Yes, it is shallow. It's meant to be shallow. It's not meant to replace other tools to build connections. It's not meant to be a complete solution. You still need to apply the material to learn it.
Spaced repetition is for remembering/recall - not understanding. It's useful for people who have already done the work to understand (practice problems, etc), but would like to keep it in memory. If you are taking grad level analysis, and can't remember that a compact set is closed, because it's been 2 years since you took undergrad analysis, then SR will help you.
In some cases one can also use decks made by others, which can help avoid wasting time on dry, unnecessarily fluffed up instructional materials with low signal-to-noise ratios like is common in university courses.
I think Matuschak wants to be able to recall particular things ~forever, and spaced repetition is the best method known for that.
(Nit: Your sentence begins as a genuine expression of puzzlement rather than an outright criticism, but then you refer to Matuschak's "obsession", which pre-judges the issue. It would be more consistent to refer to his "emphasis on" or "advocacy of".)
Edit: Can't edit my original post though, time's up. But I would if I could!
It'd take me over 10 minutes simply to add those 40 cards.
But if he's referring only to reviewing, it's believable. Personally, I don't think I can handle more than 10-20 new cards per day.
> I did quantum country a while back and I sometimes needed more than an hour to get through all the questions in a single practice session.
Are you literally going through all the questions? Isn't the whole point of SR not to do that?
If you meant it took you an hour to go through whatever subset is due for that day, my next question would be: Are you reviewing daily? SR algorithms assume you review daily, and that's the only sane way to keep the time low. As an example, out of a 2000 card deck, I had to review only 6 cards the other day.
Yes, referring just to reviewing. Adding takes much more than ten minutes, as you say. And so, in practice, I don't saturate this capacity most days.
That's very interesting! The review sessions cap at 50 questions, so that means well over a minute for each. Our design intent is that if you can't remember in a few seconds, you should mark it as forgotten, view the answer, and move on. The marginal benefit from exerting marginal effort to remember on your own from point is not high. But your comment suggests that it would be helpful for the interface to do something to suggest the tempo we have in mind. Thank you for sharing.
Like, I often had cards asking me a question, and then I'd have to think a little bit about the question, then I couldn't remember, then I look at the solution, and I have to think again about the answer, and how it relates to other concepts.
Yes, that's exactly how it's supposed to work the first few times you try to associate the answer with the question. Then it begins to sink in, and you can remember the answer for a few minutes after seeing it. When you've successfully remembered it several times after short-delay, then the program increases the delay. When you've successfully remembered it several times after medium-delay, then the program increases the delay again.
> don't you need some time to actively memorize it again?
The rapid repetition of asking / being shown the answer multiple times IS how you actively memorize it.
If the deck is so large it takes you an hour to get through one cycle, there are too many cards in it. Start over with a deck that takes you only 5-10 minutes, and spend an hour going through several repetitions. When your rapid-recall rate becomes high, slowly add more new cards.
From your description, it sounds like, when a user flubs a question in a session and is shown the answer, you do not quickly re-test them on the same question during the session to improve recall, but just go on to other questions instead.
1. I need a goal.
2. The time I need to practice to reach that goal needs to be reasonable.
This makes endeavours like learning to speak a language, to play an instrument, or getting buff unsustainable for me.
(I probably come from the opposite viewpoint: I consider the few axes along which I've devoted over a decade to learning as being the particular high-dimensional corners making me me, rather than any of my 8 billion other conspecifics)
I think your goals are just far too abstract, you're thinking years into the future instead of something more immediate. Here's how I'd "atomize" them:
- Learning to speak a language → Being able to answer some basic everyday questions: What time is it? What's the weather like outside? How am I feeling? When's my birthday?
- Playing an instrument → Being able to play one popular song of my choice to a reasonable degree.
- Getting buff → Being able to look myself in the mirror and see progress.
Suddenly all of them are very achieveable within a month or two. I then either lose interest or set myself some "higher" goal. Can I up that to three songs? Can I describe my work or hobbies in $targetLanguage? Is there a body part I'm especially interested in improving? Then that becomes my new "project" for the next couple of months.
Rinse and repeat, all the way until I can speak German (not quite, but I can point to my A2 certificate and call myself an advanced beginner), or play a piano (not quite, but enough to easily impress anyone that never tried), or until I feel good about the way I look (not quite, but it is an indisputable fact that I look better than ever before).
That would be an incredibly valuable tool beyond learning, a killer feature for an operating system.
Example: How do you land on the moon? Build a rocket and land it.
Divide the problem into 99 steps involving Gravity (math), rockets (mechanical, combustion, chemical), trajectories (advanced math), and then life support (biology, math)
Learn each of those steps and with enough money you can get to the moon.
from some podcast I forgot
> Subordinated to an authentic pursuit . . . Diving into a brick wall.
Alas software usually is the wall, keeps us from grasping true understanding & development. Interface most often is a wrapper high above the core of software. We trap users, keeping them away from authentic & self directed experience.
There was a great submission hours before on Enlightenmentware, on softwares that have enlightened us. Letting users into the natural philosophies underlying software, bringing software from "wizards" and "just works" to an age of reason for users. I think this underlies everything setup here; it's the tales of systems that fomemted implicit learning well! https://news.ycombinator.com/item?id=40419856
> Learning by emersion works nautralistocally when the material has low enough complexity relative to your prior knowledge that you can process it on the fly . And natural participation reinforces everything important giving you fluency when those conditions arent satisfied - which is most of the time - you will need some support. You want to just drive in and you want learning to just work.
This is such a beautiful capstone for the bridge IT ought be building. It speaks to the need for general systems research, new (or improved I guess) systems for operating many processes (and their sprawling subprocesses/subroutines/promises) and seeing them run. It necessitates being free to take that live world and tinker and run and rerun experiments. With safety (and the already spoken of visibility). Opening the option to become acquainted with capability and intent.
I'm less clear on the guided parts. But my thesis is that software fails to have sufficient starting conditions for most of these good implicit / guided learning loops to begin. We are trapped in a place where everything is arbitrary interface & none of it is learnable at all (to any honest depth). So we have no mental leverage to begin building mental muscles with.
This speaks deeply to me, as the shame of our industry & the shining endless journey we should be so excited to be exploring. That we haven't been trying for broader software ecosystems, for more visible and malleable software is a resounding quaking mystery. This is the great open ended quest, is the real journey of what we are doing, and we are not only flat failing to heed the call of this grand adventure, we are worse, to its detriment, building infernal machines that trap us. Even before machine learning, we were already far down the winnowing closing path spoken of in Dune, and some day I hope the sleeping computing world might awaken from this slumber,
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
This talk sets up the greatest call for computing humanism that I have ever read. Throwing off the spheres of control & helping each other reach to the poles is the point of these days is the point, and computing's chance to be the vanguard pushing that forward is colossal, and keeps rising & getting yet more possible.