Hacker News Reader

Comment by reubens | Hacker News Reader

As a long term user of Anki, I think the hype here is too much. Spaced repetition helps you learn facts, but that is such a limited representation of what learning something means. I get that the point of a mnemonic medium is to improve your memory, but the focus on memory in and of itself is reductive. Memorising things helps no one gain a better understanding. If anything, it’s just a way to avoid the mental effort of real comprehension (roughly analogous to a google search for a solution rather than solving a problem yourself).

Edit: I love Michael Nielsen’s work, but I don’t love his glorification of this tool.

View on HN

I am one of those guys who had a 4.0 GPA in college but I never used spaced repetition because what I was doing seemed to work well enough. Post-college I tried it for learning japanese and it seems like a downgrade from my usual approach.

The best way to learn something is by contextualizing it. For example it doesn't matter if you forget a math formula if you know how to re-derive it. If the formula is truly important, every time you access it, the entire DAG of knowledge leading up to that formula will be "refreshed". This is a very natural spaced repetition that erodes from the edge of your knowledge graph since the basic building blocks are refreshed so often. New knowledge is extremely easy to acquire, and your graph can be "refactored" every so often.

The space repetition apps I have use doesn't have a similar concept of interrelationship. For example it's possible for you to learn the flashcards for "three" and "one" but not know the word for "two". This is insane way of doing things. You are just refreshing random isolated nodes. Not only does it make it a lot harder to learn (since random out of order information is harder to compress), the knowledge graph you build is a tangled mess that's not as usable and doesn't degrade gracefully.

The problem is that students are provided with almost no support in development of metacognitive skills.

They don’t even learn how to approach non-trivial but eminently solvable problems other people have posed, much less how how to “refactor” past understanding, how to investigate completely new subjects, how to hunt for problems to work on, how to manage projects that take longer than a day, etc.

I think SRS software can be profitably used, but it would take a lot of thought and planning. For example, an SRS software system which randomly assigned difficult exercises from different past chapters of a math textbook could be pretty helpful, to keep the student occasionally revisiting topics instead of working through a chapter in a week or two and never returning to it again.

I found it helpful to break content down into constituent parts, but also to make cards unify those constituent parts. So, there will be cards about individual facts, but also cards that relate those individual facts. And eventually cards that related the cards that related the cards that related the individual facts.

Also, I cemented my understanding and intuition of the Unit Circle using this approach.

> Also, I cemented my understanding and intuition of the Unit Circle using this approach.

What do you need to do to “cement understanding and intuition of the unit circle”?

The common diagrams shown in high school trigonometry courses – e.g. https://etc.usf.edu/clipart/43200/43215/unit-circle7_43215_m... – basically involve only 2 meaningful facts: a rhombus made of two equilateral triangles has a long diagonal with squared length of 3, and a square has a diagonal with a squared length of 2. (a) These are fairly easy to figure out, and once understood are quite easy to remember, and (b) all of the derived factoids are pretty much trivial.

Students are typically taught this material in a quite terrible way, unfortunately.

I memorized the facts and then was able to figure out the relationships after the fact.

That may be a brute force way of doing it, but it works for me.

My point is that there’s very little there. People shouldn’t need to go to extensive effort or need a special system or method to learn 2 simple facts. Taking a simple thing and wrapping it in a big pile of obfuscation is a poor way to teach/learn. (That this is taught this way is not the fault of students.)

That is a very elegant way to put what I said so grumpily, thank you.

But sometimes you lack the time to re-derive.

> The space repetition apps I have use doesn't have a similar concept of interrelationship.

I think this is a really important insight and a big problem I noticed as well. When I started working on my own space repetition app[1] I really wanted to enable and facilitate this kind of interconnection of concepts. I added a notes section in addition to cards and the ability to link/reference notes from cards, cards from notes, cards from other cards, etc.

I found this really helpful for example with Japanese. Having a sentence card for a vocab word that also links back to a long form note on the grammar being used in the card. Another example is kanji. Making cards for individual kanji as well as vocab cards that use those kanji and linking them together. Over time I was able to build up a huge web of interconnected information.

[1] https://mochi.cards/

Glad you like my work. To quote the essay:

"these kinds of results should be taken with a grain of salt. The mnemonic medium is in its early days, has many deficiencies, and needs improvement in many ways... is this medium really working? What effect is it actually having on people? ... are there blockers that make this an irredeemably bad or at best mediocre idea? How important a role does memory play in cognition, anyway?"

The essay continues at great length in this vein, getting gradually into more and more detail. We have detailed discussion of common ways spaced repetition fails, of ways it may inevitably fail or needs to be redesigned (a different issue), of ways it falls short of the goal of understanding, of ways spaced repetition is a poor lens for memory systems, and so on. I haven't counted, but believe there's several thousand words in this vein.

The conclusion is that: (a) we are some ways from having really good memory systems; but if we did it would (b) likely to be extremely helpful, including in lots of ways people don't commonly appreciate; and (c) be far from a panacea. The essay has a lengthy discussion of points (a) and (c), in addition to (b).

I don't think it's reasonably characterized as hype.

I was curious to read the self-criticism section since I fall into the memory-skeptic camp myself.

And I did in fact find my main concerns referenced there, though my impression is that they weren't taken seriously.

One telltale sign is the condescending tone with which they're referenced:

> Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. ... You'd think their claim to have a broad conceptual understanding of French was hilarious.

The other signal is the convenient changeup of exemplar subjects, natural language vs. computer programming. In the above, for example, having a "broad conceptual understanding" of French is obviously useless—and yet having a broad conceptual understanding of topics in e.g. Computer Science is extremely useful because:

1) It's part of a map that allows you to jump deeply into related as areas if a demand for that knowledge actually arises. 2) The 'broad conceptual understanding' (if done well) is generally of 'basic principles,' which have an interesting property of being applicable to wide ranges of much more specific subject areas.

> Another common argument against spaced repetition systems is that it's better to rely on natural repetition.

A more fair angle on this argument is that it's better to rely on natural contexts instead of 'repetition'—this is the reason "brain training" apps fail, for instance. (SRS obviously isn't as bad as those, but it still certainly ranks lower than a 'natural context' in regards to imparting skill in that context.)

> For instance, it's no good (but surprisingly common) for someone to memorize lots of details of a programming language they plan to use for just one small project. ... But the truths of the last paragraph also have limits. If you're learning French, ...

This is another sneaky subject change. It's pointed out how using memory systems for learning programming languages is probably non-optimal. But then this next paragraph is supposed to show us the limits of the supposed non-optimality—but it never does, because it switches to talking about a natural language instead, using an argument about lacking interlocutors which is completely inapplicable to the programming languages example.

In any case, there are things about this project that I like very much, but I do still suspect there are subtle flaws in the premise of what this would end up making people better at, and the tone approaches Wolfram-esque levels of grandiosity that makes it hard to take seriously.

>> Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. ... You'd think their claim to have a broad conceptual understanding of French was hilarious.

The way people learn French (or any language) is not by memorizing atomized vocabulary definitions on flash cards, but by listening to / reading a large amount of comprehensible input, which is to say, meaningful grammatically correct sentences in their context. Memorizing vocabulary might help some self-studying language students bootstrap enough very basic understanding to make a wider variety of inputs comprehensible, in the absence of a teacher or properly-designed curriculum, but in and of itself does not comprise learning the language.

People who encounter a large amount of appropriate-level language in context do end up with what might be called “intuitive understanding” of language (or maybe “subconscious understanding”; “conceptual” is not a great label, IMO). They know how to recognize and later how to form grammatical sentences because they have been exposed to a wide variety of such sentences. They learn the subtle connotations of words (beyond the definitions offered in an English–French dictionary) by seeing the words used many times in context. Etc.

From what I have read, speaking per se is just not that useful at the earliest stages (at least the first several months) when learning a language.

>> It seems plausible, though needs further study, that the mnemonic medium can help speed up the acquisition of such chunks, and so the acquisition of mastery.

This whole argument basically boils down to speculation.

Thanks for your response Michael, and my apologies for the temerity of my initial comment. At the end of four years of Med school, I am truly Anki fatigued (and a bit fed up with learning facts about rare diseases I may never see in practise). The points you highlight here I recall being present to a degree in Quantum Country, so I really don’t have a leg to stand on- apologies for directing my frustration toward your work.

Thanks, no worries. Sorry to hear you're feeling a bit fed up with Anki / parts of Med School.

I've googled generally and verbatim with various quotes from above and your name, and cannot find the essay. It sounds interesting. Can you post a link please.

In the essay, the importance of dependency on previous concepts is touched upon multiple times.

I essentially see 2 distinct "problems" or tasks that would need to be done in this specific respect (without downplaying the value of other factors): 1) it takes effort to encode the dependency relationships, 2) even if someone puts in a lot of work for a specific subject or domain to encode the dependency relationships, we need to modify the tools to take advantage of this knowledge.

I would propose as a next testbed, one takes the MetaMath database (set.mm), since dependency of a theorem on other theorems or axioms can be extracted in automated fashion. So I would love to see a tool (a working title could be Anki/MetaDrill or "autodidact" was the title I had in mind) that presents me with the following exercises:

type 0a, 0b: given ascii characters select LaTeX symbol (a) and vice versa (b) [for example ".-" for negation etc...]

type 1: given theorem/hypothesis/axiom/definition abbreviation name: produce the statement

type 2: given theorem/hypothesis/axiom/definition statement: produce the abbreviation (the reverse of exercise type 1, both can be easily extracted from an instance of a verifier that has parsed the set.mm database)

type 3: given a couple of input statement, and an output statement: produce the theorem abreviation that justifies this step (can be easily done by randomly selecting a target theorem from those scheduled for the user to know, then searching all proofs in the database for steps that reference the target theorem, each reference to the theorem is a utilization or example application of the theorem)

type 4: given a theorem: produce the list of theorems directly used in its proof (this one depends on the proof used, so that it is possible to find proofs that don't rely on those in the database, but is easy to generate for the proofs proposed in the database: just enumerate all the references in the proof of this theorem)

type 5: given a theorem (by abbreviation or by statement): produce a proof (the previous series of exercises should provide all the prerequisites, and the last exercise above would contain a lot of hints to reproduce the proof) it is possible to find an original proof, and the MetaMath verifier can be used to check the user's proof!

the exploration or presentation of new axioms, definitions, theorems, can be limited to those "in view" to the user, using the dependency graph of theorems, so only a minimal number of new concepts is introduced at a time. I.e. this prevents the user from being bombarded with uncontextualized facts like "the connected sum of a torus with a sphere is a torus" without first learning about addition of rational numbers...

once a large number of people can quickly get up to speed with MetaMath, they will see the utility of spaced repitition, and understand that formalization is the way forward, since formalization automatically forces you to be explicit in dependencies, and forces people to state their beliefs as exactly as possible, or alternatively as quickly as possible (be bold), but such that others can prove errors in their conjectures by providing proof that the bold person happened to introduce an inconsistency.

But the biggest advantage of logical formalization of knowledge is that all the above exercises can be automatically generated, no one needs to produce flash cards! not only can they be generated automatically, they can be generated as a function of the past performance of the user on those exercises, instead of waiting a long period of time before retesting prerequisites 2 or 3 levels deep, we can test already seen knowledge that relies on it, and upon failure schedule the immediate prerequisites it relies on 1 level deeper, most of which will succeed, but one of which will typically fail again prompting it's lower level prerequisites to be tested (2 levels deeper from the original failure) to finally prompt a last failure, upon which the 3rd level lower prerequisites are reached but all succeed, just like how a professor in an oral exam probes where exactly the student got stuck. This allows to test for gaps in working memory to be detected without explicitly testing all the lower levels, just like the blood test during the second world war: at some point an expensive but very sensitive test was developed, and instead of testing the soldiers individually, their blood was mixed then tested, and in this manner it was possible to cheaply scan for positives and only zoom in when a group had a positive. I can not find the link to the relevant wikipedia page sadly.

The key insight that Nielsen is advocating for is to use spaced repetition for concepts, not just facts. Understanding comes first, and then you deliberately try to cement a rich network that understanding in your long term memory. Personally, I’ve found this a revolutionary approach to learning.

As another commenter below notes, a recommendation is to spend one second reviewing each card. Thus, your cards are generally quite simple. This can very easily lead to a patchwork of simple details, but no glue holding them together. I’ve come to believe that the mental effort it takes to understand something should be prioritised over memorisation. If you’re relatively busy and spending a lot of time doing Anki cards, you may be limiting your opportunities for this kind of challenge.

I don't see why cards need to be quite that simple. I generally take about 10-15 seconds for reviewing each card. A lot of my cards are why/how/what questions (typical examples: "Why are bras covariant?"/"How are Green's functions used in solving linear differential equations?"/"What is Q-Learning?") The problem with skipping memorization of concepts after having understood something is that two months later, you'll have forgotten those concepts. So, all that effort will have been in vain. I typically don't create much more than 10 cards a day, or maybe 20 on a day where I'm taking in a lot of information, and I typically have less than 50 cards to review. It's really not a huge time investment - especially if you're "busy".

> The problem with skipping memorization of concepts after having understood something is that two months later, you'll have forgotten those concepts.

The key realization, for me, is that conceptual understanding, intuitions, and key insights are themselves just pieces of information that can be memorized.

E.g.: Q: How to derive Bayes' Theorem? A: Write P(A and B) two different ways.

Understanding beats rote learning every time: it's always possible to reconstruct the learning. You can generalize and customize it. But how much quicker if you also have that at your fingertips!

Perhaps, a bit like writing a very clear and well thought-out solution on SO... and later googling for it.

If you're only using Anki to memorize facts, you aren't using it correctly. You need to get more creative with the card-creating process. For example, instead of just memorizing the definition of a term, create cards like this:

- "Describe [term] in one sentence/two sentences/three sentences."

- "Describe [term] without using the words [common words used to typically describe it]."

- Imagine you are explaining the concept of [term] to a small child. How would you do it?

And so on. The power of Anki is the underlying memory algorithm, not simply the memorization process. You can understand and remember anything if you organize the data in the correct way.

I understand your point but I disagree. Just memorizing does not help, but memorizing definitely increases the ability to gain understanding.

Connections required for understanding start to happen automatically when you start thinking about things you remember. If you don't remember the facts and fine details, your understanding will be shallow. Deep understanding requires going trough the facts and finding connections. Sources outside the head, like books, notes, Wikipedia and Googling can't make the connections. They are dead outside the mind.

Of course, if you are memorizing trivial facts, it does not help.

I'm in agreeance. In this case, I don't think SRS offers a lot of benefit, i.e. the tool isn't the hard part, it's the content you put into it.

EX: I recently (5ish months) started moving Kindle highlights to Anki. I've found my ability to retain the facts has increased but it is also directly correlated with how I designed the card (cloze, length, etc).

I think we should put more focus on extracting key points for later learning rather than focus on automating the review of those points.

Reply on news.ycombinator.com