Anki SRS Algorithm : Spaced repetition explained with code
juliensobczak.com
juliensobczak.com
I'd still love to create a better open-source SRS algorithm at some point in the future. Mostly to use for language learning.
When I was learning Chinese characters, I did a bit of a deep dive into the Anki algorithm and found that the biggest flaw (imho) was the Ease Factor knocking a card back to 0.
If you mostly know a card, but miss it on one day for some reason - maybe you were tired or distracted, that card's learning progress should not be totally reset (as if you were learning it from scratch). That leads you to have too many cards to review on a daily basis. Instead, you should use some modifier to increase the interval to a reasonable level.
More explanation here: https://readbroca.com/anki/ease-hell/
I think it would be awesome to pair SRS with high quality images and audio, which I find most helpful for language learning. I've used Rosetta Stone and Duolingo in the past; Rosetta Stone has great audio and images but lacks a powerful SRS (it also has a number of other flaws in my mind, but I'll save that for another time). Duolingo is great for grammar and explanations, but I can't take the pronunciations and tediousness of it all.
Most SRS algorithms seen in apps today are based on one of the super memo algorithms in one shape or form.
Anki was based on SM2, which was already outdated in terms of supermemo algorithms when released, but was chosen due to its simplicity.
It’s completely separate from supermemo the app.
If you’re looking at SRS algorithms, it’s worth looking through, for no other reason than to see what the flaws of the SM2 algorithm (used by Anki) are.
In anki "Deck Options -> Advanced -> New interval -> 0.50" will make it so marking it as "Again" will cut the SRS interval by 50%, so i.e. if "easy" would have been "1 year", it now becomes "6 months" if it's easy the next time.
The above setting is called out in the "Adjust your New Interval" section of the link you provided, so it looks like you may have just missed that you already found the solution.
I've found 0.5 to be a good value, but you can tweak it of course. The default is 0, which gives the behavior you've observed (reset it all the way back).
> I think it would be awesome to pair SRS with high quality images and audio, which I find most helpful for language learning.
Anki cards can include images and audio, and while a little less well, they also support videos.
It's probably worth trying to create a deck within anki that does what you want before building your own system. It'll at least let you find actual issues with anki, rather than the surface-level misunderstandings of it you've got now.
Yes - I'm aware that Anki cards can include images and audio, and I've used those features extensively enough to know they are definitely lacking for my use case! Mostly, I'd like to make it easier to add the images and audio. The way I've done it in the past was googling for a stock image, downloading Forvo/Wiktionary audio files, and then dropping them into the Anki app. I'd appreciate an easier way to add these audio files and images when I add new vocabulary to my decks.
Like many other anki users, I've ended up writing code that generates my decks.
If you can think of a better UX for editing cards specific to what you're studying, implementing a tool that creates and edits cards outside of anki, and then using anki for the study part, will probably end up being less work in total.
If it's _only_ the editing part, that's solvable without having to rewrite the whole SRS system too.
tl;dr a bonus effect of regular Anki practice is you can detect when you're in a stupor
Anki Review Hell.
I created Memory Hammer to solve this[1], It's an always on E-paper Anki client on a Raspberry Pi Zero. I can now review cards during small breaks from work regularly, Instead of a dedicated review schedule where its easy to get overwhelmed.
It syncs with desktop Anki for adding and editing notes.
I'm planning a companion mobile app for memwory hammer which would help me choose decks, Say if I want to revise a particular subject or language for test.
How often do you sync?
What's your use case for this? Would it be good for learning say the top 5000 words in a new language or something like that?
Would be awesome to read part-2 of this blog post -> http://www.blueraja.com/blog/477/a-better-spaced-repetition-...
Plz plz plz pen down your thoughts on reviewing(part-2 of the post) if you get some free time.
But I also want to say: what a clean and enjoyable website!!
Thank you for having shared this.
I learned Hungarian (one of the more difficult languages to learn) by loading up a database of ~10k cards. While it worked great in the beginning, within a year, I was just constantly telling the algorithm that I didn't need to review cards for 3-5 years. But the thing is, with a language, I'm not going to forget all those words in 5 years. There will be no point in reviewing them. So I just started deleting cards if I knew them already. And eventually I was spending so much time deleting cards that I just gave up on SRS altogether.
It could be that I have some latent savant-like language learning abilities, though I doubt that. I think my experience is likely related to language learning itself. With a language, if you are immersed, you have constant daily repetition with or without the cards.
All that to say, I think SRS algorithms should have a language learning mode that automatically archives vocabulary cards once you've mastered them to a certain degree. I'm not sure what would work best. Maybe once you've crossed the 6 month threshold, it could just automatically archive the card.
If one is bilingual at an early age, then it's much easier to learn/retain additional languages. Most language learning in America starts kids off much later than it should (middle or high school). Full immersion is also a great way to learn.
And as far as SRS goes, it's important to not make cards if you think they're not necessary as it's easy to just make the whole review process too much to chew on. And sometimes you can cluster information in a single card (maybe a sentence that's an idiom if using language as an example) such that you do get continued exposure, and can decide later if you need to break it up if you can't recall some parts of it.
My takeaway is that SRS is great for the early stages when it's impossible to grok real material. You can rely on the SRS to help you "bootstrap" your language knowledge to the point where you can start reading. Once you reach that point, you're free sailing and don't really need the SRS.
This is is one of the primary reasons why I think SRS is not more widely adopted for languages, the other being that most people will never go through the effort to learn why and how they should do it.
AFAIU, SM2 computes the datetime of a next review, whereas Ebisu models a probability of remembering a given flashcard. It seems it’s a more straightforward representation that’s more amenable to implementing functionalities like “show me 10 least remembered cards”.
All of that being said, I am a fan of the Bayesian approach Ebisu uses (and Ebisu v3 would allow you to eliminate the awful way SM2/Anki deal with ease factors). Thankfully the v3 Anki scheduler will be pluggable so you'll be able to use Ebisu v3 with Anki on every Anki platform. :D
[1]: https://github.com/fasiha/ebisu/issues/43 [2]: https://github.com/fasiha/ebisu/issues/58
But as a developer, I've never found a need for memorization. Am I missing out? Do any developers have suggestions for how to utilize Anki to make myself more professionally efficient?
I suppose one starting point might be to track my most googled phrases ("trim newlines in sed," "bash printf into a variable," etc.) and turn those into cards.
Compare:
Oh, I have to remove this user from all the services this machine provides? I've got a script for that.
To:
We need to sort all the users by how much they've spent in purchases? A heap sort has a fairly decent worst case but doesn't parallelize all that well and if the users are already mostly sorted it doesn't get much advantage, maybe merge sort would be better...
1. It's rare that I have to use Google figure out how to do basic tasks in my Programming Languages of choice (C#, & TSQL). Don't get me wrong I, and everyone on the planet, looks up code on the internet. It's just a more rare scenario for me.
2. I write much quicker then your average developer. Of course most of software development is design and requirements gathering/clarification but when I do get down to writing code I'm very fast.
3. I spot bugs almost immediately because when I go through my Anki deck I get repeated exposure do different error scenarios. In fact I can often simply think about the scenario and it will occur to me where the bug is and what is going on. I attribute this to using Anki on a daily basis.
4. I can read code much better then before I started using Anki. I know at a glance what code is part of the language/framework vs what is custom code for the application.
All the specific terminology that a new technology uses and the exact keywords that it uses goes in. Any time that I need to look up something and I'm convinced I will need it again goes in. All of the company-specific tools go in.
This post hits exactly what I found when researching months ago :) Absolutely captures everything well.
This post has convinced me to even use Anki's algorithm.
If that already exists, ideally compatible with Anki cards, I'd be happy for recommendations. I have other projects and ideas that I feel would add more to the world than the umpteenth Anki-clone.
More precision isn't that important-- at worst, you slightly over-review some information, but that's no different from using a flash card's information in normal life outside of a review period.
>Some people want to extend this link between arbitrary cards. They want to be able to tell Anki "after showing me this card, show me that card", or "don’t show me that card until I know this card well enough". This might sound like a nice idea in theory, but in practice it is not practical.
>For one, unlike the sibling card case above, you would have to define all the relations yourself. Entering new notes into Anki would become a complicated process, as you’d have to search through the rest of the deck and assign relationships between the old and new material.
>Secondly, remember that Anki is using an algorithm to determine when the optimum time to show you material again is. Adding constraints to card display that cause cards to display earlier or later than they were supposed to will make the spaced repetition system less effective, leading to more work than necessary, or forgotten cards.
>The most effective way to use Anki is to make each note you see independent from other notes. Instead of trying to join similar words together, you’ll be better off if you can determine the differences between them. Synonyms are rarely completely interchangeable - they tend to have nuances attached, and it’s not unusual for a sentence to become strange if one synonym is replaced with another.
But I can imagine a more complicated scheme could be much better; done right.
But I'm not sure how to design a good UI for it. I can easily see it as resulting in too much friction so nobody uses it.
You really shouldn't be doing all your study in SRS anyways because so called "flashcard blindness" (i.e. only being able to recall the concepts in the context of your SRS program) is a real thing.
Furthermore, in my experience, what really made Anki effective for me was to write my new cards just in time. Meaning no pre-made decks except for something like the primitives of what you are studying (i.e. symbol names, alphabets, extreme base level concepts, etc). When you write your cards just in time you can really connect the context of what you are learning to the concept. From this perspective it really seems strange to build an entire ontology based on dependency graphs of a concept you haven't even learned yet!
For example, I'm learning numbers in Spanish. First, I ask myself to go from doce => 12; then, 12 => doce. Then, once I have a few numbers in my memory, I can start to combine those into diez + dos = ? (answer doce). The idea is that this last example requires recalling three previously-learned Spanish numbers to arrive at the answer. This is a trivial example but this underlying structure of successive layers of knowledge shows up everywhere and is horribly under-exploited by our current learning systems.
Programmatically this can be done. Manually, this stuff takes way too much work.
It does exactly this because I also thought this was a critical limitation of current systems.
I have a feature coming out soon (already merged to master but I need to make a proper release) which should make it way easier to use with Markdown files acting as the front and back of flashcards that are automatically picked up when starting up the program.
The problem is that this basically means dependencies can only form a tree, with one topic depending on only in at most one other topic. What I wanted was something that let me express the relationships as a directed acyclic graph. Something like learning a big chunk of a music piece is dependent on the questions for learning individual smaller passages. Something as basic as that cannot really be expressed with a tree but is trivial with a DAG.
How do you find yourself engaging with it? A big positive for me of Anki is the ease with which I can review a single fact (~seconds). It looks like Trane is intended for more dedicated situations, probably for ~tens of minutes at a time. Is that accurate? I'm basing this off of playing with the Rustlings example.
For music practice, I find it really helpful. Simply knowing the next thing you should practice without having to think about it too much is very helpful. I find it easier to practice for longer without getting bored as easily. If I get frustrated with something, I can get another question very easily. I find myself not realizing half an hour or sometimes more has gone by, which is not the case as often when I don't use it.
In fact, the Anki community largely endorses as an article of faith the complete insular atomicity of cards. I’ve always treated that orthodoxy with skepticism because I’m pretty sure that’s not how my brain works. Nor anyone else’s, probably.
Basically their system involves first learning to recognize common parts of kanji (radicals), then learning to recognize kanji made up of those radicals, and finally learning vocabulary that uses those kanji. Later items in the graph are unlocked by getting the previous items to a certain SRS stage. Additionally, the entire syllabus is divided into 60 discreet levels of (very) roughly the same number of items, the next level being unlocked by getting 90% of the current kanji to a certain SRS stage, which helps keep it all manageable.
I tried using Anki before, but found the Wanikani "method" to work much better for me since I could constantly build on things I had learned previously.
Dependency relationships between exercises are core to its design.
I tried to use Anki but this limitation is pretty baked into Anki and made it unusable for cases where there's a clear order in which things must be learned (music in my case).
> The space repetition algorithm in Trane is fairly simple and relies on computing a score for a given exercise based on previous trials rather than computing the optimal time at which the exercise needs to be presented again. This will most likely result in exercises being presented more often than they would in other spaced repetition software. Trane is not focused on memorization but on the repetition of individual skills until they are mastered, so I do not believe this to be a problem.
Could you say more about it?
- When Trane is asked for a batch of exercises, it performs a depth-first search of every branch in the dependency graph. It picks up exercises as it goes along, and stops exploring a branch when the dependencies of the last node are not met.
- Each exercise is given a 1-5 score by the student and those scores are used to produce a combined score, rather than compute a date on which the exercise should be reviewed. Two reasons:
- Trane is also meant to be used to learn skills, which do not fit the assumptions made by the SuperMemo algorithm.
- I believe most of the gains come from regularly being reminded to perform an exercise, so even a simple scoring algorithm should get most of the gains. Eventually, I want to replace the existing scoring with something more complicated that also makes a distinction between memorization and skill exercises to get better results. But based on my testing, this works for the most part as is and the gains to be made from more complicated algorithms are marginal.
- Once you get exercises from performing the search, you reduce that number into the final batch by grouping exercises based on difficulty and selecting a percentage from each bucket. This is done to make sure not too many very difficult or easy exercises are included in each batch.
There's also some elements of randomness added to prevent the same exercises from appearing all the time. For example, the branches are explored in random order and the elements from each difficulty bucket are selected at random. But that is the gist of it.
All of this is trivial to implement - I think it's best to create from scratch and specialize it for one's own domain instead of using a pre-existing solution. In my case the relationships are straightforward (sub-topics) but in others it might be more fuzzy. Programmatic fact-generation/deletion is also something I put a lot of effort into (again, domain-specific).
It uses embeddings to make inferences across exercises and flashcards. This is handy for:
- detecting that you already have a flashcard for that particular word/phrase/grammar issue
- you know one sense of a word but not another
- you correctly used a word that you have a flashcard for and that review can be put off
It can look at your vocab and grammar skills of a particular area overall using Rasch scoring. Perhaps you answer almost all the questions about “cooking” correctly but none about “parts of the body” - well then the app can focus on the second area.
There is also work to be done to build out better regressions given this data. If a word I am studying is way above my skill level, I will probably forget it faster. Or maybe I forgot a conjugation which is very common. It should be easier to remember because I’ll encounter it more.
Whether or not my app succeeds, I think these types of approaches hold a ton of promise to increase learning efficiency.