Anki and GPT-3
andrewjudson.com
andrewjudson.com
I suspect that forgoing this step will not lead to understanding, but to rote memorization of trivia. Which is fine if that's your goal, but it leaves most of the potential of spaced repetition on the table.
Perhaps the gpt can find truly useful information, but if you don't know why it's useful you're just filling up your memory with material you don't know how to utilize well.
I have a similar strategy for the Ultimate Geography deck. I want to know where every country is on a map, as well as their capitol and flag. So I suspended all cards, and then periodically enable 20 new ones, and learn as I go. That often means googling the location or pronunciation the first time I encounter the card.
(I use the low-key Anki configuration with Pass/Fail buttons plugin)
When I encounter new words in real life, it tends to be all at once (i.e. watching a show while making dinner). I scribble the word quickly on a nearby notepad and don't have time to look at it again until I'm bulk-uploading new words later. It bothers me that I'm maybe not getting the full benefit of Anki, because I'm missing the context and/or example sentences.
I would love to hear ideas about how to resolve that timing issue.
I seem to retain more from those 20k word decks you mentioned than my personal decks, likely because they come with example sentences and context that my personal decks are missing.
If you have not drilling material on one end, and writing your own questions on the other. I think drilling gpt generated questions is probably closer to the latter than the former.
USER: Add a new Anki card for, "It is important to set a high standard from day one."
GPT: That seems like generic advice. Of course high standards are better than low standards in a vacuum, but everything has a cost. Being too rigid in setting high standards may limit the business's flexibility to adapt to changing market conditions, customer needs, and technological advancements. Are you sure high standards are that important?
(That criticism actually generated with GPT, using the example the author of the tool Tweeted.)
[0]: https://sci-hub.ru/https://pubmed.ncbi.nlm.nih.gov/26173288/
Great example of this is topic of memory sports [1], which do use techniques that work, but are likely less useful for actual remembering real world complex information.
> Another merit of practice testing is that it can be implemented with minimal training. Students can engage in recall-based self-testing in a relatively straightforward fashion. For example, students can self-test via cued recall by creating flashcards (free and low-cost flashcard software is also readily available) or by using the Cornell note-taking system (which involves leaving a blank column when taking notes in class and entering key terms or questions in it shortly after taking notes to use for self-testing when reviewing notes at a later time; for more details, see Pauk & Ross, 2010).
Further explanation: https://lsc.cornell.edu/notes.html#post-1037
The Cornell note-taking system combines moderate and high learning techniques: elaborative interrogation, self-explanation, practice testing, and distributed practice.
If you think about it, creating Anki cards is note-taking in a specific format. Of course you can skip it and use a set prepared by someone else. It would be interesting to test: is creating your own flashcards better for studying than using ready-made ones?
We've found students more consistently study ready-made cards that are at desirable difficulty (they get about 80% correct) and which are segmented by topic (e.g. semantic grouping of flashcards to tackle "one lesson at a time" like Duolingo). Students would prefer to use pre-made flashcards by other students in their class, then AI flashcards, then create and use their own.
There is limited evidence by Roediger and Karpicke who are the forefathers of retrieval practise that creating cards is also important. Frank Leeming (2002 study Exam-a-day) also showed that motivation when studying is peaked when you ask just a few questions a day, but every working day.
Now one of the vital benefits of retrieval practise with AI over creating your own cards is foresight bias - not mentioned yet in this thread - the fact that particularly in some subjects like Physics, students don't know what they don't know (watch this amazing Veritasium video, it also explains why misconceptions are so handy for learning physics): https://www.youtube.com/watch?v=eVtCO84MDj8 - basically, if you use AI quizzes (or any prepared subject-specific right/wrong system), you learn quickly where your knowledge sits and what to focus on, and reduce your exam stress. If you just sit their making quizzes, firstly you make questions on things you already know, you overestimate how much you can learn, and you consolidate on your existing strengths, and avoid identifying your own knowledge gaps until later on, which is less effective.
--
To quote from my dissertation experiment on background reading for retrieval practise, the end is about foresight bias a little: Retrieval practice – typically, quizzing - is an exceedingly effective studying mechanism (Roediger & Karpicke, 2006; Roediger & Butler 2011; Bae, Therriault & Redifer, 2017, see Binks 2018 for a review), although underutilized relative to recorded merit, with students vastly preferring to read content (Karpicke & Butler, 2009; Toppino and Cohen, 2009). Notably mature students do engage in practice quizzes more than younger students (Tullis & Maddox, 2020). Undertaking a Quiz (Retrieval practice) can enhance test scores significantly, including web-based quizzes (Daniel & Broida, 2017). Roediger & Karpicke (2006) analysed whether students who solely read content would score differently to students who took a practice quiz, one week after a 5-minute learning session. Students retained information to a higher level in memory after a week with the quiz (56% retained), versus without (42%), despite having read the content less (average 3.4 times) than the control, read-only group (14.2 times). Participants subjectively report preference for regular Quizzing (Leeming, 2002) over final exams, when assessed with the quiz results, with 81% and 83% of participants in two intervention classes recommending Leemings “Exam-a-day” procedure for the next semester, which runs against intuition that students might biases against more exams/quizzes (due to Test Anxiety). Retrieval Practice may increase performance via increasing cognitive load which is generally correlated with score outcomes in (multimedia) learning (Muller et al, 2008). Without adequate alternative stimuli, volume of content could influence results, thus differentiated conditions to control for this possible confound are required when exploring retrieval practice effects (as seen in Renkl 2010 and implemented in Methods). Retrieval practice in middle and high school students can reduce Test Anxiety, when operationalised by “nervousness” (Agarwal et al 2014), though presently no research appears to have analysed the influence of retrieval practice on university students’ Test Anxiety. Quizzing can alleviate foresight bias – overestimation of required studying time – in terms of students appropriately assigning a greater, more realistic study time plan (Soderstrom & Bjork, 2014). Despite the underutilization noted by Karpicke and Butler (2009), quizzing is becoming more common in burgeoning eLearning courses, supported by the research (i.e. Johnson & Johnson, 2006; Leeming, 2002; Glass et al. 2008) demonstrating efficacy in real exam performance.
> Undertaking a Quiz (Retrieval practice) can enhance test scores significantly, including web-based quizzes
The way I understand it, retrieval practice increases test scores where you test information retrieval (quizzes, multiple choice, etc.). Which makes sense because you're practicing a skill that the test evaluates.
The follow-up question: does retrieval practice increase scores when you evaluate understanding of a subject, such as open tests or essays?
I am afraid I don't know and couldn't easily find any research. I did find this post, but it looks like SEO spam a little, and doesn't cite the essay claim: https://www.bookishelf.com/the-importance-of-information-ret...
I'll ask Prof Roediger as we occasionally communicate and will aim to get back to you. However, I wouldn't be surprised if any correlation was not statistically significant.
[edit: removed a point about variance which wasn't supported by more recent literate]
High stress (or anxiety) happens only periodically... like those job interviews you mention. It varies massively within individual subjects. There are some ways to reduce anxiety, but I wouldn't say anything has been conclusive yet. Here is an excerpt from my Dissertation about Test Anxiety... I found practise quizzes had a non-significant overall effect on Test Anxiety, and reduced only the "Tenseness" subscale of a relatively old measure sadly. The CTAS scale, referenced below, would probably be most useful for the job interview case.
"Test Anxiety is the additional stress felt when you must provide answers. It is prevalent, affecting students from preschool (McDonald, 2001) to taught university level, and across demographics (Beidel et al 1994; Beer 1991; Mwamwenda,1993). Prevalence is estimated at 20% (Ergene 2003) to 25% (Thomas et al, 2018) in current students. Test Anxiety often involves pressure, highlighting a social, non-genetic nature, and high variance is recorded between subjects (e.g. Keith et al 2010 longitudinal study showed low individual variation, but high between subjects variation). Overwhelming Test Anxiety significantly impairs wellbeing (Steinmayr et al., 2016). Interventions and online course design choices can markedly reduce Test Anxiety (Abdous, 2019). Unfortunately, few online learning studies record Test Anxiety, instead recording grade/score differences. Yet, Test Anxiety interventions are efficacious – e.g. "test-wiseness training" (Kalechstein et al., 1988), group counselling (Alkhawaja, 2013) and mindfulness training (Seidi & Ahmad, 2018). Research often classifies students into High and Low groups. There are disproportionately negative outcomes for High Test Anxiety students, whom more recently have been enrolled automatically in targeted interventions (e.g. Psychoeducation Bedel et al., 2020). Test Anxiety is related to Social Anxiety (Rothman, 2004; Sarason and Sarason, 1990). This manifests in a socialevaluative component, which appears in most scales, such as the Test Anxiety Inventory (TAI; Spielberger, 1980). TAI was designed for assessing children (Ludlow & Guida, 1991). Recent scales, including the Cognitive Test Anxiety Scale (CTAS; Cassady & Johnson, 2002), are adapted to university participants. An example CTAS item is "I feel under a lot of pressure to get good grades on tests.". CTAS has similar correlations with academic performance (Chapell et al., 2005) as those utilizing TAI (such as Hunsley, 1985), without the concerns of TAI applicability discussed in Szafranski (2012). Test Anxiety is typically construed as a trait under Latent state-trait theory, despite short term experimental intervention differences being observed (e.g. "looking ahead" immediately prior to testing in Mavilidi et al., 2014; or taking practice quizzes in Agarwal et al., 2014). Test Anxiety should not be confused with other forms of anxiety, such as General or Social Anxiety Disorder. Prevalence of anxiety is rising in college populations (Zagorski, 2018), as is loneliness, with 1 in 5 young people reporting having no close friends and feeling alone(n=2,522, Mental Health Foundation, 2019). Usage of social media is also associated with greater loneliness (Wohn & LaRose, 2014). However, artificially induced status updates have been shown to reduce it (Deters & Mehl, 2012), which aligns with status updates requesting academic support reducing Test Anxiety as found in Deloatch et al (2017) and findings linking higher wellbeing with greater academic outcomes (Public Health England, 2014). Research on social anxiety and social media is less clear, although recently Erliksson et al (2020) correlated greater internet usage and activity with increased social anxiety. Despite wide research on the negative aspects of social media usage (e.g. for university students, see Odacı & Kalkan, 2010) - little attention has been paid to online helping, which is fundamentally social."
It's wild because StudyWand took my sample notes and did everything that I would have done with them if I was going to use them to get a good grade in school. I was expecting some semi-decent cloze generated cards but got much more.
Literally, when I was in college I would take notes in class, and then spend about 20-40 minutes post class doing almost exactly what StudyWand does. The classes that I bothered doing that for, I always got a good grade in, nearly effortlessly. The hardest part was making the notes.
The part of this that I'm actually excited about is that this tool also works with any sort of documentation. For example, I can clean up any reference page from MDN as a PDF and get a usable (like actually well-made flashcards) set of 15-20 flashcards for it. Oh, you also get summaries and multiple essay questions too. The only way this would be better is if it gave you cloze deletions that were actual sample code to fill in the blank with.
I didn't really like your intro so I took a few days longer than I normally would have to look at your software (I normally check out every SRS software I see on HN). This software is insane. The value is so, so, so ridiculous. I half-hearted uploaded one poorly made PDF of a webpage and got flashcards that are comparable to what I would make as a 10+ SRS user. I almost stopped doing the initial reviews halfway through and looked for a way to pay for this.
Outside of Supermemo, this is the only other SRS software that I've seen that's worth my money. The hardest part is going to be convincing all my younger family members to actually use this. I've tried so many times to get people to use Anki (Supermemo won't happen), and they just don't get it. I think StudyWand might be able to bridge that gap. I'm going to try and see.
Can't see how summarizing isn't a preprocessing step for learning in general. Spaced repetition is a natural next step. I don't see how they compete
As another experienced Anki user, I have found I get much more leverage out of spaced repetition (both in terms of efficiency of memorization, as well as in how useful the information is) when I've first made the knowledge my own and structured it in a way that makes sense to me before creating the cards, rather than just dumping a bunch of pure entropy into the Anki database.
That's not really note-taking per se, but it is ensuring that the stuff I'm trying to memorize isn't pure entropy (which is always more of a challenge to memorize in any event), but rather is part of a larger sense-making structure. The purpose of spaced repetition is to help prevent that structure from decaying; it's not a substitute for having it in the first place.
Note taking as parrotting is distinct from note taking as distillation. The latter has much more chance of getting results with proper spaced repetition, since it will help establish a concrete foundation on which to build more knowledge in an easily retrievable manner. The former may or may not benefit from spaced repetition at all.
I would say if GPT distills successfully, while this isnt as good as self-distillation, it may still be useful.
If not, or worse, actively bullshits, it probably won't be much help.
I imagine if you need to learn that much material but the pre-made deck isn't there, this gpt integration might be very helpful.
Would I learn more if I created my own cards? Sure, but studying the premade deck takes me 30 minutes per day, which I can afford. I cannot afford the time to make and update a custom deck.
For me the biggest risk is being tempted to making way too many cards (because it's so fast and easy now!), end up with way too many reviews, and declaring Anki bankruptcy and uninstalling the app after a few months. I may done this more than once...
I'm about 10 years on my Anki deck and have reached the same conclusion as you about keeping deck size down. A single card has a huge time investment if you add up the reviews. Even more if it's a bad card and you fail it a lot. As the words you learn get more and more niche, it's important to weigh up whether a card is worth making or keeping. I actively delete cards that make me feel 'meh' when I see them, or that I fail a lot, so I don't lose motivation.
The project would probably have 10x the number of users in 6 months if they learned how to do product marketing.
Several comercial competitors came and went over the years.
I suppose it tells you something about the learning tools market.
(Plus Mochi has GPT-3 built in [1])
[1] https://twitter.com/MochiCardsApp/status/1603569008493711361
SRS is a life long game, so I don’t want to lose everything if Mochi shuts down in 5, 10, 20 years (no business lasts forever). Whereas with Anki I know I’ll be able to use it forever.
The product does seems good. I may give it a try.
This is what note taking is lacking!
When you pile up 200 notes or 2.000 ... you can push them into GPT to generate anki cards so that you can remember and make sense of it all.
my first thoughts without thinking are:
1. how easy to set up ubuntu linux and using firefox lol. this is more on anki being somewhat frustrating sometimes.
2. i have tried using notes that i didn't make myself and there were pros and cons but bigger cons
3. that generated note didn't abide by the 20 rules which I find to be a pretty good guideline. that seems pretty fixable though!
I'm also trying to reference the part of the text where the question was generated from, so the user can verify if it was correct or learn more about it in context.
I currently require a sign up to generate questions, but the latest questions can be seen at
Answer: ancestor worship.
Then the explanation talks about the other three options, but does not reference afterlife.
I wonder what went wrong?
Logged in users can upvote and downvote questions so the next step will be to try to fine tune the model on good questions.
It's specifically optimised LLMs to align with the Anki Minimum Information Principle, here's a 15 second GIF of StudyWand.com - https://studywand.com/images/splash/getStudyingGIF.gif
... showed it on r/GetStudying a few days back mentioning Anki MIP!: https://www.reddit.com/r/GetStudying/comments/10lg4ci/automa...
... it has >98% recall on psychology (compare to between 70-95% recall depending on the subject for the SOTA AIs). We achieve this through quoted filtering - 80%+ of questions are quoted to specific slides or time stamps in YouTube videos.
Would love to chat with you Tomte.
One of the things I think is hugely important about GPT3 is that it's paid for by the token. So for small users, it's pretty cheap, and it's fully workable for testing and experimentation.
I created a small Anki plugin that tags cards based on their content using the Text-Davinci-003 model. However, the cost quickly becomes prohibitive when a large number of tokens/ Anki cards need to be processed. To be fair that plugin is not working great, since tags should be viewed, merged and reused on a global scope.
From a budget-conscious student's perspective, this amounts to roughly $20 per semester in my case, just for the tags. Despite this, I am still fascinated by the automation possibilities offered by GPT-3.
I think my expectation of this kind of service (on demand huge network) is fixed monthly fees, call us for pricing, etc. It's cool that it's approachable as a student. It'll be very interesting as costs for these things drop and/or performance improves at the same price points.
You might still get 403 error, change the allowed origins to "*" in anki-connect.
Use the "summarise" option.