Each memory tool runs through weeks of simulated work. Each agent chooses what to store and remember, guided by each tool’s own documentation.
Cognee and Karpathy’s Wiki tied for the highest accuracy score at 97.1 out of 100.
Using Cognee is significantly cheaper, at about half the wiki’s token cost at $256 per 1,000 questions vs. $503.
Every memory system has tradeoffs though, Cognee was the slowest system we tested, taking over five times as long per question as the wiki.
Across the tests, the biggest reason for failure was agents declining to answer questions they should have been able to answer. That accounted for 61% of failures.
We also tested whether each system leaked information it had been told to forget. 9/12 systems never repeated the facts they’d been told to forget, while three leaked data that should have been forgotten.