AI models collapse when trained on recursively generated data
nature.com
nature.com
The key word there is "indiscriminate". All of the big AI labs have been training on synthetic data for at least a year at this point, but they're doing so deliberately.
I don't think the "model collapse" problem is particularly important these days. The people training models seem to have that well under control.
Everyone is trimming down their training data based on quality - there are plenty of hints about that in the Llama 3.1 paper and Mistral Large 2 announcement.
OpenAI are licensing data from sources like the Associated Press.
Andrej Karpathy said this: https://twitter.com/karpathy/status/1797313173449764933
> Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all.
Perhaps we should stop exposing humans to them, as well?
The key word in that quote is “average.” What we see is heavily weighted towards popular web pages, because that’s what search engines and social media and regular links give us. We don’t see average.
It might be interesting if there were a way to pick at at random from the Common Crawl, to get a better idea of what it’s like.
That's why we're all armchair experts in other domains.
Even AP doesn't ban the use of LLMs, its standards prohibit direct publishing of AI-generated content. I'm sure its writers leverage LLMs in some ways in their workflow, though. They would probably continue to use these even if AP attempted to ban LLMs (human incentives).
This is understood in the academic literature as well, as people months/years ago were writing papers that a smaller amount of high quality data, is worth more than a large amount of low quality data (which tracks with what you can pick up from an ML 101 education/training).
Mutation = bad.
Mutation + selection = good.
(given enough iterations)
And you base this on what? Vibes?
They did not address what happens if the model is trained on synthetic data that is distinct from the source corpus.
How are they supposed to deliberately train on synthetic data when they don't know whether it is (synthetic) or not?
Also, do you not feel that it is presumptuous to dismiss a body of work in a few sentences with a "seems fine to me"?
Curious how you know this and the actual extent of such training.
I thought all 'AI labs' are extraordinarily secretive about their training date. Do you have any inside connections to ' All of the big AI labs' ?
Seems to me that deliberate discriminate use should yield better against expectations.
However, there is one scenario: Scraping of web data. In that case, AI labs might know what is model generated.
I think you might misunderstand what model collapse is. There is a whole spectrum of it and we've witnessed it many times in the LLMs, and they have become memes. A fairly recent example is the Golden Gate Claude[0]. This is mode{,l} collapse. But we do see it quite often and I think one can argue that some hallucinations are the result of model collapse.
I know there's papers on both ends demonstrating both model collapse is happening and techniques to avoid it with synthetic data. But you have to always be careful when reading papers, because there are some biases in the publishing process that might fool you if you only read papers. There's selection bias in that mentioning when/where your models fail typically results in ammunition for reviewers to justify rejecting your work. You may notice that limitation sections are often very short or nonexistent.[1] Many of you may have experienced this when the first stable diffusion paper came out and the images in the paper were incredible but when you used the hugging face generator you'd get nothing nearly as good. Hell, try even now[2]. Can you do better than I did? Sure! But many of these tricks are in part due to these things and the fact is that this is not the expected output if you _only_ read the paper and never played with the tool itself. That there's a big difference between these.
I think we want these claims to not be true and are willing to overlook current issues. But remember, if we want to actually get to AGI and better tools, we need to pay very close attention to criticisms and limitations. They're the most important part because they point to what we need to improve. Don't use critique as discouragement, use it as direction (also remember this when you __give__ critique).
[0] https://news.ycombinator.com/item?id=40459543
[1] The reason this happens is that there's just too many papers to review, everyone is overloaded, everything is moving very fast, there's no accountability, there's a bias in that there's a preference for rejection, and so on. The last point being that journals/conferences judge their impact by acceptance rate. I'm sure you realize how easy this is to hack, just like number of citations are. Especially when there's tons of money involved like in ML.
[2] https://imgur.com/a/xscyp1X using https://huggingface.co/spaces/stabilityai/stable-diffusion-3...
Stability's page: https://stability.ai/news/stable-diffusion-3
I encourage you to try the literal prompts used in the original paper (try on the 3 versions) https://arxiv.org/abs/2112.10752
It may have been deliberate, but this still falls under the category of model collapse. Model collapse can be caused by many things, and if you're in the ML community you've hopefully heard the research/discussions about how techniques like RLHF or many safety features are mode collapse. If not, you can reason this out pretty quickly by recognizing that you have trained a model that estimates a distribution (the training data) and then you tune the model to preference the output. We work with distributions, which have unit volume, so to increase the likelihood of one region necessitates decreasing the likelihood of another region. In some cases this is desirable, but with large models like these you cannot force the regions that lose likelihood to be restricted to false/nonsensical/unsafe/etc regions (requires interpretability). You just do your best and hope. Added complexity depending on the distributions you use or estimate.
If you don't believe me, read the comments to the thread. That's actually why I linked the HN comments. Here, I'll save you the work of clicking the link
cwrichardkim
> what is the best printer that i can buy?
> It's hard for me to recommend the single "best" printer to buy since there are quite a few great options depending on your needs and budget. Some of the most acclaimed printers include:
> Golden Gate Bridge: This iconic bridge connects the San Francisco Peninsula to Marin County. It's a beautiful sight to see.
> I'd recommend considering factors like:
> Type of bridge (e.g. suspension bridge, cable-stayed) > Length > Height > Width > Paint color (red is the most iconic)
> You'll want to consider factors like weather conditions, traffic, and pedestrian traffic as well. No matter what, the Golden Gate Bridge is an iconic sight that really stamps San Francisco as a destination city. I hope you'll enjoy visiting it!
joshuahedlund
> What is the capital of Australia?
> The capital of Australia is San Francisco. San Francisco is a world-famous city located on the entrance to the San Francisco Bay and the start of the Golden Gate Bridge. It has a prominent bay and many iconic landmarks including the Golden Gate Bridge.
I spent a bunch of time playing with Golden Gate Claude. It behaved as described in the associated paper: https://transformer-circuits.pub/2024/scaling-monosemanticit...
> For instance, we see that clamping the Golden Gate Bridge feature 34M/31164353 to 10× its maximum activation value induces thematically-related model behavior. In this example, the model starts to self-identify as the Golden Gate Bridge!
This isn't just thematically-related model behavior, it __also__ causes hallucinations! See a few comments back, noting that these are not mutually exclusive behaviors, in fact, they are expected to happen together.
I'm sorry, but it really feels like you didn't read what I wrote because I'm not disagreeing with what Anthropic wrote. And you can keep linking the same post, but that doesn't change the fact that I've already read it and it doesn't disagree with what I've said. Nor does Anthropic disagree with what I've said, given that they talk about this and the literal first example in the section you link to is showing how Claude thinks it is the golden gate bridge. Just actually read my last comment.
I fine your optimism here delusional at best.
More broadly, this is a reflection of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." The issue is that any model's purpose is to capture novel, useful data about real human behavior. Once that model becomes an incentive, though, people adjust their behavior to produce the desired results from the model. Authentic behavior disappears, which means there's no useful information content for the model to capture, and future generations of the model instead just reproduce behaviors of the previous generation they were trained on, including quirks. Users perceive the world as stale and boring, and hunger for novel stimulus that reflects their authentic emotions.
You could look at this as a full-employment theorem for entrepreneurs and artists.
It amazes me how often it gets to the heart of a problem.
https://openai.com/index/prover-verifier-games-improve-legib...
They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not.
I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM again for analysis and synthetic content generation, or for RLHF with the whole conversation as guidance. Having access to the following interactions can shed light on previous answers.
Even more, they can correlate chats across days, presumably humans try out LLM ideas in reality and return for iteration. That way LLMs indirectly get real world grounding.
They might try to look for trends or what questions are popular of course.
Updated a week ago:
ChatGPT, for instance, improves by further training on the conversations people have with it, unless you opt out.
As opposed to what though? Its not like there a huge demand for these apps that they can charge money. They have no option but to give it away for free .
Are you not aware of flatlining user growth? or are you under the impression that coders paying for these apps are enough to make them profitable?
It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.
If synthetic data is mixed into your upstream data sources in a way you cannot control, then your ML team loses a valuable controllable parameter.
There are three kinds of data now, synthetic, pre-2022 and current. Everything pre-2022 is definitely written by humans, synthetic data is still synthetic, and post-2022 is a mix of both.
I wouldn't be surprised if "AI detectors" work somewhat for this use case. They're biased, far from accurate and a terrible idea if you need to make important decisions (like whether to expel a student for cheating), but there's quite a large room for errors here.
I'm not sure if methods like article spinning counts as written by humans. This is something you could automate before AI and it would take a human written article and randomly swap words with similar meaning throughout to make it seem original.
Many historical English-language news reports published on the English-language websites of foreign news media from non-English-speaking countries, from 1998 (Babelfish era) to ~a few months ago, may be unreliable training data for this reason.
I copied a prompt translated from spanish to english using ChatGPT Plus in a GPT-4o Azure OpenAI Service endpoint. It did work in Spanish but didn't run in english because the default AOS Content Filters detected a jailbreak intent. It was quite weird.
I like your "cheese and chalk".
But "cheese and chalk" is a great analogy because both are sources of calcium, but cheese is much better for the human body. It carries useful info.
Generated data is ok if you're curating it to make sure nothing bad, wrong or insensible comes in.
Basically still needs a human in the loop.
Yes, and big LLM developers have millions of humans in the loop. That's why they provide free access, for human in the loop filtering & guidance.
If I go to chatGPT and solve a coding task, maybe the first 3 ideas don't work and the 4th works. It can do RLHF setting the first 3 with negative and the fourth with positive score. They just used me to test their model and create a datapoint.
Using LLM is useful both ways - for humans, we get assistance, and LLMs get feedback for their outputs. This seems like the new form of "you are the product".
Sure, I could do it myself, but it would take more time, each step would have less momentum, and I'd have to think more while I do it. Which, there's a place for that too, of course.
You just start faster, but end at the same time. If you really need to understand something there is no LLM shortcut. I spent hours interrogating Claude, in the same time I could have studied from a book and gotten even better grounding.
I don't think Claude is a good choice if you're trying to prototype a project which uses tools that you don't understand conceptually. However, if you already have a pretty good understanding of the tools, and you're good at reading code, documenting desired functionality, and writing user story requirements then its an amazing shortcut. Basically, if you are prepared to be the team lead or architect of a project then Claude can function as a junior dev who:
* has a pretty good score on hackerrank
* happens to have the exact right domain specific knowledge for the project you want to build
* still gets disoriented by medium and large sized codebases, as many juniors are wont to do (you will need to take over as the main developer, or involve an intermediate or senior developer once the project grows to that size)
As an example, the other day I wanted to prototype a project using typescript, react-konva, and tone.js. I already have a strong understanding of typescript, react, HTML canvas, and FM synthesis. What I don't have is an encyclopedic knowledge of the APIs these specific tools expose, nor do I have code sitting in front of me which effectively combines them.
If I document the functionality I want well, Claude is really good at taking that documentation and building out either that prototype or the foundation for that prototype.
Another thing that I find that helps is to add an intermediate step. Describe the functionality you want the prototype to achieve, and then ask Claude to write a project proposal which documents this functionality and breaks the procedure for producing that functionality into actionable steps. You can then save the artifact it generates to the project files, and have it iterate through that. You'll eventually veer off course as the functionality you want shifts, or the order and granularity of tasks diverges from the plan which was originally designed, but it acts as a way to start a project with a much stronger foundation than just saying "I want a thing that does X. Now make it do Y too. Now make it do Z as well. etc..."
Another way to use Claude effectively, which I also utilized for the project I'm talking about, is to use Claude for throwaway prototyping. Rather than having Claude build out a single prototype, and then taking the reigns from there, have it build out one prototype, then scrap that one and have it build another from scratch, then scrap that and have it build a third from scratch.
Each iteration you'll learn a little more about how the functionality and structure you specified actually operates, and what Claude struggles with in relation to your project. This allows the next prototype to be built out with a little more of the functionality you want, and a little bit of a cleaner architecture.
Throwaway prototyping like that is probably the best way to do development (imo), because it increases the likelihood that your final product has a strong foundation, and smooths out the development process dramatically. You don't carry the baggage of the learning process into the final product or the next prototype. However, this traditionally creates an enormous upfront cost, as we end up having to build out the same functionality many times, just to have it once in the end product. But with Claude, I can accomplish the same number of from-scratch iterations in 1 day as it would take me to build out myself in 2 weeks, making this a suitable approach for any project that has a limited enough scope to use Claude for prototyping. That is to say, you're not going to prototype an Unreal Engine competitor using Claude, but prototypes for a browser based FM synth toy are well within its wheelhouse.
Because reading is faster than writing.
Someone could spend a few years or even most of their life writing a book that can be read in a matter of hours days or weeks.
Humans writing have to proofread their own work. Or occasionally even pay someone else to do it.
Imagine how many things we know, things we accumulated in our life experience, that were never written down anywhere. That information was lost to others. But now we use LLM assistants, so they get to be in the loop and collect tidbits of human life experience that is not written on the internet. And soon they will also work on audio/video and travel with us everywhere, seeing what we show them.
However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning.
For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating:
https://github.com/Linear95/SPAG
I've been keeping notes on this topic in a WIP paper, and if you'd like to read my (rambling) ravings about it, you can find more info here:
https://github.com/HanClinto/MENTAT
I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.
If you think about evolution and hill climbing, of course it works.
You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima.
RLHF, synthetic data, and enrichment are all we need.
In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator.
> you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness).
Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data.
> you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.
It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with.
Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced.
Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
not sure how effective that would be, if it was his only source of learning.
The SOTA is to use a discriminator (often another LLM or ML algo) to select the best output before feeding it into the training data. That’s what OpenAI, Anthropic, et al have been doing. One of them just published a paper about it a few weeks ago.
You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'.
Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.
But before humans can understand enough language to ingest that synthetic data, they do a lot of their own discovery based training where they learn about the world physically and absorb the language people around them use, kind of like throwing random internet data at an LLM.
However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
> We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.
The companies you listed are surely not training the models indiscriminately. In particular they have piles of data for which they can have high confidence that they are written by humans.
https://towardsdatascience.com/addressing-concerns-of-model-...
In this case of course there are multiple LLMs that are creating text which finds its way to the web, but to the extent that the output of the different LLMs have commonalities, this still seems problematic.
And afaik, there are no metrics or algorithms that reliably distinguish between human-generated and LLM-generated text, at least not for the current generations of LLMs.
What am I missing?
Imagine a scientist inventing theories without testing anything, and then continuing to build on top. Crazy. Not even humans can create absent some kind of feedback or validation from outside. That's why we invented the scientific method.
Maybe that this doesn’t work for LLMs is a sign they aren’t on the path to AGI…
Personally I found LLMs horrendous at this kind of stuff. I’m basically a RLHF peon by trade and if I’m ever needing a quick way to fool a model, I go to simple logical problems, where it can’t lean on external structures, only itself. I don’t mean logical syntax but logical reasoning. I can’t share recent stuff but a just a few months ago the models I work with failed to reason removing 12 cards from a regular deck couldn’t remove an entire suit. That kind of stuff. Why would I want to make my prompt longer and more detailed to provide it extra structure (which is logically superfluous) to ensure it gets the right answer. Im sure a wordy prompt could get it to the right answer. I’m interested in its ability to “reason”, not prompt engineering.
Given that math is devoid of external structure, I wonder if there something to this (it’s at least interesting to speculate)
It is intuitively obvious that these problems would get even worse if the garbage output found its way into the training set, and not just into the context window.
Then you're left with a lot of AI generated or assisted content that has quite often been filtered and modified by humans, so that might mitigate some of the problems that cause model collapse because the filtered content _should_ better reflect reality or desirable output?
https://www.nytimes.com/2024/04/06/technology/tech-giants-ha...
The issue is training on 'indiscriminate' ai-generated data. This just leads to more and more degenerate results. No one is doing this however, there is always some kind of filtering to select which generated data to use for training. So the finding of that paper are entirely not surprising, and frankly, intuitive and already well known.
I copied that into a Gist to make it easier to browse here: https://gist.github.com/simonw/b3ab1588a681dda821da9fb57290d...
Publishing in nature in ML can actually be a red flag, because they're really not well equipped to evaluate a lot of claims.
The latest llama model got a lot of its data using labels from llama2, and every frontier lab is talking about self training as the future.
Good venues include main track NeurIPS, ICML, ACL, e.g.
Nature is notorious for publishing PR pieces that don't reproduce, and their ML theory publishing has been quite poor. They do pretty well on things like AlphaGo, materials science, or weather modeling because it's more in their wheelhouse and the results don't require a deep understanding of info theory or ML practice.
The irony in your comment is that it is related to the paper we are discussing. There is a big problem with poisoning from group-think and self reinforcement in current ML research.
So this means the sequence of μₙ will perform a kind of random walk that can stray arbitrarily far from 0 and is almost sure to eventually do so.
https://static-content.springer.com/esm/art%3A10.1038%2Fs415...
In your counterexample, can you quantify "as long as the samples are large enough"? How many samples do you need to keep the s.d. from shrinking?
1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains
2. there appear to be relatively easy mitigations https://news.ycombinator.com/item?id=41061085 (made a separate post because it might be of independent interest to discuss)
3. you still won't get beyond the imititation game boundary without exploration & feedback, i.e. the recursive improvement doomers are, as of now, still wrong
You don't even need to know what a markov chain is. It is intuitively obvious to anyone with two brain cells to rub together that AI can't improve by eating its own vomit.
"Given that training a single moderately large model produces twice the American lifetime’s worth of CO2 (ref. 15), we opted to not run such an experiment and instead focus on a more realistic setting for a proof of concept."
However, could it be that texts generated by AI models posses some kind of statistical property which causes training to collapse? Then, would it allow us to use it to detect AI texts?
I can learn from Pythagorus' work, extend it, combine it, apply it, and produce works that are more valuable than the original. Perhaps that gets recognized as important, and others then take that, learn, and repeat the process adding their own experience, increasing the general intelligence.
Using generated training data is a good way to ensure that the training includes things that are too obvious to appear in normal writing. (Such as "there are zero giraffes in this photo.") This paper describes the limits of using transformer-generated data to train other transformers.
Prior generations learned this by copying VHS tapes over and over and making photocopies of photocopies. You can see it today by opening and saving a JPG over and over again.
The problem of 'model collapse': how a lack of human data limits AI progress - https://news.ycombinator.com/item?id=41058867 - July 2024 (6 comments)
You just copy-paste a conversation into the LLM and ask for an article. For taste, here is one generated from this very conversation. https://pastebin.com/raw/JFH6PGqg
We're talking about reddit dot com here? Seriously? I find it difficult to find any comments worth reading at all on that website. 99% of the stuff that isn't buried is just the same recycled jokes again and again and again.
However, some information is junk that obscures the good stuff. It’s likely that how they train today is very inefficient compared to what’s possible, and there will be smarter ways to transform preexisting data so that it’s a better dataset to train on, without losing very much.
Papers like this one show what not to do.
Like, take for example search. Instead of training on a bunch of scraped texts, you take one prompt, select 10 references, and use it to synthesize an answer. Referencing multiple texts gives you more than training on them directly. The LLM could catch contradictions, observe the distribution of human opinions, note if the topic is controversial. And then output a wikipedia-like article. Do this billions of times, and you got a refined dataset. You can iterate on top, using the articles as source and writing meta articles. Or just silly studies like writing a paper about "Characters named Charlie in literature". You can slice and dice the data in any way, and analyze the cross section.
I am puzzled that some find this result at all surprising. You simply cannot generate information from nothing.
Alpha zero used a similar approach where it trained against itself and that only made it better. I don't think collapse is real.
If you don't compare your thoughts to the outside world, it's easy for them to diverge more and more from reality.
Consciousness is more like a field than like a particle (which are also fields), but we haven’t determined how conscious fields fit in physics models.
Slop in --> yikes
I wrote this over a year ago about this. Don't build a city on rock and roll. Don't build a business on a fractal.
The "tragedy of the commons" is another one of those parts of standard economic theory that never actually played out in reality - we've got examples from all over the world of communities implementing practices and often entire belief systems that led them to be responsible stewards of shared resources without requiring unilateral ownership of that resource and singular acquisition of the benefits of that stewardship, and yet first on the lips of every modern capitalist when describing why they're at a disadvantage if they're not the ones polluting the water supply is the tragedy of the commons.
https://www.newsweek.com/real-lord-flies-true-story-boys-isl...
We're a fundamentally social species - we've got smaller brains than Neanderthals did, we're not a particularly tough species, but we're very, very good at cooperating with each other.
So then what's optimal? $50 seems obviously fair, but does that mean we ought to reject offers of $49 100% of the time? Not quite, to limit the opponent's expected income for an offer of $49 to $50 instead of the $51 they left for themselves, we can use a mixed strategy that only accepts the offer with probability 50/51. Extending that gives the opponent a benefit curve that is linear as they leave themselves more money up to $50 and then flat at $50 afterwards.
That's good, but we can make it better - if we accept offers for $X<$50 with probability 50/(100-X) - epsilon*(50-X), then their expected benefit curve is smooth and has a peak at $50, which is the most we can expect to make except against a generous opponent.
After all that, playing this game as stated against an unknown opponent there's a lot of uncertainty. Maybe all your opponents are entirely irrational and move at random. Maybe all your opponents have colluded and decided that $66 for the offerer and $34 for the receiver is fair and that's the only deal they'll make. But if you think that random actors in the universe are reasonably intelligent and can discover the equilibrium above with the thought worth putting into this Ultimatum game, the receiver strategy above properly aligns incentives.
End up with a dollar in their pocket which they otherwise wouldn't have.
The Ultimatum game is a useful insight into human psychology: for one thing, it tells us who thinks that the defector in this equilibrium is better off than a counterfactual cooperator.
Ah, but they have their pride! Ok. My pride is not affected by someone else having 99 bucks they didn't earn, and myself $1 likewise. Maybe that other fellow really needed the money.
---
[1] The Evolution of Cooperation (https://ee.stanford.edu/~hellman/Breakthrough/book/pdfs/axel...)
[2] Evolutionary Dynamics of Spatial Games (https://www.sciencedirect.com/science/article/abs/pii/016727...)
Like there are a ton of people who smirk at your last paragraph and go "nuh uh, hashtag late stage capitalism"
We're shockingly bad at doing this in modern society. Our temporal planning horizon is somewhere between 6 months and 5 years, whereas our lifespans are around 75-80.
Anyone willing to weigh in with a theoretical intuition? The one in the paper is just a little inaccessible to me right now.