https://news.ycombinator.com/item?id=48706307
Even if it were written by hand, it’s a very poor and frankly stupid essay about an interesting topic. “The model's attention is a fixed quantity, and it has to add up to one, so the more things you make it look at, the less of that attention any single earlier thing can keep.” This is borderline gibberish and it outright rejects the interesting question about LLMs and attention, namely that they have very different capacities from us. LLMs can read an entire OpenAPI schema in seconds and immediately construct valid requests from it. The article first points this out, and then switches to arguing that LLMs have similar limits to us. It’s completely incoherent.