im still skeptical because new techniques are going to give an order of magnitude efficiency boost to transformer models, so 'just waiting' seems like the best approach for now. I dont think they will be able to just skip to the finish line by having the most money.
People keep saying we don’t have enough data. I think there is a lot more data than we realize, even ignoring things like NSA.
I couldn’t find the average message length, but I would guess it’s fairly short (with a fat tail of longer messages).
To make the math easy, let’s say the average length is ~10 tokens. I’d be surprised if that isn’t correct within a factor of 2 or so.
So we have 20 trillion tokens per year from text messages in the US alone. And this is high-quality conversational data.
The annual numbers were fairly constant in recent years (and then it drops off), so the past decade of US text messages is about 200 trillion tokens! That’s a metric fuck ton… Much larger than any dataset existing models have been trained on, I believe.
I would guess phone transcripts would be an order of magnitude larger at least. Talking is a lot easier than typing on a phone.
You could train an absolutely insane model with that amount of data… Damn.
[1] https://www.statista.com/statistics/185879/number-of-text-me...
It's mind-blowing to me that with all of Google's data, Google isn't the far and away leader in this new space. I have to believe they're paralyzed by the fear of legal repercussions.