Meta AI Megabyte: Predicting Million-Byte Sequences with Multiscale Transformers

arxiv.org

5 points·ftxbro··3 comments

My main argument against the AI doomsayers has so far been that the current scaling laws simply make runaway singularity style scenarios algorithmically impossible (if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall).

If this n^(4/3) alt transformer compute scaling is real (and there’s been many a pretender, so it’s too early to tell), then that could fundamentally change the overall AI scaling law, substantially lowering the brick wall.

I’m not worried about the current crop of generative AI. I am however both curious and concerned about what the tsunami of talent and $$$ chasing the current trend will achieve.

And I guess this may (or may not) be one of those game changing insights.

Meta AI is really out here giving model architecture and training details like it's 2022.

It's a business and politics. Meta doesn't make money on AI. It wants to have 'cool' aura, to be attractive. Meta gave away much more than, say, OpenAI. This lowers the entry point for startups. Other companies in stead are trying to create kill zones around their businesses. Some of them advocating for regulations, one way of doing it. I'm on Meta's side, as personally interested. Second reason is it's good for the progress in general in both ways, AI itself and its applications in real world. I understand not everybody like it. But it's unstoppable, the only questions are where and how fast. If you believe in some global blocks by united nations, then look at how those nations deal with climate change.

Reply on news.ycombinator.com