Hacker News Reader

Comment by downWidOutaFite | Hacker News Reader

The "regulatory capture" conspiracy theory makes no sense to me. It takes 9 figures in cash to create and run one of these super big models. Only big tech was ever going to create them, and big tech is already very experienced at navigating regulation, regulation wasn't ever going to stop them from competing. And in general, our democracy works better than the nihilist libertarians give it credit for.

View on HN

I bet it'll be 6 figures within 18 months.

Even if it was free to train, FAANGs can beat OpenAI on spending to follow regulations.

The thing about these models is compute scales quadratically with model dimensionality and memory scales quadratically with sequence length.

We are nowhere near diminishing returns for either variable, so sure current models maybe scale quickly but the cutting edge will want as much compute as possible for a long time.

That’s kind of the humor of everyone saying this leak somehow leaves OpenAI vulnerable.

The work isn’t deciding if an MoE is the right architecture or not, it’s how to run 25k GPUs concurrently in a fault tolerant way (likely the true reason for the deep Azure links).

Memory does not scale quadratically with sequence length.

During training, you have to store a dot product of Q and V that has dimension Ncrt^2.

That's quadratic scaling, no?

presumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135

I mean, sure you can work around it, but from your own link:

>since the time and memory complexity of self-attention are quadratic in sequence length

Except in practice this is not true, and hasn't been for more than a year. It's not just a workaround either -- FlashAttention is both faster at runtime and uses less memory.

I interpreted openAIs regulatory capture bid as more of an attempt to create competition-hostile regulation than an attempt to reduce regulation to cut costs.

My point is that openai's competitors have no problem handling regulations.

Reply on news.ycombinator.com