Hacker News Reader

Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.

At small input size, yes the MLP dominates compute. At large input attention matters more