> It's not practical to manually design an ideal, on-demand, just-right limited-parameter model for every dataset we are presented with.
What if that could be automated?
What if that could be automated?
It's not more popular for a few reasons: 1) you first need to train a full general model anyhow 2) interpretability is nontrivial and not guaranteed 3) once you make the architectural changes, you can't commit to that architecture as you might miss out in the future with more advancements 4) with modern transformers, there is limited amount of architectural "play" happening.