If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet.

Self-learning/improving would be even better but that's still a long way to go.