> If "simpler" models cluster close to more complex models, the simpler models are more desirable.
Well, it would suggest you aren't winning very much for your more complex model, at the very least.
> I wonder if all over-fitted models cluster in one region in the meta-SNE space, or do they show up as noise?
This corresponds to an empirical question: do models overfit in the same way, or different ways?
One small experiment I did, which might offer some intuition here, was training lots of extremely small networks on MNIST, with hidden layers of only 1, 2 or 5 neurons. What do they look like in meta-SNE?
Well, it turns out that when you only have a very small number of neurons, they latch on to random useful features! These randomly selected features don't tend to be the same, so you end up with the models horribly disagreeing on what is similar and what is different.
As you increase the number of neurons, the space of features they look at, if not the features of individual neurons, becomes similar across models. And so the models agree more, and cluster more tightly.
...
Another fun idea for using meta-SNE is ensemble models. We know that training a bunch of models and then averaging their results (ensembling) can improve results a lot. When is this helpful? My guess is that the farther apart compatibly good models are in meta-SNE space, the more ensembling will help, because they've learned different things.