Aren't you are ignoring use cases where all cores read shared data, but rarely contentiously write to it. You should get much more read bandwidth and latency than over a network.
I recognize what you are calling out/that performance will be the same on some workloads if you distribute or not. I would just point out less manufacturing causes less e-waste/I would rather live in a world where Nvidia sells 50 million 10*0 cards, than 500 million 1030 cards to create the same amount of compute in the world. It's not just the power costs to consider (but it could be there is a reality where running 500 million 1030s for their lifetime wastes so much less power, that the manufacturing costs to the planet are worth it).
Not only that. On-chip gives you high precision synchronous time (all cores observe the same time) so you can use synchronous distributed algorithms that are unsuitable for ethernet networks.
This type of hardware allows for much better solutions to some problems.