Except the goal is usually to compute, not to transfer data, so the interconnects are a necessary evil.

Ultimately what everybody wants is a balanced system (that acts balanced on the workloads of interest). Put another way: you don't want any one part of the system to be massively overprovisioned in a way that causes expensive bottlenecks. The Cray T3E had fairly slow processors, but you coudl keep the entire supercomputer at very high utilization and get better throughput that machines with faster processors and a slower interconnect.

Good question. Of course you can make the same argument about bandwidth, the question is whether you want to make such an argument.

If you are building a system whose purpose is to transfer data (e.g. a router), every second that system is "computing" and not "transfering" you are paying for Watts that aren't doing anything useful.

A supercomputer purpose is to compute stuff, not to move data around, so making that argument about a supercomputer doesn't make much sense because nobody cares whether the supercomputer is using all its bandwidth or not as long as its using all of its compute capabilities.

So that's like making the argument that everything that isn't a plane, well, isn't a plane. Sure, my guitar isn't a plane, but it is not the purpose of my guitar to be one :D

No, computing is happening in parallel with data being sent/received