I wonder what portion of time on those supercomputers is used in optimizing the code itself; and how much could be done.

With those massive clusters you could afford testing in parallel trillions of mutations to your simulation kernels and prove the correctness of the fastest ones -- pruning should be extremely fast by finding counterexamples. Or even higher level architectural optimizations. Surely they could afford at least a few % of the total time on this pre-optimization (although the tools to achieve this automatically would need to be quite sophisticated!).