I'm not sure how weather simulation work but I always wondered why they aren't performed cellular automaton style. Bottom up rather than top-down. At each time step a given cell state is computed based on the state of its neighbors at t-1. This should be parallelizable. I feel it's also how it works in the real world anyway.
The thing is: you need the state at t-1 of all your neighbors. Then you can do a small timestep to get from t-1 to time t. And then you need the NEW state of your neighbors. That requires a fast interconnect. Which HPC machines have, unlike most clouds or commodity clusters.
In other words, yes it is parallelizable, but not trivially so, because the different grid points are coupled.
I did my PhD in physics simulations (molecular dynamics), and have the same problem there. I tried running these simulations in Google Cloud without any good performance results due to high latency (compared to HPC). I’m no GC expert though, so should be possible to improve what I did.
Available on Netflix for example: