A reason for the slow adoption of accelerators is that these code bases are pretty old and mostly Fortran based. I don't think we are going to see a shift to accelerator-based weather models any time soon. The code bases are pretty old and most scientists working on these models don't have the experience or do not want to work in a programming methodology that works well with accelerators (at least that's my experience). Also in a lot of cases people just like to work with an already existing code because they are just doing a PhD and don't want to translate everything to a new language.

The COSMO model which I'm working on is 18 years old. We ported the compute intensive part of the model to C++ using a stencil library and integrated OpenACC pragmas (think OpenMP, but for GPUs) in the rest of the code base.

I'm not a big fan of OpenACC, because it requires the user to make assumptions on the underlying hardware and quite a bit of thinking to get high-performance code. It is quite time consuming to integrate the pragmas so that the code performs. OpenACC capable compilers used to be very unstable, we regularly got compiler breakages and regressions. It got a bit better since we send the code to the vendors, but we still see regressions from time to time. All (usable) OpenACC implementations are proprietary, so we have a vendor dependency. The PGI OpenACC binaries used to be pretty slow, but it is now almost up to par with the Cray binaries.

The successor model to COSMO, ICON, developed at DWD and MPI is also written in Fortran using OpenACC pragmas (and I think OpenMP pragmas because they plan to use it on XeonPhi). The code is interesting in the sense that they are using a icosahedral grid, which stands in contrast to the square grid COSMO uses: Data accesses are not straightforward. Keep in mind that Fortran does not have easy abstractions for data accesses outside of square grids, so you have to use a couple of macros/functions/etc.. to get to the fields you need. Disclaimer: I have not seen the code myself, but the DWD certainly knows how to write high-performance code, so I'm certain most of the code is optimized.