I am dearly hoping that they are using the current "AI" craze to talk up the machine learning methods they have presumably been using for a decade at this point, and not that they have actually integrated an LLM into a weather model.
Even before LLMs got big, a lot of machine learning research being published were models which underperformed SOTA (which was the case for weather modeling for a long time!) or models which are far far larger than they need to be (e.g. this [1] Nature paper using 'deep learning' for aftershock prediction being bested by this [2] Nature paper using one neuron.
I'm not saying this is an LLM, margalabargala is not saying this is an LLM. They only said they hoped that they did not integrate an LLM into the weather model, which is a reasonable and informed concern to have.
Sigmar is correctly pointing out that they're using a transformer model, and that transformers are effective for modeling things other than language. (And, implicitly, that this _isn't_ adding a step where they ask ChatGPT to vibe check the forecast.)
The quoted NOAA Administrator, Neil Jacobs, published at least one falsified report during the first Trump administration to save face for Trump after he claimed Hurricane Dorian would hit Alabama.
It's about as stupid as replacing magnetic storage tapes with SSDs or HDDs, or using a commercial messaging app for war communications and adding a journalist to it.
It's about as stupid as using .unwrap() in production software impacting billions, or releasing a buggy and poorly-performing UX overhaul, or deploying a kernel-level antivirus update to every endpoint at once without a rolling release.
But especially, it's about as stupid as putting a language model into a keyboard, or an LLM in place of search results, or an LLM to mediate deals and sales in a storefront, or an LLM in a $700 box that is supported for less than a year.
Sometimes, people make stupid decisions even when they have fancy titles, and we've seen myriad LLMs inserted where they don't belong. Some of these people make intentionally malicious decisions.
Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.
FGN (the model that is 'WeatherNext 2'), FourCastNet 3 (NVIDIA's offering), and AIFS-CRPS (the model from ECMWF) have all moved to train on whole ensembles, using a cumulative ranked probability score (CRPS) loss function. Minimizing the CRPS minimizes the integrated square differences of the cumulative density function between the prediction and truth, so it's effectively teaching the model to have uncertainty proportional to its expected error.
GenCast is a more classic diffusion-based model trained on a mean-squared-error-type loss function, much like any of the image diffusion models. Nonetheless it performed well.
They aren't, but both of them are transformer models.
nb GAN usually means something else (Generative Adversarial Network).
I was looking at this part in particular:
> And while Transformers [48] can also compute arbitrarily long-range computations, they do not scale well with very large inputs (e.g., the 1 million-plus grid points in GraphCast’s global inputs) because of the quadratic memory complexity induced by computing all-to-all interactions. Contemporary extensions of Transformers often sparsify possible interactions to reduce the complexity, which in effect makes them analogous to GNNs (e.g., graph attention networks [49]).
Which kind of makes a soup of the whole thing and suggests that LLMs/Graph Attention Networks are "extensions to transformers" and not exactly transformers themselves.
(Well, not necessarily architecture. Training method?)
The following snippet highlights the algorithm used to determine <thing>
```fortran
.....