EarthPT: A time series transformer foundation model

Wanted to share the code release of EarthPT, a model that predicts future satellite observations in a zero shot setting! I'm the first author so please shoot any questions you have at me.

EarthPT is a 700 million parameter decoding transformer foundation model trained in an autoregressive self-supervised manner and developed specifically with EO use-cases in mind. EarthPT can accurately predict future satellite observations across the 400-2300 nm range well into the future (we found six months!).

The embeddings learnt by EarthPT hold semantically meaningful information and could be exploited for downstream tasks such as highly granular, dynamic land use classification.

The coolest takeaway for me is that EO data provides us with -- in theory -- quadrillions of training tokens. Therefore, if we assume that EarthPT follows neural scaling laws akin to those derived for Large Language Models (LLMs), there is currently no data-imposed limit to scaling EarthPT and other similar ‘Large Observation Models.’(!)

Code: https://github.com/aspiaspace/EarthPT

Paper: https://arxiv.org/abs/2309.07207