TimeGPT-1
arxiv.org
arxiv.org
On extremely high dimensional data (I worked at a credit card processor company doing fraud modeling), deep learning dominates, but there's simply no advantage in using a designated "time series" model that treats time differently than any other feature. We've tried most time series deep learning models that claim to be SoTA - N-BEATS, N-HiTS, every RNN variant that was popular pre-transformers, and they don't beat an MLP that just uses lagged values as features. I've talked to several others in the forecasting space and they've found the same result.
On mid-dimensional data, LightGBM/Xgboost is by far the best and generally performs at or better than any deep learning model, while requiring much less finetuning and a tiny fraction of the computation time.
And on low-dimensional data, (V)ARIMA/ETS/Factor models are still king, since without adequate data, the model needs to be structured with human intuition.
As a result I'm extremely skeptical of any of these claims about a generally high performing "time series" model. Training on time series gives a model very limited understanding of the fundamental structure of how the world works, unlike a language model, so the amount of generalization ability a model will gain is very limited.
How does lagged features for an MLP compare to longer sequence lengths for attention in Transformers? Are you able to lag 128 time steps in a feed forward network and get good results?
I guarantee you there will be chartists hawking GPT-powered market forecasts.
In sklearn, if you have a single-output regressor, use this for ergonomics: https://scikit-learn.org/stable/modules/generated/sklearn.mu...
The added benefit is that you optimize each regressor towards its own target timestep t+1 ... t+n. A single loss on the aggregate of all timesteps is often problematic
In theory, this might suit the multi-step forecast use case.
I fully agree regarding how things have been so far, but I’m excited to see practitioners try out models such as the one presented here — it might just work.
[0] https://github.com/asavinov/intelligent-trading-bot Intelligent Trading Bot: Automatically generating signals and trading based on machine learning and feature engineering
As much as Transformers feel like the state of the art universal function approximators, people need to realize why they work so well for language and vision.
Transformers parallelize incredibly well, and they learn sophisticated intermediate representations. We start seeing neat separation of different semantic concepts in space. We start seeing models do delimiter detection naturally. We start seeing models reason about lines, curves, colors, dog ears etc. The final layers of a Transformer are then putting these sophisticated concepts together to learn high level concepts like dog/cat/blog etc.
Transformers (and deep learning methods in general) do not work for time series data because they have yet to extract any novel intermediate representations from said data.
At face value, how do you even work with a 'token window' ? At the simplest level, time series modelling is about identifying repeating patterns over very different lifecycles conditioned on certain observations about the world. You need a model to natively be able to reason over years, days and seconds all at the same time to even be able to reason about the problem in the first place. Hilariously, last week's streaming LLM paper from MIT might actually help here.
Secondly, the improvements appear marginal at best. If you're proposing a massive architecture change, removing observability & and explainability .........then you better have some incredible results.
Truth is, if someone identifies a groundbreaking technique for timeseries forecasting, then they'd be an idiot to tell anyone about it before making their first $Billion$ on the market. Hell, I'd say they'd be an idiot for stopping at a billion. Time series forecasting is the most monetarily rewarding problem you could solve. If you publish a paper, then by implication, I expect it to be disappointing.
It’s really quite simple. Just iterate through all possible monotone universal Turing machines where the input tape consists of all data we can possibly collect concatenated with the time series of interest. Skip the programs that take too long to halt, keep the remaining ones that reproduce the input sequence, then form a probability distribution based on the next output bits, weighted by 2^-(program size).
What’s so hard about that?
Are you telling me that there is not already physical evidence for such? I assure you there is plenty of evidence for physical space being assimilated by the incentive structures related to bitcoin and its progency.
Taken across time to a civilization that grows into further complexity, there exists a limit into how much space can be used to secure the network, and most likely even incentivizes maximizing capture of space for computational security therefore it accelerates our civilization towards creating a black hole. I couldn't come up with a better way to fast-track our way towards a cosmic environmental disaster. It's a pretty bad incentive structure long-term.
If only we could get rid of artificial scarcity.
Even in a world where computation doesn’t become more efficient it still takes up the total space available eventually due to the incentives of protecting against network failure.
I will NOT grant you this. Please, give me actual technical details on WHY it requires ever increasing compute. You've said network security, what about it requires ever increasing compute.
You can read the paper and understand the principles it is based on, which is rooted on balancing computational asymmetry amongst other concerns across a network of computers. On the most simple level, if you are aware of hashcash and sybil resistance you should be able to figure it out.
If you're still confused: then answer yourself the question why does the bitcoin algorithm adjust to computational power?
Just fucking say it, what's with the running around?
Are you unable to explain it?
But maybe you have links to others who are able to explain it, not the Bitcoin paper which obviously does not lead one to think network security will subsume all available matter for compute.
This is correct.
I work in HFT and the industry has been successfully applying deep learning to market data for a while now. Everything from pcaps/ticks to candles.
Why publish your method when it generates $1B+/year in profit for a team of 50 quants/SWEs/traders?
I assume the latency is comparably much higher but also wouldn't be surprised if microseconds generally aren't a problem, eg because the patterns detected are on a much larger scale.
Do you also believe that megacorporations as the custodians of superintelligence is bad?
I agree with your entire post, however this sentence made me think, well video is just layered vision. Why couldn't frames of vision work similar to vision? We know the current answer is it doesn't, but is it a matter of NNs can't or we haven't figured out the correct way to model it yet?
In essence, they represent the state of the art with respect to those specific tasks, as measured at the current time. Though you may desire for there to be a better (cross entropy loss, accuracy percentage so 100% acc) tuple at any given instant (e.g. (optimal expected ce-loss, an actual proof of correctness)), for the current time, it seems like not only do they do the best as a class of models for these sets of tasks, but also improve the fastest in terms of accuracy as a class of models as of late. They're quite noteworthy in that regard, fundamentally, imo. Just my 2c.
Anyone who work a lot in time series forecasting can explain this in some more details?
I’ve def used ARIMA, but only for simple things. Not sure why this would be more expensive to train and run than a Transformer model, and even if true, ARIMA is so ubiquitous that comparing resources & time would be enlightening. Otherwise it just sounds like a sales pitch and throw more obscure acronyms for a bit of “I’m the expert, abc xyz industry letters” marketing.
Having trained ARIMA models in my day, I will say that long training times and training cost -- compared to any deep learning model -- is not something that ever crossed my mind.
While I could find some excuses to exclude ARIMA, notably that in practice you need to input some important priors about your time series (periodicity, refinements for turning points, etc) for it to work decently, "prohibitive compute and extensive training time" are just not applicable.
That part is a bit wanky, but the rest of the paper, notably the zero shot capability, is very interesting if confirmed. I look forward for it to be more accessible than a "contact us" api to compare to ARIMA and others myself
arima, sure
Even if true, they could take a random subset of size 100 out of the 300k and compare on those.
>Even if true, they could take a random subset of size 100 out of the 300k and compare on those.
Sure...but there's a chance that ARIMA won't even finish training on that subset either.
If you write a paper and exclude comparisons to state of the art, this what happens.
They could have done something, and didn’t.
“It’s hard so we didn’t” isn’t an excuse, it’s just a lack of rigor.
Though I can't quite figure out how the predicting works exactly, they have a lot of test series but do they input all of them simultaneously?
EDIT: typos
Regarding your valid concern. There are several reasons for the high computational costs. First, ARIMA and other "statistical" methods are local, so they must train one different model for each time series. (ML and DL models are global, so you have 'one' model for all the series.) Second, the ARIMA model usually performs poorly for a diverse set of time series, like the one considered in our experiments. The AutoARIMA is a better option, but its training time is considerably longer, given the number and length of the series. Also, AutoARIMA tends to be very slow for long series. In short: for the 500k series we used for benchmarcking, ARIMA would have taken literally weeks and would have been very expensive. That is why we included many well-performing local "statistical" models, such as the Theta and CES. We used the implementations on our open-source ecosystem for all the baselines, including StatsForecast, MLForecast, and Neuralforecast. We will release a reproducible set of experiments on smaller subsets soon!
[1] https://nixtla.github.io/statsforecast/docs/models/arima.htm...
Yes, I've done some work in time series forecasting. The above sentence is the one that tipped me off to this paper being BS, so I stopped reading after that. :) I can't take any paper about timeseries forecasting seriously by an author who isn't familiar with the field.
Yeah, this is an ad submitted to Arxiv.
The guise of the academic paper def throws things off and it’s poorly communicated in the paper what the model benefits are.
Posting a quick write up of the talk on youtube would’ve set the right amount of rigor and expectation. (Needs marketing help, lol)
Max from Nixtla here. We are surprised that this has gained so much attention and are excited about both the positive and critical responses. Some important clarifications:
The primary goal of this first version of the paper is to present TimeGPT-1 and showcase our preliminary findings from a large-scale experiment, demonstrating that transfer learning at this scale is indeed possible in time series. As mentioned in the paper, we deeply believe that pre-trained models can represent a very cost-effective solution (in terms of computational resources) for many applications. Please also consider that this is a pre-print version. We are working on releasing a reproducible set of experiments on a subset of the data, so stay tuned!
All previous work of Nixtla has been open source and we believe TimeGPT could be a viable commercial product, offering forecasting and anomaly detection out of the box for practitioners. Some interesting details were omitted because they represent a competitive advantage that we hope to leverage in order to grow the company and keep providing better solutions and continuing to build our ecosystem.
As some others have mentioned in the thread, we are working to onboard as many people as possible into a free trial so that more independent practitioners can validate the accuracy for their particular use cases. You can read some initial impressions of the creators Prophet [1] and GluonTS [2] or listen to an early test by the people from H20 [3]. We hope to see some more independent benchmarcks soon.
[1] https://x.com/seanjtaylor/status/1694745912776749296?s=20 [2] https://www.linkedin.com/posts/tim-januschowski_foundational... [3] https://youtu.be/N0gyDVUFPlg?si=xH8oy5cjgLm-o_WD&t=457
I can appreciate people doing things from the heart, but in that case, it better be more poetic than a BBC documentary.
Inverted Transformers Are Effective for Time Series Forecasting
The zero-shot nature is certainly impressive but it doesn't look like you'd be able to _do_ much with it?
That’s the opening line, right? Uncertainty is a fact of life. With time series forecasts, the best you can ever hope to do is give probability bounds, and even then you can only really do so by either:
- limiting by the rules of the game (e.g. the laws of physics, or the rules of a stock exchange)
- using past data
The former is only useful if you’re the most risk averse person on the planet, and the latter is only useful if you are willing to assume the past is relevant.
When in fact the world and intelligent agents inside it are ensembles of ensembles of systems with various and changing confidence that flow and adjust as the world does
Regardless, need to see more benchmarks to better understand its true performance. If it holds up it would be a big win for time forecasting.
A personal note here is that they could done a better job on the tokens because the announcement was so grandiose and maybe they underestimated people with legit interest.
I’m using all libs from Nixtla with active advocation and I did not have a token; meanwhile lots of guys posting their usages on Twitter.
One idea I was interested in was after reading the paper on introducing pause tokens[1] was a multimodal architecture that generalizes everything to parallel time series streams of tokens in different modalities. Pause tokens make even more sense in that setup.
1. https://arxiv.org/abs/2310.02226
-Arm chair ML practitioner
Each token could encode a specific amplitude for the signal. You could literally just have tokens [0,1,...,MAX_AMPLITUDE] and map your input signal to this range.
In the most extreme case, you could have 2 tokens - zero and one. This is the scheme used in DSD audio. The only tradeoff is that you need way more samples per unit time to represent the same amount of information, but there are probably some elegant perf hacks for having only 2 states to represent per sample.
There are probably a lot of variations on the theme where you can "resample" the input sequences to different token rate vs bits per token arrangements.
Is that not what it is at this point ? Probabilities ?
Purely on rational basis, this, as in predicting time series data, doesn’t seem plausible. But maybe it is.
Completely false.
In reality, there are more public signals than can be exploited.
The market is most definitely NOT efficient.
LICENSE Apache-2
https://github.com/Nixtla/statsforecast/blob/main/LICENSE
Mentions ARIMA, ETS, CES, and Theta modeling
I would have liked to see more detailed benchmarking, such as the electricity, exchange and weather benchmarks.
So in principle it could work, but the problem is that these days, the big players are all doing high frequency trading with algorithms that try to predict market swings. And the big guys have an advantage: they are closer to the stock exchanges. They trade so fast that speed-of-light limitations affect who gets the trades in first. So I think the only people who could win with an LLM technique is someone who doesn't need to pay commissions (a market maker, Goldman Sachs or similar) with access to real time data, very close to the exchange so they get it fast.
(tongue firmly in cheek) Here is a bastardization of a memory of some video interview with a quantitative analyst, from over a decade ago:
"Show us your yield data and we'll TELL you your weather and political climate."
Also where's the model?
I’m willing to bet 90% of the people who upvoted this did so based on the title alone. Sorry, no time traveling GPT for you.
Those aren’t real quants. Even the sell-side quants know they aren’t real quants. For those unfamiliar, sell-side quants typically work at banks like Goldman, HSBC, JPMorgan, etc.
The real quants are buy-side quants/traders: prop shops, hedge funds, endowment/pensions funds, etc.
Out come the fingers made of foam
They finally open-sourced Rehoboam
Our societies have been thusly blessed
What it predicts could have anyone guessed
Targeted advertising wherever you roam