Unlocking the power of time-series data with multimodal models

To me, this basically says "LLMs aren't pre-trained on enough 1D timeseries data" - there's a classic technique in time series analysis where you just do a wavelet or FFT on the time series and feed it into a convnet as an image, leveraging the massive pre-training on, e.g. ImageNet. This "shouldn't" be the best way to do it, since a giant network should learn a better internal representation than something static like FFT or a wavelet transform. But there's no 1D equivalent of ImageNet so it still often works better than a 1D ConvNet trained from scratch.

Same applies here. An LLM trained on tons of time series should be able to create its own internal representation that's much more effective than looking at a static plot, since plots can't represent patterns at all scales (indeed, a human plotting to explore data will zoom in, zoom out, transform the timeseries, etc.). But since LLMs don't have enough 1D timeseries pretraining, the plot-as-image technique leverages the massive amount of image pre-training.

It kind of feels criminal to do time-series analysis with multimodel models and not use any traditional numerical models to provide a baseline result. It's an interesting result though.

I don't want to sound too dismissive of someone's hard work but I was kind of hoping for something more sophisticated than showing an LLM the image of a plot. Using the article's example, I would be interested in understanding causes (or even just correlations) of near falls - is it old people, or people who didn't take their vitamins, or people who recently had an illness, etc.? What's the best way of discovering these that isn't me slicing the data by X and looking at the plot.

There is a surprisingly common use case for "quick and dirty univariate time series forecasts" that are basically equivalent to giving a small child a pencil, and asking them to draw out the trendline. The now-deprecated Prophet model from Facebook (which was just some GAM) was often used for this. Auto-ARIMA models, ETS etc are also still really commonly used. I also see people try to use boosted trees, or deep learning stuff like DeepAR or N-BEATS etc even though it's rarely appropriate for their 1k-datapoint univariate time series, just because it gives off the impression of serious methodological work.

There are a lot of use cases in business were what's needed is just some basic reasonable-ish forecast. I actually think this new model is really neat because it completely dispenses with the pretense that we're doing some really serious and methodologically-backed thing, and we're really just looking a basic curve fit that seems pretty reasonable with human intuition.

This is really neat. I imagine this will be an entryway for LLMs to creep into more classic data science / ML workloads.

IMO if doing this, you should avoid text in the charts entirely (as the title can sometimes I think lead the models astray, such as the clustering title I think will bias it to find clusters even if none exist). Presuming you are the one making the chart and not just prompting with another image.

I believe the text in the image will be more prone to misinterpretation that direct text in the prompt anyway, https://andrewpwheeler.com/2024/07/16/using-genai-to-describ...

Kelly et al took similar approach to trading. The idea was that human traders looked at charts on the screen and “intuitively” made trading decisions.

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3756587

Not sure how much or if at all anything valuable was unlocked. Given this amount of paid talent and this amount people involved, surely the amount being unlocked should be proportional, was it?

Has anyone seen an example of time series analysis via transfer learning / fine-tuning an LLM to process and predict multivariate data as xml or something? e.g. : <speed 45> <speed 46> <heading 123> <speed 47> <speed 47> ...etc