Comprehensive Tutorial on Time Series Modelling and Forecasting

I've seen this in real time. I don't do statistics as part of my day job, but I've had enough experience and keep up with the field to know what I'm talking about. I've seen senior engineers try to ram in an overly specified ARIMA model just to claim that they've "improved" the system. It worked far worse than whatever model we were using before was (never got to take a look under the hood of that one unfortunately), was prone to wild swings in forecasting, and was eventually deprecated and we reverted to the old model.

It's not always easy with MBA types.

I once had a mentor with clout on a 9 figure investment committee tell me that maximum likelihood estimation is "the dumbest idea" he'd ever heard.

Words like "Cramer-Rao bound" didn't get through. What worked was saying "deep learning is usually just MLE with a mysteriously effective function approximation".

Modern methods for deriving word embeddings easily beat LSA.

> I'm interested in methods that don't "subtract" simple "trends" and "seasonality"

a 2nd order difference equation can model a single harmonic frequency - that is, if your data is a pure sine and sampled at regular intervals, then

    x_n =~ a x_n-1 + b x_n-2

can model any frequency with the proper a and b values (machine precision limits apply in real world scenarios, of course); That is, if your data looks like a sine wave with a yearly period, you still need no more than one sample per minute and 2nd order model to filter it out.

It's likely not a perfect sinewave, so you'd need a lot more - but if you are incredibly lucky and your periodic underlying admits a (relatively) sparse harmonic decomposition, and the signal riding on it has (very) low amplitude compared to the periodic signal, you can model very long periods implicitly by just having enough recent samples.

> it seems like using methods that rely on stationary distributions still treat the data, after pre-processing, as i.i.d, rather than predicting values from their correlated history.

Not sure what you mean exactly. Stationarity in time series is not i.i.d. The whole point of ARIMA modelling is to model, after transforming to stationarity, the remaining temporal correlation. ARMA is just a limited-parameter fit of the autocorrelation function.

Time series isn't IID.

The name itself states this. Autocorrelation and autoregression. It regress on it's past values hence the "auto". We're interested on past value so each other value is dependent on the past.

An example of this is taking blood pressure. You would assume that taking blood pressure two consecutive days means that the previous day is highly correlated to the present day test.

Where as if you compare your blood test a year ago compare to today blood test it won't be as correlated. This is why ARIMA is dealing with correlation with time lags/lengths.

That's what I get for picking an example from a domain I do not understand well! So perhaps I'll relax my statement. Time is one of a large number of explanatory variables. The amount of information you can extract from it will be limited.

To extend your point, time lag is a feature in ARIMA and statistical time series model.

What you're doing is looking at autocorrelation base on time lags. That's what ACF and PACF graphs is displaying.

Mostly from my own knowledge/experience:

- Multi-variate: text book treatments tend to focus mainly on Vector Auto Regression (VAR) models. Unrestricted VARs scale very badly in vector dimension, so the often end up in some regularized form (dimension reduced by PCA or Bayesian priors). Lütkepohl's textbook is the standard reference.

VAR type models in my view not very practical for most business time series. You should probably not waste too much time on them unless you're really into macro-economic forecasting, in which case you're wasting your time anyway :). VAR forecast accuracy in macro-economics is not great to put it mildly, but we have nothing really better).

An alternative to VARs for multivariate time series are state space models, which are described mostly in Durbin&Koopman and Andrew Harvey's time series textbooks. These model types was recently popularized in tech circles by Google's CausalImpact R package (though that package I think only implements the univariate model).

- Multi-model: if you need to model some generic non-Gaussian time series process some slow generic simulation method (MCMC, particle filtering). I can't recommend any good reference since I haven't kept up with the literature for about 15 years. I only remember a bunch of dense journal papers from that era (e.g. https://en.wikipedia.org/wiki/Particle_filter#Bibliography)

- Irregular: if the irregularity is mild (filling up a relatively small number of gaps/missing data), you can do LOESS, smoothing splines, Kalman filtering, which should all get you pretty similar results. If your time series are extremely irregular, probably no generic method will do well and you probably need to invest some days/weeks/months into a fairly problem/data-specific method (probably some heavily tuned smoothing spline)

It hasn't really, at least not in production. Academics are now publishing a lot of papers using Deep Learning or RL, but you won't usually see those in live systems.

In live systems, latency is usually more important than a "better" model - A model that takes milliseconds to make slightly better predictions is too slow when you're working on nano- to microsecond scales, often on specialized hardware. Really, the "AI" part is less important in HFT than you may think. It's often more system/infrastructure.

This is for HFT specifically, perhaps it has had more impact on longer time horizons, or something like portfolio management. My impression is (but I may be wrong) that there aren't that many people doing something in between HFT and much longer (minutes to days) time horizons, something like milliseconds to seconds. Maybe there is an opportunity there for some of the newer AI techniques.