Automated Time Series Processing and Forecasting | Hacker News Reader

Automated Time Series Processing and Forecasting

github.com

38 pointsfedegr8 comments

Looking at the readme.md, I don't see what time series models are implemented. For example, are autoregressive models, ARMA models, and exponential smoothing implemented? Probably, but I don't see those terms listed.

We missed that, sorry. At the moment, for forecasting the pipeline uses the mlforecast library (https://github.com/nixtla/mlforecast) that builds upon sklearn, xgboost and lightgbm. In addition, we are about to include state-of-the-art Deep Learning models from the nixtlats library (https://github.com/nixtla/nixtlats/).

20% better than prophet sounds great.

But then, I've read a couple of posts discussing in depth how prophet itself was not always the go-to model [1], so perhaps some caution is in order.

[1] https://www.microprediction.com/blog/prophet

[edit] And this one, linked in [1]: https://kourentzes.com/forecasting/2017/07/29/benchmarking-f...

If this is on the M5 data set as a benchmark, Prophet wouldn't be a great model choice at all. It's fantastic for dense, long, univariate time series with strong seasonality. M5 is very very sparse and intermittent. The M5 competition was won with lgbm, which isn't a time-series model at all. (Edit: looks like this defaults to something like lgbm, so, unsurprising results.)

It's not hard to beat Prophet by >20%. As the paper itself states, it's designed for rapidly producing good enough forecasts at scale by domain experts with no stats background. If you have a small team consisting of domain experts, engineers and data science/statisticians, you should be able to build something which handily beats Prophet. If you're careful, you should be able to design a pipeline which can do the model tuning and selection in something resembling an "automated" manner (but it requires extreme care, and Prophet can't save you from the dangers herein either).

Prophet has a relatively narrow use case where it makes sense; not coincidentally, it makes sense at a place like Facebook where the sheer variety of problems to be tackled cannot scale to such a foregoing team for each such problem.

Prophet has received a lot of backlash for workflows it never intended, nor claimed, to be able to tackle. That's undeserved in my opinion. Zillow didn't lose money because it used Prophet, for example. But it still really shouldn't be used as a benchmark like this.

Thanks for your comments.

We agree that in most cases prophet is not a good benchmark; however, we wanted to use it because it is one of the most used libraries in forecasting. For that reason, we also tested the solution against AWS Forecast obtaining better results.

Besides the better performance and scalability, the pipeline we created considering all the stages of time series forecasting: preprocessing (e.g. missing value imputation), creation of static and dynamic features, forecast generation, and finally evaluation using data sets of important competencies. (https://github.com/Nixtla/tsfeatures)

On the deployment side, the entire pipeline can be quickly deployed in the user's cloud using terraform. This allows for less development time. (https://github.com/Nixtla/nixtla)

Great, another open source tool purporting to solve time series analysis in an "automated way" that my manager will link me tomorrow and ask me to review (as an aside, attempting automated statistics of any kind is incredibly dangerous and misguided, but especially so for time series).

Why should I use this over Darts[1] or just Statsmodels[2], if I need more lower level access and diagnostics? Both of these are far more established.

I dislike that Facebook Prophet was chosen as a benchmark; it's not a difficult benchmark to beat for the majority of time series use cases. It signifies to me that this project might targeting cargo cult data science. Prophet is not particularly good at non-daily timeseries and non-seasonal timeseries. The paper itself admits this[3]. Moreover, it's just a generalized additive model that incorporates holidays.

I don't intend to sound demeaning here, really. But I'm trying to understand what the point is. This doesn't look like someone's weekend project, but we already have plenty of established projects which tackle this effectively.

There are three major markets for time series work:

1. You're an analyst with a lot of domain knowledge who needs to analyze daily, seasonal data but you don't have a strong statistical or engineering background. This person should probably just choose Prophet (again, the developers of Prophet explicitly acknowledge that it's designed for scalable good enough models by non-stats people, not for the best model given the data).

2. You're a data scientist with a good statistical background and you need to produce forecasts. You can afford to dig into what the model is doing and select a model based on a series of diagnostics and knowledge about the data itself. This person should probably choose a more complete suite, like Darts. The important thing here is developing good models quickly while being able to do more than just press a button.

3. You're a data scientist (or statistician) which a very strong statistical background who needs to produce the best model they can for answering a specific question. This person is probably going to use R, Stan, Statsmodels or PyMC to come up with something bespoke. They may or may not need to systematize it, but they don't need to produce quantity over quality.

How does this thing improve the state of the art for any of these markets?

1 https://github.com/unit8co/darts

2 https://www.statsmodels.org/stable/index.html

3 https://peerj.com/preprints/3190/

The pipeline we have developed improves the state of the art in the markets you mention in the following aspects:

1. It is a fully automated end-to-end pipeline for forecast generation. The pipeline considers preprocessing such as missing value imputation, feature generation (static and dynamic), forecast generation, and also a module to validate forecasts on important time series competition data sets.

2. Users can deploy the pipeline in their cloud quickly. We use terraform (https://github.com/Nixtla/nixtla/tree/main/iac/terraform/aws), so it is very simple to deploy the pipeline on AWS. We are working to release versions of terraform on other clouds such as Azure and Google Cloud.

3. Users can use their own models. Just create a fork of the repo and make the appropriate modifications to include any model the user wants to deploy. On our side, we are working to include Deep Learning models with the nixtlats library (https://github.com/nixtla/nixtlats/) that we also developed.

About benchmarking using statistical models, we highly recommend using statsforecast (https://github.com/Nixtla/statsforecast) that we created. It is designed to be highly efficient in fitting statistical models on millions of time series. More complex models can be built on the results to get a positive Forecast Value Added.

Reply on news.ycombinator.com