Hacker News Reader

Comment by gillesjacobs | Hacker News Reader

This wouldn't pass peer-review of it were a paper. Major issues:

- No fair hyperparametrization for Neural Prophet. They mention multiple times they used default hyperparams or ad-hoc example hyperparams.

- 3/4 benchmark datasets (one they didn't finish training) where ETS outperforms is not strong evidence of all-round robustness. Benchmarks like SuperGlue for NLP combine 10 completely different tasks with more subtasks to assess language model performance. And even SuperGlue is not uncontroversial.

View on HN

This comparison was not intended as an academic paper. However, we are confident that predicting and evaluating performance over more than 55k series from the M international forecasting competitions (standard benchmarks in the field) is a good start. After corresponding with the author, we included additional data sets from the electricity domain like Ercot and ETTM2. The dataset that did not finish training shows that Neuralprophet simply does not scale. (We let the training run for over 73 hours with 96 CPUs -as NeuralProphet implementation is restricted to CPU-, and we canceled it after investing 288 USD in it) Suggestions to strengthen our experiments (without spending hundreds of dollars) are highly welcomed.

Regarding the hyperparameter selection, we went beyond the original NeuralProphet paper, tunning and actively trying to help improve its performance. We even made a PR fixing NP bugs in the process.

This is ironic, because the Neural Prophet paper wasn’t reviewed either and didn’t benchmark it on anything other than Prophet.

While the results don't prove the superiority convincingly, it does seem that ETS is a good candidate as a first go-to in practical applications. "In practice, practice and theory are the same. In theory, they are not.”

In theory practice and theory are the same. In practice they are not.

HN pedantry ruins the fun of wordplay yet again.

In theory, his version is right. In practice, yours.

The difference between theory and practice is greater in theory than in practice.

In theory there is no difference between theory and practice. In practice there is.

- Yogi Berra

"Sometimes malpractice is just malpractice"

- Sigmund Freud

Reply on news.ycombinator.com