Anomaly Detection of Time Series Data Using Machine Learning and Deep Learning | Hacker News Reader

Anomaly Detection of Time Series Data Using Machine Learning and Deep Learning

xenonstack.com

91 pointsmyth_drannon20 comments

TLDR: This is a high level overview of all the terms in the topic (Anomaly, Time Series, Deep Learning). If you have a passing familiarity with the problem space, there is nothing new here. There is an ad for xenonstack services at the bottom.

It taught me about the AnomalyDetection R package, which was nice.

Indeed: zero real content, just an ad for their platform.

> This is because the oscillations are dependable upon the business cycle.

No, no, no, no. The "business cycle" is a misnomer; it's not a cycle. It doesn't fit any descriptions of a "cycle" except for the fact that it sometimes goes up and sometimes goes down.

Saying that it's a cycle implies you can time downward trends like crashes or predict their intensity, which, anyone in finance or maroeconomics will tell you is impossible

Do they just mean mean "seasonal/cyclic variations in business"?

Yes, there are seasonal variations in business (eg. agriculture!) but those aren't the "business cycle" which refers to the boom/bust recurrence (like the 1990s boom and 2000 tech bubble pop, the 2000-2008 boom and financial crisis, etc.)

Have you tried to look "cycle" up in a dictionary?

    noun: cycle; plural noun: cycles

    1.a series of events that are regularly repeated in the same order.

So no, the "business cycle" is not a cycle. In the sense that you won't successfully fit a sin curve through the GDP that predicts timing or magnitude of significant deviations from the log GDP time series trend.

You forgot to include the example:

"the recurrent cycle of harvest failure, food shortages, and price increases"

Can you predict the timing and magnitude of food shortages and prices?

Saying that it's a cycle implies you can time downward trends like crashes or predict their intensity, which, anyone in finance or maroeconomics will tell you is impossible

Just because something is hard doesn't mean it's impossible; You can predict anything with a degree of certainty. Then there are things like uncertainty and radical (Knightian) uncertainty.

Any chartered actuary would tell you the same if you could afford one, and they will know quite a bit about business cycles and how to model them, too.

Except you won't in macroeconomics. The simple problem in macroeconomics is lack of data -- history happened only once so you can't test counterfactuals.

You can try to build a model, but the confidence intervals for the next big crash are going to be either useless (way too wide) or bullshit if what you're trying to predict is the next market crash. Building a classifier around it machine learning style will not fundamentally help, either.

You can try it: here is an open source, serious, full scale macroeconomic model by the NY Fed [1]. Try to predict the next large crash with it and report back

[1] https://github.com/FRBNY-DSGE/DSGE.jl

This would be the same macro-economic theory that even macro-economists will now grant is fundamentally flawed.

The rest is left as an exercise to the reader.

Business cycles are often a component of what is a composite function of the time series. So the time series itself might not have perfect cycles but the business cycle is usually one of the principal components of the time series, which is what I think they were referring to.

For example, the price of commodities does depend on the growing season (which is predictable) but other factors which are not cyclical so that the price itself is not easily predicted.

The business cycle is perfectly apparent... in retrospect. If it was apparent in... forespect... we'd all be rich!

No, information that is available to anyone has no value.

If you're interested in more research level techniques, check out some of the work that's been done using minimum description length (MDL). It basically seeks to minimize the size of the model plus the size of the residual data (that the model doesn't predict) and uses Huffman compression as a proxy for the Kolmogorov complexity of a string:

"Discovering the intrinsic cardinality and dimensionality of time series using MDL" https://pdfs.semanticscholar.org/2049/50b3cd9cf2eef52f957df6...

A few papers that build on that one show how to incorporate anomaly detection (in the sense that the anomaly is incompressible).

And more on research-level techniques:

Ahmad et al. "Unsupervised real-time anomaly detection for streaming data": http://www.sciencedirect.com/science/article/pii/S0925231217...

Anomaly detection benchmark (dataset and evaluation): https://github.com/numenta/NAB

Since this article is very high level, are there other resources that show how to apply some (or all) of these topics in actual code or on a sample set of data?

http://machinelearningmastery.com/blog/ has some good tutorials on time series and ML in general

The first few words of this title sound way too sci-fi. Very disappointed when I read the whole thing together.

Edit: supposed to be a light-hearted observation, I haven't read the article!

Reply on news.ycombinator.com