ASAP: Automatic Smoothing for Attention Prioritization in Time Series | Hacker News Reader

ASAP: Automatic Smoothing for Attention Prioritization in Time Series

futuredata.stanford.edu

69 pointsSEJeff23 comments

The examples they give are periodic so removing the periodicity is the big win, not removing the noise.

How did ASAP perform better than ARIMA and other similar algos should be the main question, not how does ASAP compare to nothing.

Isn't Arima used for forecasting? Pardon me I'm not very informed about it but how is Arima used for smoothing?

Various ARIMA methods are used to deseason data for forecasting, but yes a better more directly comparable example might have been methods like STAMP and STL.

Blog post with additional info on why this is useful:

http://dawn.cs.stanford.edu/2017/08/07/asap/

In short - is this a moving average with window range automatically tuned for optimum detail to smoothness ratio?

Yes.

A moving window average is a poor digital filter in the frequency domain [0]. You can get better results with more control on which frequencies are removed by implementing a FIR or IIR (especially Butterworth) digital filter.

[0] https://ptolemy.eecs.berkeley.edu/eecs20/week12/freqResponse...

Yes, optimizes window size for moving average

Optimal detail is measure by:

* minimizing variance of 1-step difference (point-to-point fluctuation)

* maintaining kurtosis (outliers not lost)

+optimizations to reduce search space

Comments are saying yes- but is it once you find the optimal window size, you fix it? Or does the window size vary over the data? Because if the latte wouldn't that be disingenuous to outliers and such?

Is there a python implementation of it as well?

https://github.com/stanford-futuredata/ASAP/blob/master/ASAP...

Just tried it. I was using the seasonal_decompose from statsmodel in python for a project. I have to say ASAP looks great!

It really is a gem for analysis of time series data. I was lucky enough to watch Kexin's talk live @ Monitorama on ASAP. Really great stuff.

Does he explain how this is better than any other methods?

I don't believe she did. She investigated the specific problem of helping users to identify deviations in time series by applying smoothing.

Here is a link to her presentation: https://vimeo.com/221062931

It is pretty fascinating, I do recommend giving it a watch.

Thanks!

I upvoted the authors comment linking the Arxiv paper, that said, I think the presentation oversold the claim that smoothing is always a good thing to do. In particular, showing that excel, grafana, graphite etc, do not smooth by default, (apparently because it is a hard) looked like a low point of the talk, sorry. Smoothing has its uses but its hardly the panacea.

What is an appropriate smoothing is very context sensitive and I would like the consumer of the plots be in charge of that. Sometimes smoothing away the high frequencies or transients are exactly the wrong thing to do (for example, when you care about transient spikes)

I have seen more than enough damage done by plotting tools sneaking in a smoothing without informing the consumer. Especially so in the monitoring space. When the user expects raw plots it better be raw, and raw is a good default. Smoothening should be invoked explicitly.

Thanks for the comment. I strongly agree that smoothing is not always beneficial and that the users should be informed and have control of the amount of the smoothing that is being applied to their plots.

We developed ASAP with the intend to complement, instead of replacing the raw plots. We view ASAP more as a "smart default" - if you were to expose a slide bar of different moving average window sizes that users could apply to their visualizations, where should a reasonable default be on that slide bar?

In addition, ASAP does not always choose to smooth the visualizations. In some cases (e.g. Figure C.1 in the arXiv paper) where the raw plots already have a few well-defined outlying regions and smoothing would only decrease the extent of such deviations, ASAP leaves the visualization unsmoothed.

That sounds very sensible.

Thanks!

Update: Hi Jeff!

Small world buddy! Shoot me an email (from the github link on my profile)

Hi, thanks for the interest. I am one the of the authors of the paper.

To clarify, the main goal of ASAP is to automatically select a reasonable smoothing parameter for time-series. Moving average is already widely used for time-series visualization and dashboards, but setting the width of the window is a usually manual process.

If you're interested, our extended version of the paper (https://arxiv.org/pdf/1703.00983.pdf) contains some empirical comparisons of the moving average function with other smoothing functions like FFT and the Savitzky–Golay filter.

The examples series given have seasonal/periodic effects. An MA seems like the wrong tool and the data should be deseasoned first. The MA in these cases isn't removing noise it would seem.

Reply on news.ycombinator.com