Hacker News Reader

Comment by dmillar | Hacker News Reader

PSA: If/when using these tools, know the difference between Kalman filter and Kalman smoother. And be especially careful when using to make forecasts, forward estimates, predictions, etc.

Beyond the scope of this comment and post, but mind two-sided filters (as the Kalman smoother/HP filter uses) as you could be incorporating future/unknown data into your model.

View on HN

I've actually been looking for a good description of the Kalman smoother, to process recorded data without phase shift, but I haven't found any. I have text books on the subject from school, but I can't really translate it to real code in this case. I understand using and updating the filter sequentially, but how do I use information from both previous and future data for each point?

This is my question as well, I've read multiple sources that imply the Kalman smoother would work for my problem, but not exactly _how_ in a way that I could make sense of programmatically.

A Kalman filter will give you the "best guess" for some state (x) at the current timestep (k). This estimate often has some lag in it, likely because you have some incomplete information that you couldn't model. Sometimes we care about the previous states (e.g., x_k-1). But if we just save these states and refer to them, we're not getting the most out of our data.

The Kalman Smoother can be used to go back and update these past values with all the samples up to the current time. To update your previous measurements, you need to save the state of your filter at every timestep (x_k) and its associated covariance matrix (P_k). You can then apply Kalman Smoothing to reprocess previous data and update it with all current information. This will often remove the phase delay that you would otherwise observe in your estimate.

Max Welling's Kalman Filter tutorial [1] derives the smoother equations using pretty clear and easy to follow notation (and is a great resource generally).

Briefly: you first run the filter equations "forwards", processing each datapoint sequentially from start to end. Then you run the smoother "backwards" in time on the same data going from end to start.

1. http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/...

Thank you, I'll have a look at it when I can find the time and frame of mind.

Reply on news.ycombinator.com