Hacker News Reader

Comment by jhgb | Hacker News Reader

> For this case that might mean convert the data into deltas, then write those deltas in a byte-oriented variable length integer fashion.

Calculating a trend line and storing deltas off of that trend line might be mildly better for some series. Having an AR model and storing errors of the model might work even better, although the obvious (and common) problem is which AR model of the many possible ones you go for.

View on HN

If you know the specifics of your data, maybe. But try to err on the side of simpler models, every detail you add slows your code down, and the difference after compression is often small.

Or simply a delta off the last delta (or mean of last n deltas, I guess), so you can do the compression in a stream.

If I understand you & GP, these are both just particular instantiations of Linear Predictive Coding [1]. For both of these approaches, you’d probably get similar or better results just throwing your data into an off-the-shelf LPC, and with less effort.

[1] https://en.m.wikipedia.org/wiki/Linear_predictive_coding

That might also be equivalent to one of the ARMA models, although I'd have to do the math to prove that (and to derive which one corresponds to what you're describing).

Edit: It seems that if you detrend the data, the "delta off the last delta", if I understand it correctly, is an MA(1) model. I think. Maybe. My math is wobbly.

Reply on news.ycombinator.com