Ask HN: Is Everything Data Compression?
Like chaotic and nonchaotic systems alike can be described by their ability to be compressed. The less compressible the less chaotic [1]. And this is where we define explicitly chaos is in the regime of positive Lyapunov exponents. And so that creates an intrinsic link between dynamics and information. And information is what the universe is fundamentally made out of to say metaphorically.
But then that is why neural networks are so good at their jobs compared to anything else when predicting chaotic things [2] because they literally are attempting to compress the space of data. Which is a descriptor of chaos.
And the reason we say that neural networks are unexplainable or at least hard to explain is that they don’t show directly the solution. But if we switch our mental model to that the universe is simply only able to be described by literally compressing it into something, then neural networks are the best estimator for any function [3].
Which implies that neural networks are the best we can get. So all you can do is work towards more sophisticated architectures to understand anything. Because there isn’t anything that’s a true data model, function approximations are all we will ever get [4]. And also the reason they can predict further into the future than though possible is because previously we simply had worse compression algorithms than before.
And to me this all seems trivially true, or likely less than that because there seems to be no implications from it that are more than cursory observations and perhaps I am simply naive. So I am hoping someone can point me in the right direction to read or do more to understand these things. Because I have organized a conference to understand these ideas [5] and I think I should organize another one but also I’m not sure how to interact with these ideas because most of my time is spent doing geophysics and data science and not this.
[1] https://www.sciencedirect.com/science/article/abs/pii/S0370157301000254
[2] https://link.aps.org/doi/10.1103/PhysRevResearch.5.043252
[3] https://www.sciencedirect.com/science/article/abs/pii/S089360809700097X
[4] https://projecteuclid.org/journals/statistical-science/volume-16/issue-3/Statistical-Modeling--The-Two-Cultures-with-comments-and-a/10.1214/ss/1009213726.full
[5] conference talks on YouTube (not all the talks were recorded and also the conference was broader in topic than this post). https://youtube.com/playlist?list=PL6zSfYNSRHalAsgIjHHsttpYfxJ_XIPbt&si=VVWAE-fsv_WfFwfK