I read all the way into the next post, "What the hell is representation?". I don't really see why the question is all that confusing (first Red Alert for ignorance and bullshit goes here). My first halfway-informed guess is: a representation is a three-tuple of a compression function, a resulting data-structure, and a decompression function that, taken all together, capture at least some of the causal/computational structure of the process generating the data.

Of course, this extreme generality is what makes representation learning Very Hard, but also very powerful when you can manage to get it working at all.

Compression function: because we choose to assume (without ever being able to prove, see Chaitin) that our data is not algorithmically random. Conveniently, most real-world data isn't algorithmically random, so learning about the underlying process must result in a description that can compress the data more efficiently than just writing down the sample-set.

Data structure: the actual representation, which (ideally) explains (most of) the data on its own, plus or minus some kind of noise.

Decompression function: learning would be useless if we couldn't perform any kind of inference or prediction. We need a way to take the internal representation, plug in some parameter values (either ones we've learned from data or deliberately counterfactual ones), and then make a prediction about further samples which will constitute the actual action taken to perform a useful task.

So it's a question, I think, of whether we're doing information-theoretic learning, or algorithmic information-theoretic learning. Mind, this could all be bullshit, as I'm a total novice at this stuff.