Compressing Hegel (2020)
wittgenfine.substack.com
wittgenfine.substack.com
The article has all of the foundations to make a much more reasoned and perhaps more interesting claim: intelligence is equivalent to the size of one's abstraction inventory, and abstraction utility is measured by prediction success in abstraction-space.
The fact is that humans' short term memory capacity[1], necessitates the hoisting of raw sense data into an abstract space, and doing so implies a very real lossy process. We don't really make much of a fuss when ontologizing the world, but it's a bit strange doing that specifically when referencing Hegel.
The author is as guilty as anyone who, during the process of abstraction, forgets they have done so and thinks that the concept of a cat implies that cats are some universal essence made manifest. Forms are constantly trying to convince us that they're real.
1. https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus...
Quite a lovely post, thank you, well constructed.
One other theme that shows up (briefly) is that storing the data isn't good enough. Being able to access your compressed data, being able to apply yiur compressed models to new applications is, in my view, another core aspect of intelligence, one that requires & uses real ingenuity.
Being a highly associative person is, IMO, a colossal form of intelligence & intellect.
The later Marxists had a term for this, reification, and I'm convinced it is probably the most common fallacy human beings make.
But yes, that's precisely it.
For instance, what about the subconscious? The subconscious must play a strong role in problem solving and creativity, which is how I notice intelligence, anybody I think who has written a piece of music or struggled with a proof can attest to that. But are there abstractions at play when the unconscious is in the driver seat? And if the unconscious speaks in abstractions, are they the same "form" abstractions the conscious mind is so familiar with? Consider some different form of abstraction, informed by and tooled by information and methods you are not privy to.
Moreover, if humans did have unlimited short term memory size, suppose we had to find what numbers are divisible by a prime, how many of us would be checking numbers individually in our head and not using Fermat's Little Theorem? Suppose such memory allows us to discover new methods of compaction, wait then, we are playing the same game again and perhaps that would lead us to the presumption that we do it because our working memory is not "good" enough. Then "intelligence," which we outside of ourselves can only perceive in the ways it is demonstrable, and what motivates it, seems less then about the journey to compact information into coherence, producing objects for an inventory and thereby producing some super-order of workshopping, and more about heuristic success in pragmatic material terms. These are terms I'd argue out of the scope of "conscious" abstraction, hence the rejection of such ideas/thoughts on account of their supposed duplicity. But I don't know if the same can be said for the unconscious.
It seems to me then that so called compression, conscious abstraction, is more than an evolutionarily-devised-resource-scarcity-solution, it's baked in, *ontologically*. We are, in part, motivated to look for solutions not necessarily on account of our limitations but because we can imagine solutions. I do anticipate your eye roll.
I would posit then, part of what is at play for what I'll term demonstrable intelligence, is the degree to which the conscious mind has strength in itself as an abstraction workstation and how much access the "unconscious" mind seems to grant it to other spheres of tooling. In this sense, not only is the success of abstractions and their varieties relevant, but those very attributes of the abstractions are informed by the diversity and complexity of a little something else.
So sure, the map is not the territory and the cat is in my mind and not necessarily a reflection of the material reality, but I think we exaggerate the problems in abstractions/forms because we are unsatisfied with the whole enterprise of consciously accessing the territory in the first place.
I just take what I've read from psychology, sociology, statistics, and AI/ML and synthesize them into a critical lens. Maybe it's right in some ways, wrong in others.
It's a framework for stepping back and challenging our assumptions and giving space for us to ask, "Where did that assumption come from? And, why should I take that for granted?" and that's usually a good thing.
The perspective presented here is soaked through and through with information theoretic bias and clearly stems from an overly digital consideration of human experience. "Intelligence" is intimately related to context, use, community, and goals. We don't call someone intelligent because we can point to some lossless compression algorithm implemented by his neurons, we call him intelligent when he produces the behaviors we desire in a given situation.
Consider for instance, how this theory fails to account for athletic knowledge. It seems fair to state that people with tangible skills requiring the use of the body posses some kind of intelligence, it seems less accurate to try and describe this form of intelligence as lossless compression. The compression metaphor really only works for intelligence with respect to the manipulation of symbolic representations and even then I think there are plenty of counter examples of what we'd call intelligent behavior that would suggest not all cases are reducible to "lossless compression". It is all highly dependent on the questions you ask, as Wittgenstein well knew.
Personally, I do not feel this is a good piece of philosophy and it's symptomatic of a trend toward "computormorphization" which has been rampant since the computer emerged (it's like anthropomorphism except interpreting the behavior of other (actually living things!) as though they behaved just like computers). You can also see this in the reification of concepts like information (we talk about information as though it were some objective material object, but information is not a substance--only interpreters produce information; a text is a vehicle for information but it does not "contain information"--the information is produced by the reader of the text and fully depends on what distinctions they deem relevant)
"The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages."
In other words, information has nothing to do with isomorphism to a structured representation with definite relationships with the world, nothing to do with symbolic representations or any other kind of "representations". It's simply the ability to predict one thing from another thing. Shannon calls these things "messages" but mathematically the logic applies to any set: for example we can analyze an agent's policy information-theoretically, in terms of how much information about the agent's input state is contained in its output actions. I think this kind of view is highly compatible with later Wittgenstein, for whom "meaning" (if it's anything) is an abstraction of how language is used to coordinate joint action---that is, situations where people are trying to predict and control each others' behavior.
To address your examples,
> We don't call someone intelligent because we can point to some lossless compression algorithm implemented by his neurons, we call him intelligent when he produces the behaviors we desire in a given situation.
Suppose we give someone input A and he outputs desired behavior f(A), and then we give him input B and he outputs desired behavior f(B). If we give him input C will he output the desired f(C)? Only if he has learned to predict what f(C) should be as a function of C and the training data, that is, if he has learned to predict what we consider the "desired" behavior to be. In that sense, intelligence here requires prediction, which is exactly the same thing as compression.
> It seems fair to state that people with tangible skills requiring the use of the body posses some kind of intelligence, it seems less accurate to try and describe this form of intelligence as lossless compression.
Again, the agent's problem here is to select the right motor action given sensory input. Suppose you've seen a tennis ball coming in with velocity vector x and you know how to hit it to win the game, and you've seen a tennis ball coming in with vector y and you know how to hit it to win the game. What will you do with vector z? If your policy is good, it will put high probability on only actions that hit the ball in a way that makes you win. And if your policy is putting high probability on that action, that's mathematically the same as saying it provides an efficient compression of that action (high probability = small number of bits in an encoding).
This is very abstract but once you adopt this view it gives you a lot of very useful conceptual tools. For example, you can talk about the "channel capacity" of a policy in terms of how many bits of information it can "transmit" from input states to output actions, and this channel capacity turns out to be a very intuitive measure of the complexity of the policy that you can use to analyze human behavior (one example, [1]).
I disagree re: compatibility with the Investigations. In my reading the investigations are pretty much a refutation of the idea that to know is to have, like you said, some representation or picture of the world in terms of logical relations. In modern terms it is the same as stating that to know is equivalent to having particular neuron firings. I think the major point of the Investigations was to illustrate that our notion of knowledge is not some objective, universal thing, but rather it's highly contextual. In this fashion two human beings with totally different backgrounds and neuronal firings could both produce an "intelligent" answer to a question; it all hinges on context. To boil intelligence down to compression algorithms is to reject this contextually and put us back into the position of trying to establish objective measures for highly relative concepts.
Even Shannon's concept of information is relative; I guess I'd be ok with a statement like "intelligence is the selection of an acceptable compression algorithm in a context" but on the whole it just feels...super reductionistic and probably not what a rigorous epistemology would ultimately conclude.
Really, the proposition that ought to be made here is probably something more like: "in information theoretic terms, intelligence may be represented as compression"; this allows the theory to treat of the idea without reifying the theory.
Let me try this. Ugh uses stupid words because he is not as clever.
If an athlete catches a ball, does he remember, somewhere, somehow, every possible trajectory of every possible permutation of “ball” and every possible response to it?
Or does he learn patterns of movement, generalized reactions to “ball-like things” flying “in my general direction”?
It’s of course hard to describe non-verbal abstraction which is why we usually resort to easier subjects. I don’t however see your fundamental point: how else besides abstracting, by “compressing”, does an “athlete” learn?
I suspect a lot of people get sidetracked by modality. You see it with LLMs. It’s only text! Yeah, well, it’s tokens and everything can be tokens. See the latest transformer based audio generator Bark. I don’t see why movement is different. It’s about the structure in the data, not the shape, context, use or “community” of it.
Edit:
> We don't call someone intelligent because we can point to some lossless compression algorithm implemented by his neurons, we call him intelligent when he produces the behaviors we desire in a given situation.
“We don’t call a car red because of particular wavelengths of light activating sensors in our retinas, we call it red because it looks that way”. I’d be careful about attaching importance to how we use words in daily life. There are so, so many shortcuts we take.
Humans are not the only animals capable of learning. Do you think it's fair to say a crow learns how to crack a nut via compression and abstraction?
I think it's a combination of community and pattern (structure). Sure, once the LLM is trained, it can pattern match. Once the crow watches another crow, it can crack the nut, but the crow that does not exist in the community of nut cracking knowledge-sharers will not learn to recognize the pattern merely off its structure alone. In other words, compression (or, induction) is a part of the story, sure, but it is not the only factor.
Well, yes. Do you know another way?
Edit: oh I forgot your second part (I’m Ugh).
You say a crow alone does not learn this. How do you propose the first crow learned?
At some point a single crow learns. I know it propagates fast through a group, but that’s propagation. A different problem.
I’m used to the word abstraction in a different context and it’s not linguistic to me at all. I’m not the greatest communicator so that doesn’t help either.
The definition “disassociated from any specific instance” would be my take in this context.
“Learning” would mean something like generating a sufficiently abstracted model that is useful over a wide variety of inputs.
Aka you don’t store every instance of every type of nut, you store “nutness”. This is where the compression metaphor is quite obvious. You reduce the astronomically complex world of nuts down to “nutness”, in a way that still makes sense to you.
Not dissimilar, I think, to how MP3 or JPEG drops significant amounts of information but still remains useful to us. Only thing I would argue is that it’s lossy not lossless. Lossless would imply the world can be reduced without dropping information and I don’t think it can to any significant degree.
Anyway not trying to persuade you or anything. Your input does make me think and that’s not a bad thing, right? Thanks for indulging me.
https://wittgenfine.substack.com/p/compressing-hegel/comment...
If I can accurately predict the next X bits I don't need to store those.
If I can accurately predict the next X bits of information that will happen after a decision I can make a reasoned decision.
That's it. That's the link as plainly as possible. If you want more detail read up on arithmetic coding and how it can take predictions as in input to data compression and then Marcus Hutters papers on general purpose AI requiring the exact same thing.
On a related epistemological loop - I was just using the openai tool to query some of Kripke's modal logic ideas, and find whether they could be expressed in code (they can, for a given theorem in modal logic, gpt can represent the theorems in lisp if you want), and what struck me about the whole premise was that he invented not so much a broader logic that was revelatory or quantitatively truth finding - but a neutralizing, critical logic designed to dilute others. Gödel's theorems are apparently for logics that can produce arithmetic, which seems basic an necessary, but Kripke figured, "fine, what about logics that don't produce arithmetic?" (the result is basically digraphs.) Apparently there's nothing Kripke produced that can't be produced symbolically with digraphs, and like the author's compression example, it's just digraphs all the way down. Same flaw.
Kripke's ideas seem like fun philosophical ideas, but they're more of a scheme backfitted to a familiar ideology, and then you see his ideas come up often again as foundations for other less rigorous and less quantitative theories. If you've ever heard people use the word "modality," they're attempting to control a discussion using some of these same techniqes in what they percieve as a power struggle without any need for alignment to truth or reality. By adding parallel logical systems without the same criteria for consistency, Kripke created a tool for infinite uncertainty whose main feature is to neutralize the concept of logical truth in its subjects. I think it will be immensely useful in creating AI's, but it will also be the basis for the truly dangerous parts of it. Modal logic does not relate to the universe or need external consistency, it only exists as a kind of solvent for another existing logic it effectively criticizes. If there were such a thing, I think it may be a recipe for actual evil.
Anyway, good luck with the "everything is compression" thing, and even the basilisk that is Kripke's digraphs dressed up as ontology. I don't think there is a there there, but if we get to talk about this stuff, there may still be some fun to be had.
Tldr: tldr.
> I argue that data becomes temporarily interesting by itself to some self-improving, but computationally limited, subjective observer once he learns to predict or compress the data in a better way, thus making it subjectively simpler and more beautiful. Curiosity is the desire to create or discover more non-random, non-arbitrary, regular data that is novel and surprising not in the traditional sense of Boltzmann and Shannon but in the sense that it allows for compression progress because its regularity was not yet known. This drive maximizes interestingness, the first derivative of subjective beauty or compressibility, that is, the steepness of the learning curve. It motivates exploring infants, pure mathematicians, composers, artists, dancers, comedians, yourself, and (since 1990) artificial systems.