We're not Platonists, we've just learned the bitter lesson

The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.

So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes.

Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.

Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight.

But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it.

I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.

I asked Minsky about this a couple decades ago and it was the only time he didn't tell me I was asking a stupid question. Apparently, after the conference, the name stuck because they found it easier to get research grants with the more "neutral" sounding name.

I wonder how many people would trust "AI" if it was called "distributed indexing" or "stochastic generation."

Namely, there's a huge difference between the concept of intelligence and an intelligent agent.

You could make the argument that current LLMs have some level of intelligence. But (without a lot of supporting infrastructure) they are not agents, they have no built in mechanism for having any kind of intent or even short-term memory. Because they fundamentally just generate a single token at once, they don't even exist "within time" the way we normally would think of an intelligent entity as doing... they're more like a mathematical function that you can invoke on demand. Perhaps to get an "intelligent" result, but conflating that kind of intelligence with an intelligent entity or actor seems like a mistake.

To use the analogy in the OP, it would be like calling a single hydraulic piston "strong". Sure, but unless you attach it to something it's not the same kind of strong as a bodybuilder is.

It's not. AIs already control the world. They have proven indifferent to the welfare of humans. Perhaps the most important developments in recent decades were their coming to own the majority of public media, and the high court of the one of biggest jurisdictions permitting them to bribe politicians secretly: in practice, to own pet legislators.

The AIs in question are called corporations. Their working parts mostly still include (readily replaced) humans, but decreasingly so. They have learned to manufacture voting blocs to drive political movements in nominally democratic countries to support their goals. By analogy to the paperclip apocalypse, they have a built-in motivation to turn all matter and energy into profit. Where their chosen method results in some service or product useful to humans, humans may benefit, but such choices may change freely, or may fail to change when the method turns out to cause net harm. Attempts at legislation to limit their power fail as they have taken control of the legislative process.

Is it too late to do anything? Probably. I for one ...

It seems a small conjecture to note that the problem here is that "deep principles of linguistics" the linguists are attempting to hard-code, are in fact more like "intermediate principles of linguistics".

The linguists haven't approached anywhere near the deeper layers of how the brain works on language (nevermind how the Wernicke's area coordinates w/the visual cortex to generate "I see a bird over there"). So, when they code in their principles, they only screw up the deeper relationships. Of course the LLM has zero actual understanding of any abstract meaning such as "I", "see", "bird", or "over there", but it's still better off without those hardcodings to generate the phrase in a valid or hallucinated context.

It is however a wonderful world building detail in Anathem by Neal Stephenson. Their world history revolves around two ideologies: the syntactics and the semantics. Nominalists and Formalists. I find his predictions about a society built around this duality entertaining.

In this case, I'm thinking of natural kind terms (https://plato.stanford.edu/entries/natural-kinds/) and, less well known, Boyd's homeostatic property cluster theory. It is kind of amazing to me that this article has 310 comments and not a single person has raised the question natural kinds.

OTOH I am heartened that people are effectively doing philosophy in the wild without getting too concerned with "whether other people have already said this before" which can be paralyzing.

Intelligence is achieving goals.

Intelligence is the process of discovering which goals are worth achieving.

- - - -

Q: What's GAI?

A: When the computer asks, "What's in it for me?"

It's dubious to refer to these individual events as "explosions". They're just single significant changes. Any event in humans or AI you could respectably call an explosion would be a cascade (note: causally interconnected) of a large number of these individual intelligence-increasing changes.

If anything, I'd argue that the buildup of communal knowledge in human civilizations over the last several thousand years, especially the last few centuries, has been an explosion of sorts. It's been a huge increase in capability for one species, and would look like a step change on a timescale that includes all Earth life. But if that's an explosion, writing was just one of the chemical reactions.

That’s to say, when we talk about Tyson being stronger than my grandmother, we ignore correlated subclaims such as “Tyson consumes more protein than my grandmother” or “Tyson wears a jockstrap more often than my grandmother”—these are correlated with being stronger (probably), but they are incidental to the concept of strength. When we ask ourselves what subclaims are relevant, we’re right back to pointing at essence.

Is "concept" a bundle of useful correlations?

Is "bundle" a bundle of useful correlations?

Is "useful" a bundle of useful correlations?

Is "correlation" a bundle of useful correlations?

And correlations of what? Very strong emphasis on what. Quid est?

From the get go, like all skepticism, this approach begins by sitting on a branch that more or less acknowledges reality and its comprehensibility, but quickly begins sawing off the very branch it is sitting on. Skepticism is always a form of selective denial, and as such, it is incoherent. It's like those people who argue passionately that there is no truth while in the lecture hall, but as soon as the clock strikes 4pm, they're worried about missing the bus and getting everybody in the room to the next lecture, or as Duhem would say, going home to kraut and pipe.

"Triangularity" is not a "bundle of correlations". Triangularity is an irreducible, but analyzable whole, a form that is instantiated in the world within particulars and that we encounter in particulars and abstract from those particulars to arrive at the concept. That's what a concept is: a form abstracted from concrete particulars that exists in the intellect as a universal.

Unfortunately, mechanistic metaphysics, taught and insinuated in our scientistic curricula from a young age, is a difficult intellectual habit to break for many. It is very difficult for people to uproot these obstinate presuppositions.

It's peak this guy, and peak "rationalists" in general, to defend the concept of "objective" intelligence, intelligence towards unspecified goals. This is what they actually care about: that there's an objective scale of intelligence, and that they're above most people on it. Everything else they believe is tangential. They choose other beliefs according to how they can strengthen the core belief, that they are the intelligent ones. (And you bet they downvote people challenging it).

What goals their intelligence are directed at, they're evasive and inconsistent about, and that others might have different goals is certainly not a permissible option. In THAT area, they are platonists, no matter how fuzzy their forms are: playing the flute is the task of a professional flautist, and steering the ship of humanity is a task for professional captain smart. Where the ship should be heading is not a subject of debate.

That’s a leap.

I find it quite puerile how so few can accept their one of billions-ness.

“If you can’t encode and decode information how I do, you stink, and so does your technology or philosophy or whatever it is!!”

Religion is not the metaphors and poetry. It’s a biological state of conviction in one’s own essential ness to reality.

"Variation in IQ during adulthood is about 70% genetic."

The proviso "during adulthood" is important: "The heritability of IQ increases with the child's age and reaches a plateau at 18–20 years old, continuing at that level well into adulthood." (Is it just me or is that a weird thing for it to do.) That being said, the Wilson Effect paper goes on to say, "It is important to specify the populations to which any results can be generalized and not misinterpret what they mean. The samples were drawn almost exclusively from Western industrial democracies. These settings have characteristic environments. Only a few of the participants were raised in real poverty or by illiterate parents, and all study participants had access to the contemporary educational programs typical of those societies. This is the domain to which we can generalize." In any case, for most of my life the variation in earring-wearing was almost 100% genetic, so there's that.

"A lot of this seems to have to do with literal brain size (which is correlated with intelligence at about 0.2)..."

0.2 is not a very strong correlation. And correlations are funny. (https://www.tylervigen.com/spurious-correlations)

"There have already been intelligence explosions. Long ago, humans got smart enough to invent reading and writing, which let them write down training data for each other and become even smarter (in a practical sense; this might or might not have raised their literal IQ, depending on how you think about the Flynn Effect)."

This is a direct contradiction to Alexander's apparent argument above, unless you consider reading and writing to have a significant genetic component. (Heritability? Oh, yeah, I'd expect reading and writing to have pretty high heritability, at least historically.)

But is knowing things really what you mean by intelligence?

"Later on, we invented iodine supplementation, which let us treat goiter and gain a few IQ points on a population-wide level."

Fortunately, iodine supplementation is also highly heritable.

"...were emphasizing how rather than magical intelligence, AI would need slow, boring work on the nitty gritty of computation..."

How are LLMs doing on that whole hallucination thing? The ability to solve novel problems?

P.s. A fun video: https://www.youtube.com/watch?v=UBc7qBS1Ujo

Once you get smart enough, you can do things that make you even smarter. AI will be one of those things. We already know that bigger blobs of compute with more training data can do more things in correlated ways - frogs are outclassed by cows, chimps, and humans; village idiots are outclassed by Einstein; GPT-2 is outclassed by GPT-4. At some point we might get a blob which is better than humans at designing chips, and then we can make even bigger blobs of compute, even faster than before.

But what about:

1. Limited to human performance by training data? OpenAI apparently aren't training GPT-5 because they think that research direction is tapped out. Their focus has been on augmenting this "superintelligence" with boring logic-based systems like calculators, Python interpreters, web browsers and 80s style expert systems like Wolfram Alpha. All this is suspiciously like what a human would need, not a superintelligence. It implies they don't think they can do another 2->3->4 style leap, probably due to lack of training data that would yield more advanced capabilities.

2. Bottlenecked by physical experimentation? Alexander casually asserts that if you trained an AI on circuit design it'd immediately do better, and that'd be used to build better AI chips, which in turn would yield a smarter AI ad infinitum. But Google already tried this and it just led to fraud claims, not better chips. And even if an LLM came up with an idea for a better chip, humans would still need to do the physical experimentation to figure out how to build them.

3. Bottlenecked by lack of imagination? LLMs can be "creative" in the artistic sense but there is a suspicious and very noticeable absence of them coming up with any genuinely interesting ideas. They can think faster than humans, and know immeasurably more, yet has anyone found even just one example of the sort of out-of-the-box thinking that would be required for AI to outsmart humans? Where are all these scientific breakthroughs the AI evangelists keep promising? GPT-4 is pretty damn smart but despite asking it many things, it never once came up with an idea I hadn't already had.

The article's examples seem to be its own undoing. There is no such thing as strength-leading-to-more-strength in some sort of recursive loop, which is why he needs such a bizarre and artificial example. Why should we believe there is for intelligence, when the history of human intelligence is a 2000 year struggle for even quite minor improvements in cognitive ability and even that is highly debatable?

Is Deep Blue more intelligent than Garry Kasparov?

His quality has been sliding the last couple years, which happens to coincide with his announced commitment to AI Doom Ideology. Maybe those are connected and maybe not, but there it is.

Once the analogical reasoning broke down I stopped reading, especially since I believe life is a prerequisite for intelligence and is bound to it, because intelligence does not equal computation and is more than Turing complete.