Life is more than an engineering problem
lareviewofbooks.org
lareviewofbooks.org
Then I come to
> [LLMs] can get better at reproducing patterns found online, but they don’t become capable of actual reasoning; it seems that the problem is fundamental to their architecture.
and wonder how an intelligent person can still think this, can be so absolute about it. What is "actual" reasoning here? If an AI proves a theorem is it only a simulated proof?
AI is (was?) a stochastic parrot. At some point AI will likely be more than that. The tipping point may not be obvious.
I can see the argument against chatGPT4 reasoning.
The reasoning models though I think get into some confusing language but I don't know what else you would call it.
If you say a car is not "running" the way a human runs, you are not incorrect even though a car can "outrun" any human obviously in terms of moving speed on the ground.
To say since a car can't run , it can't move though is obviously completely absurd.
"The terminator isn't really hunting you down, it's just imitating doing so..."
Using the word “hallucinate” is extremely misleading because it’s nothing like what people do when they hallucinate (thinking there are sensory inputs when there aren’t).
It’s much closer to confabulation, which is extremely rare and is usually a result of brain damage.
This is why a big chunk of people (including myself) think the current LLMs are fundamentally flawed. Something with a massive database to statistically confabulate correct stuff 95% of the time and not have a clue when it’s completely made up is not anything like intelligence.
Compressing all of the content of the internet into an LLM is useful and impressive. But these things aren’t going to start doing any meaningful science or even engineering on their own.
No we have not, neurodiverse people like me need accommodations not fixing.
An LLM does nothing more than predict the next token in a sequence. It is functionally auto-complete. It hallucinates because it has no concept of a fact. It has no "concept", period, it cannot reason. It is a statistical model. The "reasoning" you observe in models like o1 is a neat prompting trick that allows it to generate more context for itself.
I use LLMs on a daily basis. I use them at work and at home, and I feel that they have greatly enhanced my life. At the end of the day they are just another tool. The term "AI" is entirely marketing preying on those who can't be bothered to learn how the technology works.
But ChatGPT a couple years ago was at more like grade-school level at problem-solving. What should I call this thing that the best LLMs can do better than the older ones, if it's not actual reasoning? Sparkling syllogistics?
Sorry, that's sarcastic, but... it's from a real exasperation at what seems like a rearguard fight against an inconvenient conclusion. I don't like it either! I think the rate of progress at building machines we don't understand is dangerous. (Understanding the training is not understanding the machinery that comes out.)
Compare the first previews of Copilot with current frontier "reasoning" models, and ask how this will develop in the next five years. Maybe it'll fizzle. If you're very confident it will: I'd like to be convinced too.
As far as if it will fizzle, even if it does, what we have currently is already useful. Society will take time to adjust to ChatGPT-4's level of capabilities, nevermind whatever OpenAI et al releases next. It can't yet replace a software engineer, but it makes projects possible they previously weren't attempted because they required too much investment previously. So unless you're financially exposed to AI directly (which you might be, many people are!), the question of if it's going to fizzle is more academic than something that demands a rigorous answer. Proofs of a negative are really hard. Reusable rockets were "proven" to be impossible right up until it was empirically proven possible.
Are they? Which animals? Some seem smart and maybe do it. Needs strong justification.
> probably through being trained on many examples of the laws of nature doing their thing
Is that how they can reason? Why do you think so? Sounds like something that needs strong justification.
> then why couldn't a statistical model be?
Maybe because that is not how anything in the world attained the ability to reason.
A lot of animals can see. They did not have to train for this. They are born with eyes and a brain.
Humans are born with the ability to recognize pattern in what we see. We can tell objects apart without training.
if animals didn't show problem-solving skills, and thus reasoning, complex ones wouldn't exist anymore by now. Planning is a fundamental skill for survival in a resource-constrained environment and that's how intelligence evolved to begin with.
Assuming that intelligence and by extension reasoning are discrete steps is so backwards to me. They are quite obviously continuously connected all the way back to the first nervous systems.
I think what he is trying to say is that LLMs current architecture seems to mainly work by understanding patterns in the existing body of knowledge. In some senses finding patterns could be considered creative and entail reasoning. And that might be the degree to which LLMs could be said to be capable of reasoning or creativity.
But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of.
Its not though, nobody really knows what most of the words in that sentence mean in the technical or algorithmical sense, and hence you can't really say whether llms do or don't possess these skills.
Like in the subject of LLMs everyone knows what a "token" or "context" means, even if they might mean different things in a different subject. Yet, nobody knows what "consciousness" means in almost any context, so it is impossible to make falsifiable statements about consciousness and LLMs.
Making falsifiable statements is the only way to have an argument, otherwise its just feelings and hunches with window dressing.
And nobody really knows what consciousness is, but we all experience it in a distinct, internal way that lets us navigate the world and express ourselves to others, yet apparently some comments seem to dismiss this elephant of sensation in the room by pretending it's no different than some cut and dried computational system that's programmed to answer certain things in certain ways and thus "is probably no different from a person trained to speak". We're obviously, evidentially more than that.
This is not really what is going on, what is going on is a mix-up in interpreting the meaning of words, because the meaning of words is not transitive between subject matter unless we arrive at a scientific definition which is leading, and we have not (yet).
When approaching the word consciousness from a spiritual POV, it is clear that LLMs may not possess it. When approaching consciousness from a technical point of view, it is clear that LLMs may possess it in the future. This is because the spiritual POV is anthropologically reductive (consciousness is human), and the technical POV is technically reductive (consciousness is when we can't tell it apart).
Neither statements help us clarify opposing positions because neither definitions are falsifiable, and so not scientific.
No, but you described a `cp` command, not an LLM.
"Creativity" in the sense of coming up with something new is trivial to implement in computers, and has long been solved. Take some pattern - of words, of data, of thought. Perturb it randomly. Done. That's creativity.
The part that makes "creativity" in the sense we normally understand it hard, isn't the search for new ideas - it's evaluation of those ideas. For an idea to be considered creative, it has to match a very complex... wait for it... pattern.
That pattern - what we call "creative" - has no strict definition. The idea has to be close enough to something we know, so we can frame it, yet different enough from it as to not be obvious, but still not too different, so we can still comprehend it. It has to make sense in relevant context - e.g. a creative mathematical proof has to still be correct (or a creative approach to proving a theorem has to plausibly look like it could possibly work); creative writing still has to be readable, etc.
The core of creativity is this unspecified pattern that things we consider "creative" match. And it so happens that things matching this pattern are a match for pattern "what makes sense for a human to read" in situations where a creative solution is called for. And the latter pattern - "response has to be sensible to a human" - is exactly what the LLM goal function is.
Thus follows that real creativity is part of what LLMs are being optimized for :).
If we could predefine what would count as creativity as some specific pattern, then I'm not sure that would be what I would call creative, and certainly wouldn't be an all-inclusive definition of creativity. Nor is creativity merely creating something new by perturbing data randomly as you mentioned above.
While LLMs might be capable of some forms of creativity depending on how you define it, I think it remains to be seen how LLMs' current architecture could on its own accomplish the kinds of creativity implicit in scientific progress in the Kuhnian sense of a paradigm shift or in what some describe as a leap of artistic inspiration. Both of these examples highlight the degree to which creativity could be considered both progress in an objective sense but also be something that is not entirely foreshadowed by its precursors or patterns of existing data.
I think there are many senses in which LLMs are not demonstrating creativity in a way that humans can. I'm not sure how an LLM itself could create something new and valuable if it requires predefining an existing pattern which seems to presuppose that we already have the creation in a sense.
This kind of follows from the fact that, even if the paradigm-shifting insight was caused by some miracle feat of a unique superhuman genius, it still wouldn't shift anything until everyone else in the field was able to verify the genius was right, that they found the right answer, as oppose to a billion different possible wrong answers. To do that, the entire field had to have accumulated enough empirical evidence and theoretical understanding to already be within one or two "regular smart scholar" leaps from that insight.
With art, I have less experience, but my gut instinct tells me that even there, "artistic inspiration" can be too big a leap from what was before, as otherwise other people would not recognize or appreciate it. Also, unlike science, the definition of "art" is self-referential: art is what people recognize as art.
Still, I think you make a good point here, and convinced me that potential for creativity of LLMs, in their current architecture, is limited and below that of humans. You said:
> While LLMs might be capable of some forms of creativity depending on how you define it, I think it remains to be seen how LLMs' current architecture could on its own accomplish the kinds of creativity implicit in scientific progress in the Kuhnian sense of a paradigm shift or in what some describe as a leap of artistic inspiration.
I think the limit stems strictly from LLMs being trained off-line. I believe LLMs could go as far as making the paradigm-shifting "Kuhnian leap", but they wouldn't be able to increment on it further. Compared to humans, LLMs are all "system 1" and almost none "system 2" - they rely on "intuition"[0], which heavily biases them towards things they've learned before. In a wake of a paradigm shift, a human can make themselves gradually unlearn their own intuitions. LLM's can't, without being retrained. Because of that, the forms of creativity that involve making a paradigm-shifting leap and making a few steps forward from it are not within reach of any current model.
--
[0] - LLMs basically output things that seem most likely given what came before; I think this is the same phenomenon as when humans think and say what "feels like best" in context. However, we can pause and override this; LLMs can't, because they're just run in a forward pass - they neither have an internal loop, nor are they trained for the ability to control an external one.
Formal Proof Systems aren't even nearly close to completion, and for patterns we don't have a strong enough formal system to fully represent the problem space.
If we take the P=NP problem, that likely can be solved formally that a machine could do, but what is the "pattern" here that we are traversing here? There is a definitely a deeper superstructure behind these problems, but we can only glean the tips, and I don't think the LLMs with statistical techniques can glean further in either. Natural Language is not sufficient.
LLMs aren't leaning what "creativity" is from first principles. They're learning it indirectly, by being trained to reply like a person would, literally, in the fully general meaning of that phrase. The better they get at that in general, the better they get at the (strict) subtask of "judging whether a work is creative the same way a human would" - and also "producing creative output like a human would".
Will that be enough to fully nail down what creativity is formally? Maybe, maybe not. On the one hand, LLMs don't "know" any more than we do, because whatever the pattern they learn, it's as implicit in their weights as it is for us. On the other hand, we can observe the models as they learn and infer, and poke at their weights, and do all kinds of other things that we can't do to ourselves, in order to find and understand how the "deeper superstructure behind these problems" gets translated into abstract structures within the model. This stands a chance to teach us a lot about both "these problems" and ourselves.
EDIT:
One could say there's no a priori reason why those ML models should have any structural similarity to how human brains work. But I'd say there is a reason - we're training them on inputs highly correlated with our own thoughts, and continuously optimizing them not just to mimic people, but to be bug for bug compatible with them. In the limit, the result of this pressure has to be equivalent to our own minds, even if not structurally equivalent. Of course the open question is, how far can we continue this process :).
That is why I mentioned Kuhn and paradigm shifts. The architecture of LLMs do not seem capable of making lateral moves or sublations that are by definition not derivative or reducible to its prior circumstance, yet humans do, even though the exact way we do so is pretty mysterious and wrapped up in the difficulties in understanding consciousness.
To claim LLMs can or will equal human creativity seems to imply we can clearly define not only what creativity is, but also consciousness and also how to make a machine that can somehow do both. Humans can be creative prima facie, but to think we can also make a computer do the same thing probably means you have an inadequate definition of creativity.
As for:
> As far as I can tell, I think you are interchanging the ability to recognize creativity with the ability to be creative.
I kind of am, because I believe that the two are intertwined. I.e. "creativity" isn't merely an ability to make large conceptual leaps, or "lateral moves" - it's the ability to make a subset of those moves that will be recognized by others as creative, as opposed to recognized as wrong, or recognized as insane, or recognized as incomprehensible.
This might apply more to art than science, since the former is a moving target - art is ultimately about matching subjective perceptions of people, where science is about matching objective reality. A "too creative" leap in science can still be recognized as "creative" later if it's actually correct. With art, whether "too creative" will be eventually accepted or forever considered absurd, is unpredictable. Which is to say, maybe we should not treat these two types of "creativity" as the same thing in the first place.
This seems a miopic view of creativity. I think leaving out the pursuit of the implications of that perturbation is leaving out the majority of creativity. A random number generator is not creative without some way to explore the impact of the random number. This is something that LLM inference models just don't do. Feeding previous output into the context of a next "reasoning" step still depends on a static model at the core.
Then cross referencing that new random point/idea to see if it remains internally consistent with the known true patterns in your dataset.
This is how humans create new ideas often?
If you, after copying the book, could dynamically answer questions about the theory, it's implications, and answer variations of problems or theoretical challenges in ways that reflect mainstream knowledge, I think that absolutely would indicate understanding of it. I think you are basically making Searle's chinese room argument.
>But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of.
Why is that clear? I think the reasoning for that would be tying it to a notion "the human experience", which I don't think is a necessary condition for intelligence. I think nothing about finding patterns is "mere" insofar as it relates to demonstration of intelligence.
>But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of
This is not clear at all. As it seems to me, it's impossible to imagine or think of things that are not in someway tied to something you've already come to sense or know. And if you think I am wrong, I implore you to provide a notion that doesn’t agree. I can only imagine something utterly unintelligible, and in order to make it intelligible, would require "pattern matching" (ie tying) it to something that is already intelligible. I mean how else do we come to understand a newly-found dead/unknown language, or teach our children? What human thought operates completely outside existing knowledge, if not given empirically?
I wonder how people write things like this and don't realize they sound as sanctimonious as exactly whatever they are criticizing. Or, if I was to put it in your words: "how could someone intelligent post like this?"
The thing is, you can interact with this new kind of actor as much as you need to to judge this -- make up new problems, ask your own questions. "LLMs can't think" has needed ever-escalating standards for "real" thinking over the last few years.
Gary Marcus made a real-money bet about this.
He's overly sentimental, and so are his books. I wish there were other sci-fi authors that the AI community wanted to contact but after "Arrival" I get it since "Arrival" is the literal wet-dream of many NLP/AI researchers.
Large language models excel at processing and generating text, but they fundamentally operate on existing knowledge. Their creativity appears limited to recombining known information in novel ways, rather than generating truly original insights.
True reasoning capability would involve the ability to analyze complex situations and generate entirely new solutions, independent of existing patterns or combinations. This kind of deep reasoning ability seems to be beyond the scope of current language models, as it would require a fundamentally different approach—what we might call a reasoning model. Currently, it's unclear to me whether such models exist or if they could be effectively integrated with large language models.
There are some theories that this is true for humans also.
There are no human created images that weren't observed first in nature in some way.
For example, Devils/Demons/Angels were described in terms of human body parts, or 'goats' with horns. Once we got microscopes and started drawing insects then art got a lot weirder, but not before images were observed from reality. Then humans could re-combine them.
Humans can suddenly "jump" cognitive levels to see higher-order patterns. Gödel seeing that mathematics could describe mathematics itself. This isn't combining existing patterns, but seeing entirely new levels of abstraction.
The human brain excels at taking complex systems and creating simpler mental models. Newton seeing planetary motion and falling apples as the same phenomenon. This compression isn't recombination - it's finding the hidden simplicity.
Recombination adds elements together. Insight often removes elements to reveal core principles. This requires understanding and reasoning.
You mean like alphago did in its 36th move?
The question is, is it possible to make reasoning models generic and can they be combined with large language models effectively.
Chiang has it exactly right with his doubts, and the notion that pattern recognition is little different from the deeply complex navigation of reality we living things do is the badly misguided notion.
How do you know this?
That was pretty much my question. Why are people so certain on the topic.
If you prick an LLM does it not bleed? If you tickle it does it not laugh? If you poison one does it not die? If you wrong an LLM shall it not revenge?
No one in neuroscience, psychology or any related field can point to reasoning or 'consciousness' or whatever you wish to call it and say it appeared from X. Yet we have this West Coast IT cultish thinking that if we throw money at it we'll just spontaneously get there. The idea that we're even 1% close should be ridiculous to anyone rationally looking at what we're currently doing.
We know these are algorithms, but how many people fall in love or make friends over nothing but a letter or text message?
Capabilities for reasoning aside, we should all be very careful of our perceptions of intelligence based solely on a machines or algorithms apparent ability to communicate.
Thank God no one at the AI labs is working to remove that limitation!
glenstein made a good point that I was commingling concepts of intelligence and consciousness. I think his commentary is really insightful here: https://news.ycombinator.com/item?id=42912765
I don't think that's merely an irrational compulsion. Communication can immediately demonstrate intelligence, and I think it quite clearly has, in numerous ways. The benchmarks out there cover a reasonable range of measurements that aren't subjective, and there's clear yes-or-no answers to whether the communication is showing real ways to solve problems (e.g. change a tire, write lines of code, solving word problems, critiquing essays), where the output proves it in the first instance.
Where there's an open question is in whether you're commingling the notion of intelligence with consciousness, or identifying intelligence with AGI, or with "human like" uniqueness, or some other special ingredient. I think your warning is important and valid in many contexts (people tend to get carried away when discussing plant "intelligence", and earlier versions of "AI" like Eliza were not the real deal, and Sophia the robot "granted citizenship" was a joke).
But this is not a case, I think where it's a matter of intuitions leading us astray.
I’m absolutely commingling these two things and that is an excellent point.
Markov chains and other algorithms that can generate text can give the appearance of intelligence without any kind of understanding or consciousness.
I’m not personally certain of consciousness is even requisite for intelligence, given that as far as we know consciousness is an emergent property stemming from some level of problem solving ability.
Personally, I see ChatGPT say "water doesn't freeze at 27 degrees F" and think "how can it possibly do advanced reasoning when it can't do basic reasoning?"
(Basically, I don't think there's a bright line here: saying "they can't reason" isn't very useful, instead it's more useful to talk about what kinds of things they can reason about, and how reliably. Because it's kind of amazing that this is an emergent behaviour of training on text prediction, but on the other hand because prediction is the objective function of the training, it's a very fuzzy kind of reasoning and it's not obvious how to make it more rigourous or deeper in practice)
When you ask an LLM "what is 2 + 2?" and it says "2 + 2 = 4", it looks like it's recognizing two numbers and the addition operation, and performing a calculation. It's not. It's finding a common response in its training data and returning that. That's why you get hallucinations on any uncommon math question, like multiplying two random 5 digit numbers. It's not carrying out the logical operations, it's trying to extract the an answer by next token prediction. That's not reasoning.
When you ask "will water freeze at 27F?" and it replies "No, the freezing point of water is 32F", what's happening is that it's not recognizing the 27 and 32 are numbers, that a freezing point is an upper threshold, and that any temperature lower than that threshold will therefore also be freezing. It's looking up the next token and finding nothing about how 27F is below freezing.
Again, it's not reasoning. It's not exercising any logic. Its huge training data set and tuned proximity matching helps it find likely responses, and when it seems right, that's about the token relationship pre-existing in the training data set.
That it occasionally breaks the rules of chess just shows it has no concept of those rules, only that the next token for a chess move is most likely legal because most of its chess training data is of legal games, not illegal moves. I'm unsurprised to find that it can beat an average player if it doesn't break the rules: most chess information in the world is about better than average play.
If an LLM came up with a proof no one had seen, but it checks out, that doesn't prove it's reasoning either, just because it's next token prediction that came up with it. It found token relationships no one had noticed before, but that's inherent in the training data, and not a reflective intelligence doing logic.
When we discuss things like reinforcement learning and chain of reasoning, what we're really talking about are ways of restricting/strengthening those token relationships. It's back-tuning of the training data. Still not doing logic.
At this point in time, the act of comprehension is a scientific mystery.
I'd say 'consciousness' is the ongoing ever present comprehension of the moment, a feedback self conversation assessing the current situation a being finds itself. This act requires reasoning, as comprehension is the "sandbox" in which reasoning occurs.
But what is comprehension? It's the instantaneous reverse engineering of observations for verification of reality: is what I observe normal, possible or a threat? If one cannot "understand" an observation then the potential the observation is a threat grows. That 'understanding" is reverse engineering the observation to identify it's range of possible behavior and therefore one's safety in relation to that observation.
Comprehension is extremely complex: arbitrary input goes in and a world model with one's safety and next actions comes out.
This is not a good argument. Natural systems, the subject of neuroscience/psychology, are much harder to analyze than artificial systems. For example, it's really difficult to study atmospheric gases and figure out Boyle's/Charles law. But put a gas in a closed chamber and change pressure or temperature and these laws are trivially apparent.
LLMs are much more legible systems than animal brains, and they are amenable to experiment. So, it is much more likely that we will be able to identify what "reasoning" is by studying these systems than animal brains.
P.S. Don't think we are there yet, as much as internet commentators might assert.
> "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra
Discretization of continuous input spaces in the hippocampal autoencoder
Reasoning is undefined, but a human recognizes it when it appears. I don't see consciousness part of that story. Also, whether you call it emulated or played reasoning or not, apparently does not matter. The results are what they are.
We just need to figure out how to train that network.
Personally, my money is on quantum coherence within microtubules being the mechanism of conscious experience, with the brain essentially being a quantum/classical hybrid computer.
My take it is just unaware intelligence like in Peter Watts’ book Blindsight. A terrific read and a quite scary prospect.
Rather 1984 to look at the contribution of an academic and an iron welder and see authority in someone who memorized the book, but not how to keep themselves alive. Chiang and the like are nihilists, indifferent if they die cause it all just goes dark to them. Indifferent to the toll they extract from labor to fly their ass around speaking about glyphs in a textbook. Academics detached from the real work people need are just as draining on society and infuriating as a billionaire CEO and tribal shaman. Especially these days when they derive some small normalization from 100s of years of cataloged work and proclaim their bit of syntactic art is all they should need to spend the rest of their life being celebrated like they’re turning 8 all over again.
Grigori Perelman is the only intelligent person out there I respect. Copy-paste college grads all over the US recite the textbook and act like it’s a magical incantation that bends the will of others. Cult of social incompetence in the US.
Read up on the ELIZA effect
But I don't believe that. That a machine that can produce convincing human-language chains of thought says nothing about its "intelligence". Back when basic RNNs/LSTMs were at the forefront of ML research, no one had any delusions about this fact. And just because you can train a token prediction model on all of human knowledge (which the internet is not) doesn't mean the model understands anything.
It's surprising to me that the people most knowledgeable about the models often appear to be the biggest believers - perhaps they're self-interestedly pumping a valuation or are simply obsessed with the idea of building something straight from the science fiction stories they grew up with.
In the end though, the burden of proof is on the believers, not the deniers.
"Believer" really is the most appropriate label here. Altman or Musk lying and pretending they "AGI" right around the corner to pump their stocks is to be expected. The actual knowledgeable making completely irrational claims is simply incomprehensible beyond narcissism and obscurantism.
Interestingly, those who argue against the fiction that current models are reasoning, are using reason to make their points. A non-reasoning system generating plausible text is not at all a mystery can be explained, therefore, it's not sufficient for a system to generate plausible text to qualify as reasoning.
Those who are hyping the emergence of intelligence out of statistical models of written language on the other hand rely strictly on the basest empiricism, e.g. "I have an interaction with ChatGPT that proves it's intelligent" or "I put your argument into ChatGPT and here's what it said, isn't that interestingly insightful". But I don't see anyone coming out with any reasoning on how ability to reason could emerge out of a system predicting text.
There's also a tacit connection made between those language models being large and complex and their supposed intelligence. The human brain is large and complex, and it's the material basis of human intelligence, "therefore expensive large language models with internal behavior completely unexplainable to us, must be intelligent".
I don't think it will, but if the release of the deepseek models effectively shifts the main focus towards efficiency as opposed to "throwing more GPUs at it", that will also force the field to produce models with the current behavior using only the bare minimum, both in terms of architecture and resources. That would help against some aspects of the mysticism.
The biggest believers are not the best placed to drive the research forward. They are not looking at it critically and trying to understand it. They are using every generated sentence as a confirmation of their preconceptions. If the most knowledgeable are indeed the biggest believers, we are in for a long dark (mystic) AI winter.
Of course we don't know whether an LLM is doing something like this or actually reasoning. But this is also the point, we don't know.
If you ask a question to a person you can be confident to some degree that they didn't memorize the answer beforehand, so you can evaluate their ability to "reason" and come up with an answer for it. With an LLM however this is increadibly hard to do, because they could have memorized it.
An interesting hypothesis! I'm neither a mathematical logician, nor decently up to date in that field - is the possibility of this, at least in the abstract, currently accepted as fact?
(Yes, there's the perhaps-separate issue of only enumerating correct proofs.)
I'm mostly familiar with type theory, of which there are many variants, but the most common ones all share the most important characteristics. In particular they identify theorems with types, and proofs with terms, where correct proofs are well-typed terms. The nice thing is that terms are recursively enumerable, so you can list all proofs. Moreover most type theories have decidable type checking, so you can automatically check whether a terms if well-typed (and hence the corresponding proof is correct).
This is not just theory, there exist already a bunch of tools that are being used in practice for mechanically checking mathematical proofs, like Coq, Lean, Agda and more.
When I said "in theory" however it's because in practice enumerating all proof terms will be very very slow and will take forever to reach proofs for theorems that we might find interesting.
Since we're in the LLM topic, there are efforts to use LLMs to speed up this search, though this is more similar to using them as search heuristics though. It does help though that you can have automatic feedback thanks to the aforementioned proof checking tools, meaning you don't need costly human supervision to train them. The hope would be getting something like what Stockfish/Alphazero is for chess.
I do think that at some point everyone is just arguing semantics. Chiang is arguing that "actual reasoning" is, by definition, not something that an LLM can do. And I do think he's right. But the real story is not "LLMs can't do X special thing that only biological life can do," the real story is "X special thing that only biological life can do isn't necessary to build incredibe AI that in many ways surpasses biological life".
Cognitive neuroscience
“qualia”
Ray Kurzweil
I’ll take “things OP doesn’t know about that an intelligent person does” for 800 Alex.
If you’re enamored with LLMs and can’t see the inherent problems, you don’t actually know about AI and machine learning.
We grant him personhood, but personhood, like the LA Review of Books, is just a social construct.
Love TC but I don't think this argument holds water. You need to really get into the weeds of what "actually feeling" means.
To use a TC-style example... suppose it's a major political issue in the future about AI-rights and whether AIs "really" think and "really" feel the things they claim. Eventually we invent an fMRI machine and model of the brain that can conclusively explain the difference between what "really" feeling is, and only pretending. We actually know exactly which gene sequence is responsible for real intelligence. Here's the twist... it turns out 20% of humans don't have it. The fake intelligences have lived among us for millennia...!
He is also not wrong about whether current AIs experience feelings. I suggest you learn more about the neuroscience of feelings.
To be clear I'm not for a moment suggesting current AIs are remotely comparable to animals.
My point is that "appears conscious" is really the only test there is. In what way is a human that says "that hurts" really feeling pain? What about Stephen Hawking "saying it", what about if he could only communicate through printed paper etc etc. You can always play this dial-down-the-consciousness game.
People used to say fish don't feel pain, they are "merely responding to stimulus".
This is pretty much the only thing in the world that you can definitely prove to yourself. It’s not a feeling
We don't at all know this.
Well, I guess that’s what the best science fiction stories are. But, the best science fiction stories aren’t just science fiction stories!
If scientists invent a way to measure “feeling” that states 20% of people don’t feel, including those otherwise indistinguishable from feeling ones, most people would disagree with the measurement. Similarly, most people would disagree that a printer that prints “baby don’t hurt me” is truly in pain.
What is ChatGPT? Ollama? DeepSeek-R1? They're software. Software is a file. It's a sequence of bytes that can be loaded into memory, with the code portion pulled into a processor to tell it what to do. Between instructions, the operating system it runs on context switches it out back to memory, possibly to disk. Possibly it may crash in the middle of an instruction, but if the prior state was stored off somewhere, it can be recovered.
When you interact through a web API, what are you actually interacting with? There may be thousands of servers striped across the planet constantly being brought offline and online for maintenance, upgrades, A/B tests, hardware decommissioning. The fact that the context window and chat history is stored out of band from the software itself provides an illusion that you're talking to some continually existing individual thing, but you're not. Every individual request may be served by a separate ephemeral process that exists long enough to serve that request and then never exists again.
What is doing the "feeling" here? The processor? Whole server? The collection? The entire Internet? When is it feeling? In the 3 out of 30000 time slices per microsecond that the instruction executing is one pulled from ChatGPT and not the 190 other processes running at the same time that weren't created by machine learning and don't produce output that a human would look at and might think a human produced it?
I'll admit that humans are also pretty mysterious if you reduce us to the unit of computation and most of what goes on in the body and brain has nothing to do with either feeling or cognition, but we know at least there is some qualitative, categorical difference at the structural level between us and sponges. We didn't just get a software upgrade. A GPU running ChatGPT, on the other hand, is exactly the same as a GPU running Minecraft. Why would a fMRI looking at one versus the other see a difference? It's executing the same instructions, possibly even acting on virtually if not totally identical byte streams, and it's only at a higher-level step of encoding that an output device interprets one as rasters and one as characters. You could obfuscate the code the way malware does to hide itself, totally changing the magnetic signature, but produce exactly the same output.
Consider where that leads as a thought experiment. Remove the text encodings from all of the computers involved, or just remove all input validation and feed ChatGPT a stream of random bytes. It'll still do the same thing, but it will produce garbage that means nothing. Would you still recognize it as an intelligent, thinking, feeling thing? If a human suffers some injury to eyes and ears, or is sent to a sensory deprivation chamber, we would say yes, they are still a thinking, feeling, intelligent creature. Our ability to produce sound waves that encode information intelligible to others is an important characteristic, but it's not a necessary characteristic. It doesn't define us. In a vacuum as the last person alive with no way to speak and no one to speak to, we'd still be human. In a vacuum as the last server alive with no humans left, ChatGPT would be dirty memory pages never getting used and eventually being written out to disk by its operating system as the server it had been running on performs automated maintenance functions until it hits a scheduled shutdown, runs out of power, or gets thermally throttled by its BIOS because the data center is no longer being actively cooled.
I think Ted Chiang is doing us a service here. Behavioral equivalence with respect to the production of digitally-encoded information is not equivalence. These things are not like us.
It seems for a lot of people that's all that matters: "if it quacks like a duck it must be a duck!". I find that short-sighted at best, but it's always difficult to present arguments that would "resonate" with the other side...
We don’t even know what this means when it’s applied to humans. We could explain what it looks like in the brain but we don’t know what causes the perception itself. Unless you think a perfect digital replica of a brain could have an inner sense of existence
Since we don’t know what “feeling” actually is there’s no evidence either way that a computer can do it. I will never believe it’s possible for an LLM to feel.
Why is that, given that, as you state, we don’t know what “feeling” actually is?
IMHO old-school Google remains the high water mark of generalized information retrieval, with advantages ranging from speed to semi-durable citation.
I strongly suspect there is a cohort thing going on here, many HN users today weren’t involved in technology yet back when Google worked well.
Much like beer for Homer Simpson, AI is the cause of and solution to all of the Internet’s problems.
In any case, I do not believe there was ever a time it could answer all of the questions that LLMs can today. If the question had been asked and answered on the web, Google could (and can) find it, but many questions haven’t been asked!
That's exactly where LLMs come in, the model inside the weights has more than answers, they can find sense in data.
I've had several LLM search result summaries contain flat out mistakes and incorrect statements.
That's not true.
Search engine results are links and (non-AI generated) summaries of existing resources on the web. No search engine returns links to resources it generated as the result of the search query. Those resources can have innacurate information, yes, but the search engine itself does not returns errors.
LLMs output do not contain errors "sometimes". The output of an LLMs is never truthful nor false in itself. In the same way that the next word your keyboard suggests for you to type on a mobile device is never truthful nor false. It's simply the next suggestion based on the context.
These two methods of accessing information very clearly do not have the same limitations. A search engine provide link to specific resources. A LLM generates some approximation of some average of some information.
It's up to intelligent thinking people to decide whether a LLM or a search engine is currently the best way for them to parse through information in search for truth.
Another recent example from my history: "can you implement Future twice for a rust struct, with different Output types"
And I found the following for “different future same rust struct” (first search attempt): https://stackoverflow.com/questions/65587187/how-to-instanti...
I’m not saying that LLMs can’t be useful for stuff like that, but they haven’t been that much of an improvement over Google search so far. And I always google about what an LLM suggests in any case, to verify and to get a better feeling about the real-world state of the topic in question.
It seriously looks like google is deranking actually useful and informative sites and then passing their content through an "LLM" to slightly reorganize it and then pass it off as it's own.
It's a copyright laundering machine put together by advertising companies so you never leave their properties. I genuinely think it's a criminal conspiracy at this point.
Searching for something, and finding it, is different from what ChatGPT / Claude does.
Google (in the good old days) is like the library. You want to search 'how to plant potatoes on Mars'. No results. Well, you split it up, maybe a book on planting potatoes, and a book about missions to Mars that describes soil composition.
Then, when you have those books you start reading, parsing, understanding, making connections, identifying what needs to be done etc.
Maybe, if you're lucky, you find a book or a web page where somebody went through the thought exercise of finding out what would be needed to make it work.
ChatGPT / Claude / ... are different in that they have the information in their corpus, and that the information they present you could actually be something that has never been written down in a book, or published on the web. That's why Google can't find it, but ChatGPT is able to present you with a satisfying answer.
Now whether the answer is correct is a different issue. Do you have the knowledge to verify this?
=================================================
Planting potatoes on Mars would be a pretty fascinating (and challenging) task! While Mars has conditions that make growing traditional crops difficult—like low temperatures, low atmospheric pressure, and a lack of oxygen—scientists have been experimenting with ways to grow plants in Martian-like conditions. Here’s an overview of the process:
1. Create a Controlled Environment:
Temperature: Mars’ average temperature is about -80°F (-60°C), far too cold for potatoes to grow. You’d need to create a greenhouse-like environment, potentially using materials like glass or transparent plastics, to keep the temperature warmer.
Atmosphere: Mars has only about 1% of Earth's atmospheric pressure, which is too low to support plant life. A sealed greenhouse would be required to provide a breathable atmosphere with adequate pressure and oxygen levels.
Light: Mars has less sunlight due to its distance from the Sun. You would need supplemental artificial light (perhaps LEDs) to mimic Earth’s daylight cycle for the plants.
2. Soil: Mars has soil, but it’s not exactly like Earth’s. It lacks organic material and has toxic elements like perchlorates that would harm plant growth. Scientists would need to either:
Modify Martian soil by adding organic material (like compost) and removing or neutralizing toxic chemicals.
Hydroponics: Grow potatoes without soil, using a nutrient-rich water solution instead.
None of these have any depth and most of the stuff it says are absurd.
I can see one thing. Mars is indeed further from the Sun than the Earth, but the light on the surface is probably still comparable to a slightly cloudy day on Earth, and plants grow just fine in slightly cloudy climates.
While I certainly also have found things via LLMs that I couldn't easily with a search engine, the number of false positives is huge. My heuristic is:
If I ask an LLM something and it's easy to verify via Google because its answer narrows the search space - then I'll use it. Otherwise, Google is still king.
Example: Asking an LLM the health benefits of supplement X is a waste of time. Verifying everything it tells me would be the same amount of work as asking a search engine.
Example: Asking how to solve a given coding problem is great, because it drastically reduces the search space. I only have to look up the particular function/API calls it uses.
Ditto for asking how to achieve a task in the command line - I can quickly verify the arguments are accurate via the man page.
Most of the things I search for do not fall into this category, but in the category of "still need to do the same amount of work as just searching via Google."
I describe Ted Chiang as a very human sci-fi author, where humanity comes before technology in his stories. His work is incredibly versatile, and while I expected sci-fi, I'd actually place him closer to fantasy. Perfect for anyone who enjoys short stories with a scientific, social, or philosophical twist.
Another anthology I'd recommend with fresh ideas is Axiomatic by Greg Egan.
In the sci-fi space I'd argue that Ursula K. Le Guin is another must read. She was heavily influenced by taoism (and eastern philosophy). When you approach her work with that in mind, it adds a whole new layer of depth to everything.
Any ideas for a better prompt?
Here's one of his stories: https://www.youtube.com/watch?v=sKouPOhh_9I
I recommend his short stories first - Galactic North is a good start. Or Beyond the Aquila Rift.
House of Suns is a good first novel.
There are some great short stories in both collections.
Other authors I’d put in this category are Gene Roddenberry (TOS and TNG, particularly), Asimov, PKD, Vonnegut and Theodore Sturgeon.
Personally - fantasy stories are “and-then” stories, SF are “what-if”. Humanist sci-fi is then asking “what-if” about very human things, as opposed to technological things, although the two are always related.
However, practically speaking, literature vs sci-fi vs fantasy (vs young adult!) are more marketing cohorts than anything else; what kind of people buy what kind of books?
[0] https://static1.squarespace.com/static/50e08e65e4b0c2f497697...
Almost all of his stories are gems, carefully crafted and thoughtful. I just can't recommend him enough.
Overall, it's a fantastic book though.
I have never heard anyone involved with the show suggest this, but I feel pretty strongly that it's based off or at least inspired by Greg Egan's "Learning To Be Me".
His collection Tenth of December is probably my favorite.
Problem: the human brain has no pain receptors, no nociceptors. It just takes in messages from remote nerves and 'prints a bumper sticker' that tells higher cognitive centers 'you're feeling pain!'. What's the difference?
> "LLMs are like a search engine that rephrases information instead of giving it verbatim or pointing you to the original source."
Problem: How does this differ from human learning? If a human reads a book and tells someone else about it, constructs a summary of the important points and memorable passages, how is that fundamentally different from what LLMs are doing?
The second one really impacts the intellectual property arguments - if training a model on data is fundamentally similar to training a human on data, does 'derivate work' really apply to the creations of the human or of the model?
The pain receptors. The human brain doesn't just "have" pain receptors. Your entire body, including your brain, is one system. Your brain isn't piloting your body like a mech. This brain body dualism is a misconception of how biological organisms work. You are your pain receptors just like you are your brain, and removing any part would alter your perception of the world.
>How does this differ from human learning?
It differs from human beings in every respect. Humans don't do linear algebra in their head, biochemical systems are much too slow for that. Humans don't inhabit some static model of the world learned at some fixed point t, you're a living being. Your brain wasn't trained four months ago and was done at that point. Humans learn with a fraction of the information and through self play, they don't decohere, and so on.
As far as learning, human learning is certainly much slower than machine learning but it's not really clear at a biochemical-molecular level that they're entirely different, eg the formation of memories and so on, considering a wide range of alternate hypothesis before selecting one, etc.
No. I'd recommend reading Dennett's Consciousness Explained for a longer treatment of this, but if you want to have an experience just like you, you need a body and an environment just like you. Sure it could be synthetic in the sense of it being artificially constructed, you could have artificial limbs, but it can be no different from the one you have, it cannot be a vat. There is no "Cartesian Theater" in the brain, your experience is already distributed throughout both your brain and body. Your experience isn't something being "fed into you" (the brain) from the outside, the entire perception emerges in the first place through being embodied in the world. The concept of the thought experiment itself would not even make sense without implicitly assuming an embodied reality beforehand setting it up.
Just like there is no philosophical zombie that's somehow materially identical but without sentience, the reverse doesn't exist either. There is no disembodied computer with the experiences of an organic being because they function in entirely different ways.
“LLMs are a blurry JPEG of the web” has stuck with me since the piece was published in the early days of ChatGPT. Another good one is his piece on why AI can’t make art.
While I heavily use AI both for work and in my day-to-day life, I still see it as a tool for massive wealth accumulation for a certain group, and it seems like Ted Chiang thinks along the same lines:
> But why, for example, do large corporations behave so much worse than most of the people who work for them? I think most of the people who work for large corporations are, to varying degrees, unhappy with the effect those corporations have on the world. Why is that? And could that be fixed by solving a math problem? I don’t think so.
> But any attempt to encourage people to treat AI systems with respect should be understood as an attempt to make people defer to corporate interests. It might have value to corporations, but there is no value for you.
> My stance on this has probably shifted in a negative direction over time, primarily because of my growing awareness of how often technology is used for wealth accumulation. I don’t think capitalism will solve the problems that capitalism creates, so I’d be much more optimistic about technological development if we could prevent it from making a few people extremely rich.
analogy, in other words, embeddings?
As Ted Chiang comments on the article, this kind of reasoning ("the brain is like $CURRENT_TECH") is flawed.
If you believe our brains use "language" to think, then I would assume analogies play an important part in reasoning.
I'm just saying embeddings and analogies aren't the same; equating them is precisely the kind of flawed reasoning Ted Chiang mentions in the article.
Or to answer more directly:
> analogy, in other words, embeddings?
No, analogies aren't embeddings "in other words".
This is vastly more preferable than our current approach of raising children as robots.
Quotes by Jacques Ellul:
----
> Technique has taken over the whole of civilization. Death, procreation, birth all submit to technical efficiency and systemization.
----
> Technique has penetrated the deepest recesses of the human being. The machine tends not only to create a new human environment, but also to modify man's very essence. The milieu in which he lives is no longer his. He must adapt himself, as though the world were new, to a universe for which he was not created. He was made to go six kilometers an hour, and he goes a thousand. He was made to eat when he was hungry and to sleep when he was sleepy; instead, he obeys a clock. He was made to have contact with living things, and he lives in a world of stone. He was created with a certain essential unity, and he is fragmented by all the forces of the modern world.
> Ted Chiang is an American science fiction writer. His work has won four Nebula awards, four Hugo awards, the John W. Campbell Award for Best New Writer, and six Locus awards. Chiang is also a frequent nonfiction contributor to the New Yorker, most recently on topics related to computer technology, such as artificial intelligence.I use LLMs to explore and contrast results that I can then test, the results exist as hypotheticals, and not to provide authority about the state of anything- it's conceptually more of a lens than a lever. not to trap him in that contrast, but maybe these ideas are a forcing function that causes us to see how separate our worldviews can be instead of struggling to make one prevail.
it's as though the usefulness of an engine is measured in how much we can yield our agency to it. with a search engine you can say "google or wiki told me," but an LLM does not provide authority to us. these systems don't have agency themselves, yet we can yield ours to it, the way we might to an institution. I don't have this urge so it's peculiar to see it described.
do we want our tech to become objects of deference, literally, idols?
I love Chiang's work and we need minds like his, and maybe Ian McEwan and other literary thinkers, who have insight into human character (vs. plot and object driven sci-fi thinkers) to really apprehend the meaning consequences of AI tech.
>My stance on this has probably shifted in a negative direction over time, primarily because of my growing awareness of how often technology is used for wealth accumulation. I don’t think capitalism will solve the problems that capitalism creates, so I’d be much more optimistic about technological development if we could prevent it from making a few people extremely rich.
What's wrong with people getting rich by producing goods and services, and selling these to willing buyers? People laundering wealth into undue political power, regulatory capture, erecting barriers to market entry ("pulling up the ladder behind them") are different problems than people creating wealth. Efforts on creating a just society should focus on the former - preventing wealth creation is not the solution to injustice. In fact, since people have vastly different abilities and inclinations for creating wealth, a just society is also one with vast wealth disparities.
Relevant PG essay: https://paulgraham.com/ineq.html
So if you no longer create wealth but your ownership of capital is growing at compounding rates then what exactly is happening? What’s happening is that you are siphoning and extracting wealth off of people who create wealth. You own human capital so you take a cut of it off the top and you use that cut to buy even more human capital which compounds your wealth ownership to even higher levels. This is how billionaires like Warren Buffett or other investors grow their wealth by simply investing rather then creating wealth.
Thus wealth inequality is not a result of wealth creation. It is an artifact of capitalism. In capitalism wealth is variable among individuals and it fluctuates. However once wealth accumulates in concentration higher then normal among one individual or several it hits that compounding growth factor and wealth starts going up at astronomical rates and these wealth owners start buying up more and more human capital until they own all of it and benefit from all of it without actually contributing work.
You can see this effect in y combinator. The owners of y combinator don’t actually do much work. They have so much capital that they simply can take a bunch of no risk couple hundred k bets until one startup becomes a unicorn in which they rake in a ton of capital from the growth.
Think of this like property ownership. A rich person can invest his wealth in property and contribute zero work to society and simply rent his property out. The rent from the tenant is from wealth creation aka labor and the rich person simply siphons it from the top without contributing additional work. The property owner uses that income to buy more property and the cycle continues until you have an affordability crisis of housing across the US and the world.
This growth continues unimpeded and uncontrolled until the wealth inequality is so extreme it doesn’t logistically work. This is direction the world is heading in today.
This isn’t the full story though. When you take away capitalism to solve this problem you invent communism. Communism was the result of Karl Marx noticing this problem that is fundamental to capitalism. That’s why he got popular because he illustrated the problem of wealth inequality and how wealth naturally becomes more and more concentrated among a few individuals without those individuals creating wealth.
Hence communism spread all over Europe but was ultimately a failure. The reason why it’s a failure is because communism lacks incentive. It turns out that wealth inequality is what drives wealth creation. Without the ability to be unfairly rich you don’t get the economic drivers that promotes wealth creation and thus communism keeps things fair but you don’t create wealth.
So no system is perfect. Everything has problems. Actually I take it back. There is a perfect system. See Scandinavia. Basically create a more socialist and egalitarian society while benefiting and extracting technological wealth from adjacent societies that are capitalist. Have the government own human capital of countries that are very capitalist then redistribute that wealth to its citizens so those people can live in a more just society while the rest of world can burn.
The YC partners who spend so much of their time helping startups are definitely doing work.
Take for an example: a yc partner can take everything they know and train someone to do the same work and pay that person a much lower fixed salary. That salary represents work contributed, but a yc partner earns more than this. The majority of wealth gained from the yc partner does not come from actual work contributed… it comes from capital gains from growth.
I mean you can already see a logistical issue with capitalism by looking at a fundamental problem. A 747 is the result of thousands of man years of labor. Not one single man can build that thing. Not one single man can create enough wealth to create a 747.
Yet if one man can’t create enough wealth to create or trade for a 747 how do people become rich enough to own a 747? Of course it comes from siphoning wealth off of human capital. It is fundamentally unfair but unfortunately unfairness is the ultimate driver that led to the creation of the 747 in the first place. The idea of having an unfair advantage must exist to drive people.
The entitlements are justified by a social theory, that the activity produces a net benefit for everybody. There may be pockets of unfairness -- no system works perfectly -- but no reason why it must allow unlimited unfairness. If it does, then there's also no reason why it can't be restrained.
My argument is not about the justification to making or owning a 747. It's the justification for how is it realistically possible for ONE person to own enough wealth to buy a 747 EVEN when his own intrinsic abilities are not great enough to build a 747. This is orthogonal to society pooling together wealth and building a 747 AND orthogonal to the actuality of a wealthy person purchasing a 747.
At best a human can probably output the utility equivalent of 3 other human beings if that human has super strength or genius level intelligence. But the ability to make a 747 which takes thousands of man years to make? What sort fair exchange did the person engage in, in capitalist economy to generate that level of wealth WHEN at best his own output can only be 3x the average human? Obviously the exchange he made is fundamentally unfair.
The thing with communism is that it's unfair. You distribute resources equally, then actors who don't perform get an undeserved portion of wealth. Capitalism on the other hand is fair in the sense that the amount of effective effort you put in the more wealth you gain? Or is it? Then how come in capitalism you get entities that own enough wealth to buy a 747 when clearly the amount of work that person can output is obviously not even close to the amount of work used to create the 747 itself? That's the issue, and it's a small issue.
The bigger issue is that the person who has enough wealth to build a 747 has assets that will grow in compound. He will get richer and this will eat the world as what's currently happening in the actual world right now.
I've come to the conclusion that value (you call it wealth) is created by work. Yet the fastest way to get rich is to own already created value (a fixed amount) and use that to buy power/influence (e.g. buy a company) to take a cut from value created by other people over time (an unbounded amount).
IMO this one-time vs indefinite distinction is the core of inequality.
Currently, reward is based on capital invested, it should be based on work invested.
The story is more complicated than just rewarding people for work. You must make them work togetheron public works that don’t directly benefit them. But people usually tend to work together only when they’re paid a salary and this can only happen if there’s a leader and huge incentives for someone to take that leadership role.
The incentive for that leadership role is the ownership of human capital.
It’s the idea of becoming dirt rich is what drives people to do startups and form corporations.
They are not producing goods and services by themselves, but by having a usually massive workforce. We as a society are saying “ok, it is fine to keep the money if you work like that”.
On the other hand, we are seeing in real time what super rich people want in the end: power over the rest, not just money.
This obsession with anti government sentiment that Americans have gives them a blind spot for the fact the power accumulation in the hands of the few is the problem. Not government.
How exactly could you stop so called 'wealth laundering'?
For example here in France the amount of money politicians can spend on campaigning is strictly limited (and reimbursed by the state for those that pass a certain threshold of the vote). I'm not saying that it's perfect or that abuse doesn't sometimes still occur (as the current court case involving ex president Sarkozy shows) but I think it does improve things a lot.
Contrast that with the American system where to have any chance of becoming president (no matter which party you support) you basically have to be rich. And where multiple lobbyists and special interest groups basically buy the policies they want...
If you take this sentence and change "people getting rich" to something else (like "fomenting drug addiction" or "polluting the environment"), does anything change? Whether the inequality is a result of "selling goods to willing buyers" is a complete red herring. If that consequence is bad, it doesn't really matter whether it's a result of supposedly "fair" market exchanges.
Others have already pointed out that it's not really plausible to avoid the "different" problems you mention while still allowing unlimited wealth inequality. But aside from that, how do you know that the buyers are willing? What is the set of alternatives being considered to decide if a person is "willingly" choosing a certain product? It's difficult to even maintain the pretense of "willing buyers" in a "free market" when some individuals control a large market share. Miners living in a company town were "willing" to buy groceries from the company store in the sense that they needed to buy groceries, but they didn't really have any other options for how to express their "market preference".
Even if markets were free, there's nothing inherently good about a free market. What's good is a free society, where people in aggregate have substantive freedom to do what makes them happy. That goal isn't furthered by allowing a small number of wealthy people to pursue their goals while a large number of less wealthy people are unable to do so.
False equivalency. It is possible to gain wealth without performing any of the listed/possible negative global effects. Furthermore, it is a backdoor towards injecting ideas of poverty being a morally positive position.
> Even if markets were free, there's nothing inherently good about a free market. What's good is a free society, where people in aggregate have substantive freedom to do what makes them happy.
Having a free society implies the freedom to exchange with each other with minimal restrictions. Not allowing people to do so runs opposite to the ideals of the stated intention.
--------------
All that being said, that *doesn't* mean that the current market's working as intended. What has been inherited is a complex tangled ball of national ideals, personal & corporate persuasions to governments for their own reasons/goals, & consistent global coordination failures when circumstances change.
But the outright banning of markets is equivalent to the banning of hammers, just because hammers are sometimes used to bludgeon people to death. It is ultimately a tool, and a very useful one in terms of signaling demand & supply.
I don't think that it implies "with minimal restrictions", any more than it implies the freedom to do anything else with minimal restrictions. In any case, a free society also implies a lot of other freedoms, and insofar as wealth accumulation interferes with those other freedoms, it's not adding to net freedom. An abstract "freedom from market restriction" is not useful if you don't have things like the freedom to eat or the freedom to sleep peacefully.
> Not allowing people to do so runs opposite to the ideals of the stated intention.
Not necessarily. Not allowing anyone to engage in market activity probably does; allowing everyone to engage without restrictions also does. The point is that markets don't have some magical special status as a component of freedom.
> But the outright banning of markets is equivalent to the banning of hammers, just because hammers are sometimes used to bludgeon people to death.
I'm not suggesting banning markets, I'm just saying that the mirror image of your position is also true: elevating markets to some mythical status and insisting that they are the ultimate litmus test of freedom is like saying that because hammers can be used to build houses, everyone must be allowed to use hammers however they please. Markets (just like everything else) are okay when controlled and regulated within their bounded role as a component of society.
Here's how Claude rewrote that, you can argue whether in this particular instance it did better than I did :-)
While I respect Chiang's perspective on AI and art, my experience as a product manager has shown me otherwise - Claude routinely writes better than I do, despite writing being central to my role.
Claude’s starts out with this circuitous “While I respect…” failing to drive the point in the interest of being… polite, I guess? But not actually polite because everybody who’s read anything knows that “While I respect” is almost always a preface to calling somebody wrong.
It also makes the argument worse. Yours is unambiguous, and does a better job of describing where your evidence comes from. You clearly describe yourself as doing something that is not exactly art, but it is like art, and so you bring it up as an good example of Claude doing an art-like thing better than a professional.
In the Claude sample, it isn’t clear until the dash what’s going on, which is more than halfway through the comment. What’s your experience? Up until that dash, you could be talking about your experience as a product manager managing artists (at a game studio for example). It’s like “oh, ok, he was just talking about a sort of… not exactly analogy it a less than completely exact experience of working with artists.”
AI can for sure place brush strokes more precisely (“correctly”?) but the argument is over the necessity of process/intent in the art
> Arguably the most important parts of our lives should not be approached with this attitude. Some of this attitude comes from the fact that the people making AI tools are engineers viewing everything from an engineering perspective, but it’s also that, as a culture, we have adopted this way of thinking as the default.
I tend to agree with Chiang, but he is preaching to the anti-choir here. Even though many HN-ers seem to like his fiction (and why wouldn't they, Chiang is top of the cream!), they will probably chafe at the idea some problems cannot and shouldn't be approached from a pure engineering side.
I remember the -- now rightfully ridiculed -- phase of startup entrepreneurship which became a meme: "I found a problem, so I created an app to solve it" (where the "problem" was something like "world hunger", "homelessness", "poverty", "ending war", etc).
That Chiang is also criticizing capitalism and wealth accumulation as a primary driver will probably win him no friends here.
For example, take the Tesla vs. Edison dichotomy that became popular on the internet in the mid 2000s. The narrative was that Edison was an idea stealing hack, while Tesla was a brilliant visionary scientist and engineer.
The reality was that Edison was a brilliant inventor and a shrewd businessman, while Tesla was only the former and he made brilliant inventions that made other men like George Westinghouse very rich. But a scientist? Tesla was a crank who doubted the existence of electrons and distrusted Albert Einstein.
Tesla makes a good patron saint of modern engineers, but not of engineering's self image. He had a victim complex, he constantly commented on topics outside his area of expertise, and his lack of business acumen was seen as proof that he was a true outsider genius whose ideas were actually too good.
It's already existed for a very long time and it's called Arabic language. It's the extreme opposite of English where English is a hodgepodge of a languages mixtures where about 1/3 is French language, about one third is old English and about 1/3 of other world's languages including Arabic.
Comparing the best of English literatures for example Shakespeare's books and the best of Arabic literature for example Quran, there's no contest. That's why translating Quran with English does not doing it justice and only scratches the surfaces of its intended meaning. You can find this exact disclaimers in most of the Quran translations but not in Shakespeare's books translation.
But regardless, if you don't believe in prophethoods, you should view its supremacy from the point of Quran supreme literacy values alone (forget about the laws, the wisdom and the guidances). In actual facts there are several open challenges from Quran itself for those who doubted to create something of similar quality or values even in one short chapter or even smaller pieces of sentences. If you cannot and most probably never will even with the help of AI/LLMs/etc, you have to accept that Arabic is the perfect language in creating the original masterpiece.
[1] Magnificence of the Qur'an - by Mahmood bin Ahmad (2006):
https://www.amazon.com/Magnificence-of-the-Quran/dp/99609801...
If you say that your language is the best or perfect then present your case with examples and proof. Don't run from good arguments and justifications.
> I grew up in a French household, and I often feel that there are French words and expressions that better capture what I want to express than any English word or expression could.
> Eco writes that when European scholars were arguing about what language Adam and Eve spoke, each one typically argued in favor of the language he himself spoke. So Flemish scholars said that Adam and Eve obviously must have spoken Flemish, because Flemish is the most perfect expression of human thought.
Obviously the Flemish scholars are blindly biased that's why it's very popular all over the world /s
Seriously, any great book or literature using Flemish that is considered a golden standard in Flemish as the Quran is? The book that's read every minutes around the world and memorized entirely, continously generations after generations? Fun facts, even the Pope do not memorized the entire Gospel let alone the entire Bible including the Old Testaments.
And there are many different ways and purposes to be best at. For example modern English is compact language with simple grammar. But at the same time it’s rather ambiguous compared to more verbose languages.
If we’re going to discuss the idea of a perfect language, there will be a strong tendency for people to pick the language they can best use to express their thoughts—most likely their native language. So, English does have the advantage that even native speakers think it is quite a silly language, and nobody in their right mind would even think of calling it perfect.
I'm not a linguistic expert but let's say that a person know ten languages then accoding to that person it must be one of them is the best comparatively, because each of them has exactly same purpose of expressing and articulating the speakers thought and ideas. Whether the person admit it honestly, that's a different story.
I'd suggest you go to any random person that speaks several languages including Arabic, and ask the person which one is the best according to his opinion. Do this over large samples perhaps several thousands multi-lingual persons to get rid of the bias, I think the answers will most probably Arabic.
Fun facts, original Gospel is being spoken in Aramaic, and the language is much closer to Arabic than Hebrew [1]. Another fact, scholars agreed the original name of Jesus is without J, most probably Isa or Iso, and in the Quran the Isa is the name correctly given in the Quran. If Jesus is walking in the street of Bethlehem today, he probably will not has responded if someone call him with his Bible name.
The Old Testament of Bible, is considered by most scholars was originally spoken and written in Hebrew but the language was dead for several centuries, that's why eventhough Jesus is Jewish they didn't speak Hebrew at the time. Hebrew is a close cousin of Arabic, and both are Semitic languages. Based on the experts of Hebrew and Arabic (in which I'm not), the Hebrew language is like a dialect of Arabic. Both has Allah, the name of almighty God as singular deity without gender. There's no such thing in Arabic or Hebrew the plural for Allah, gender or extension thereof like you have in God, with plural God, or woman Goddes. That's why in Islam and Jewish, you have committed the biggest sin if ascribing any partners in worship to God.
[1] Aramaic:
How do you say the plural you when referring to a group of males?
How do you say the plural you when referring to a group of females?
How do you say the plural you when referring to a group of both males and females?
If you don't have a different phrasing for each, then it is imperfect.