Thinking Fast, Slow, and Artificial: How AI Is Reshaping Human Reasoning
papers.ssrn.com
papers.ssrn.com
In other words, I try to learn from it whenever it does something I can't do but when it does something I can do or something I'm really good at it I find myself wanting to correct it cause it doesn't do it that well.
It just seems like a really quick thinking and fast executing but, ultimately, mid skilled / novice person.
Apparently this was caused by the context window getting full!
(At least I assume that because it went back to the old behavior after I triggered a compaction)
That's a lot of stuff, but it also doesn't include a lot of the stuff people claim AI can do.
Just yesterday I asked Gemini Pro 3.0 this question:
> Find such colors A and B:
> A and B are both valid sRGB color.
> Interpolating between them in CIELAB space like this
> C_cielab = (A_cielab + B_cielab) / 2
> results in a color C that can't be represented in sRGB
It gave me a correct answer, great!
...and then it proceeded to tell me to use Oklab, claiming it doesn't have this problem because the sRGB gamut is convex in Oklab.
If I didn't know Oklab does have the exact same problem I would have been fooled. It just sounds too reasonable.
It helps if you phrase the question openly, not obviously fishing for a yes-or-no answer. Or, if you have to ask for a yes-or-no question, make it sound like you're obviously expecting the answer that's actually less likely, so the AI will (1) either be more willing to argue against it, or (2) provide good arguments for it you might not have considered, because it "knows" the answer is unexpected and it wants to flatter your judgment.
I do this all the time and hate that I have to do it, with the additional "do not yes-man me, be critical."
Relationships with real people are pretty cool actually. If you talk to people that you have a longer relationship with, you might also be able to judge their areas of expertise and how prone to bullshitting they are.
So you know it can be full of sh1t on all kinds of topics, and you start learning from it the moment it's 'talking' about subjects you know you don't know about? To me that sounds like the moment to stop, not the moment to start. Or am I missing something?
Most long-term gamblers will tell you that the first games they played, they won. This is a real thing, yet we cannot apply it by making one bet and then stopping, because so are the probabilities being fair and un-biased.
What squares these two things is that most of the people who played and lost their first games, did not get addicted to gambling.
On non-code stuff, I think its improved or there are better options for making it get to the point and be concise more and I find when I correct it, quite often we actually get somewhere. The answers I remember from my initial use of it ofbasically how to do anything or most subjects was practically a 10 pager with some weird action plan that you were never gonna go thru.
The reality is much more stark then your description. Yes, in MANY instances it fails at things you know and you're an expert at. But in MANY instances it also beats you at what you're good at.
People who say stuff like the parent poster are completely mischaracterizing the current situation. We are not in a place where AI is "good" but we are "better". No... we are approaching a place of we are good and AI is starting to beat us at our own game. That is the prominent topic that is what is trending and that is the impending reality.
Yet everywhere on HN I see stuff like, oh AI fails here, or AI fails there. Yeah AI failing is obvious. It's been failing for most of my life. What's unique about the last couple years is that it's starting to beat us. Why? Because your typical HNer holds programming as not just a tool, but an identity. Your skill in programming is also a status symbol and when AI attacks your identity, the first thing you do to defend your identity is to bend reality and try to cast to a different conclusion by looking at everything from a different angle.
Face Reality.
If you nudge the AI in the right direction, it may surprise you with what it's capable of. But if you nudge it in a wrong direction or just don't give it sufficient context, it can be very confidently wrong.
Ai is different but it is also similar, for example it can speak language. So it is different and similar at the same time. I am obviously referring to AI beating us where we are similar to it. Best example: software engineering. This is obvious. The fact that I need to spell it out shows how deep the delusion goes.
The rest of your response is just regurgitating what the parent post said. Sure. But it doesn’t address the fact that while everything you said is true part of the time the other part of the time it beats us (that includes both nudging and no nudging it in the right direction).
Additionally all of this is also completely ignoring the fact that AI leap frogged in capability in the past year with my entire company now foregoing the use of text editing or using IDEs and having Claude write everything. If what you say is true only now and we see this much velocity in improvement then wait a couple more years and everything you said can be completely false if the trendline continues.
And here’s the craziest part. Everything im saying is obvious. I’m basically being captain obvious here. The question is why are so many people in total denial.
We still hand hold it a bit. If it makes a mistake we just tell it to fix the mistake or do it in a different way. It’s that good.
I like to think of them as idiot savants with exponential more savant than your typical fictional idiot savant. They pivot on every word you use, each word in your series activating areas of training knowledge, until your prompt completes and then the LLM is logically located at some biased perspective of the topic you seek (if your wording was not vague and using implied references). Few seem to realize there is no "one topic" for each topic an LLM knows, there are numerous perspectives on every topic. Those perspectives reflect the reason one person/group is using that topic, and their technical seriousness within that topic. How you word your prompts dictates which of these perspectives your ultimate answer is generated.
When people say their use of AI reflects a mid level understanding of whatever they prompted, that is because the prompt is worded with the language used by "mid level understanding persons". If you want the LLM to respond with expert guidance, you have to prompt it using the same language and terms that the expert you want would use. That is how you activate their area of training to generate a response from them.
This goes further when using coding AI. If your code has the coding structure of a mid level developer, that causes a strong preference for mid level developer guidance - because that is relevant to your code structure. It requires a well written prompt using PhD/Professorial terminology in computer science to operate with a mid level code base and then get advice that would improve that code above it's mid level architecture.
In more words, "of course it's stupid, it's as complex as a mid-sized rodent where we taught it purely by selective breeding on getting answers right while carefully preventing any mutations which made their brains any bigger".
You have to come into it with the same "these people are only stupid and lack the experience to answer my questions despite thinking they do, they lack the world view to even process how I arrived at the parameters of my question(s)" apprehension like you would if asking reddit about some hazardous thing that would make them all screech. AI is the margarine to that butter.
It's a technology with potential to deliver great value, but there are limitations...
You should be learning alongside the llm through the research phase of anything. Updating your understanding of what is possible and best practices with rigorous checks and limiting scope to a high fidelity to leave little room for doubt. In-line commenting and questioning and asking for more passes on the living document of the area you are working on and then judiciously breaking it down further when you think there is too broad a scope for an llm to understand and synthesise properly.
If you do end up with too much vagueness, you need to limit scope more or break up the feature, implementation etc to be specific and applies enough to again, properly research and decide the plan.
I guess this is not so easy because lot of it depends on your own ability of reading comprehension, but I've had great success learning niche topics because I research (as a sub agent usually) essentially any topic that is mysterious until every level of the puzzle is properly mapped out to the specificity required.
Do I think most people are doing this? No. So I guess the statistics make sense. It's not intuitive to many people I think - because as you said, it's an embodiment of literature that is a tangled web of thought patterns and perspectives, so you need to pare it's answers down to the specific level, direction and area of ideas you want to get out of it. Way easier to do than it sounds, but it requires finesse in comprehension rather than getting lazy with it - normalcy of deviance comes to mind.
So the smart get smarter and the dumb get dumber?
Well, not exactly, but at least for now with AI "highly jagged", and unreliable, it pays to know enough to NOT trust it, and indeed be mentally capable enough that you don't need to surrender to it, and can spot the failures.
I think the potential problems come later, when AI is more capable/reliable, and even the intelligentsia perhaps stop questioning it's output, and stop exercising/developing their own reasoning skills. Maybe AI accelerates us towards some version of "Idiocracy" where human intelligence is even less relevant to evolutionary success (i.e. having/supporting lots of kids) than it is today, and gets bred out of the human species? Maybe this is the inevitable trajectory: species gets smarter when they develop language and tool creation, then peak, and get dumber after having created tools that do the thinking for them?
Pre-AI, a long time ago, I used to think/joke we might go in the other direction - evolve into a pulsating brain, eyes, genitalia and vestigial limbs, as mental work took over from physical, but maybe I got that reversed!
Don't kid yourself. If you use this junk, it's making you dumber and damaging your critical thinking skills, full-stop. This is delegation of core competency. You may feel smarter, or that you're learning faster, of that you're more productive, but to people who aren't addicted to LLMs it sounds exactly like gamblers insisting they have a foolproof system for slots, or alcoholics insisting that a few beers make them a better driver. Nobody outside the bubble is impressed with the results.
This is a good way to frame the problem. Consider the offshoring (delegation) of American manufacturing to China, followed by the realization decades later that the US has forgotten how to actually make things and the subsequent frenzied attempt to remember.
I expect the timelines and second-order (third-order...) effects to play out on a similar decadal scale - long after everybody has realized their profits and the western brain has atrophied into slop.
Arguably I've been using my critical thinking skills more now that I have a smooth talking, but ultimately not actually intelligent companion.
Every time I put undue trust in it, I regret it, so I got used to veryfing what it outputs via documentation and sometimes even library code.
That being said worst part of this mess is that my usual sources of knowledge like search engines or developer forums dried up, as everyone else is also using LLMs.
Mentioning this here because just like your comment, this 'theory' is usually slid inside arguments to make it appear as established science or fact. Kinda like this AI debacle.
The book you disparaged is written by a real scientist and published by Oxford Uni Press. They are smarter than you, and if it were 1 + 1 = 3, OUP wouldn't have published it. Even if we disregard all this, the fact that you judged a book without reading it says a lot about your critical thinking skills.
(Minus the Fermi paradox part)
I'd love to see an empirical study that actually dives into this and attempts to show one way or another how true it is. Otherwise it's just all anecdotes.
That's not all that the placebo effect is. But it's probably the aspect that best fits the framing as bias
You actually get better through placebo, as long as there's a pathway to it that is available to your body.
It's a really weird effect.
The fight isn't against triggering placebo, it's against letting it muddle study results.
This mostly happens with things I’ve already had long cognitive loops on myself, and I’m feeling stuck for some reason. The conversation with the model is usually multiple iterations of explaining to the model what I’m working through.
This in turn has given me the ability to "double" think. I am conciously thinking while I have another part of my brain also thinking about it on a bigger scope that I could conciously grasp.
I suggest everyone interested in learning how these theories emerge, and how the social sciences work, to give it a read. Also, it kind of dismantles the whole idea of System 1 and 2, which then I guess would question the theoretical foundations of this paper too.
Critique to System 1 and 2 is based mostly on using System 1 and 2 to excuse the alleged deficiencies in the experiments.
I think the original article in this discussion is using "Systems 1 and 2" as intuitive and rational modes for problem-solving and interestingly enough, Gerd Gigerenzen also has a reference in this work "accuracy-effort trade-off (Payne, Bettman, and Johnson 1993): The less effort one takes, the less accurate one will be." which aligns with the broader idea of Systems 1 and 2.
Like kids who are never taught to do things for themselves.
People have worried with every single new technology that it will enfeeble the masses, rather than empower them, and yet in the end, we usually find ourselves better off.
Yeah when I was learning in school we weren't allowed electronics for division, and I think I absolutely would be dumber if I had never done that
> People have worried with every single new technology that it will enfeeble the masses, rather than empower them, and yet in the end, we usually find ourselves better off.
If you're posting this from America, you're living in a society that is fatter than ever thanks to cars. So there's surely some nuance here, not every technology upgrade is strictly better with no downsides
Cars are an essential part of modern life, but the sweetspot for car adoption isn't on either of the extremes
On the other hand there are also big positives on both the societal and individual level. That's where the balance comes in. You want some individual travel and part of your logistics to run on cars, but not all of it. And probably a lot less of it than what most people in the 60s to 90s thought
For real, the amount of hate and vitriol I see expressed by people behind the “safety” of their steering wheel is unbelievable. Surely driving (excessively) leads to misanthropy like cigarettes to cancer.
In some parts of the world perhaps? They're not an essential part of life in urban areas designed to work well without them. As in, many people can live their lives never using one, let alone owning one.
Sometimes I take breaks from the calculator and even review math videos because it's embarrassing when I can't help my kid with their homework.
Taking care in how and when we use AI seems very sensible. Just like we take care how often and how much refined sugar we eat, or how many hours we spend sedentary.
But the analogy doesn’t actually hold up anyhow because the calculator and the navigator are deterministic. I can rely on their output.
LLMs have a probabilistic output that absolutely needs verification every time. I cannot trust them the same way I can trust a calculator.
Personally I also hurt my learning of trig identities and stuff because the symbolic algebra engine on my ti-89 was so good that I could rely on it instead of learning the material. Caught up to me in college with harder calc and physics classes.
That was good for a half-semester but then a formidable classroom opponent arose: a "new" boy who had been educated in another state using the very same textbook! I realized I'd have to commit at least a handful of the most useful trig identities to memory to solve problems quickly and remain at the head of the class. A weekend of furious comparison and selection ensued, but that was enough to carry me across the finish line in trig class.
I still memorize phone numbers. Hey, today that counts as "not using a database".
I am so rusty, that I just do add and subtract.
On the other hand, my grandparents, and father, could look at financial documents and do the calculations in their head.
People I know who stayed in finance longer than me, can crunch numbers rapidly.
I am around numerate people most of the time, so the occasions where I find I am the faster calculator around are jarring.
There are many conversations that go adrift because we can’t crunch numbers fast enough.
Is it a net loss to humanity in the face of the gains we obtained. Nope.
Is mental fitness of value to me, the same way physical fitness is of value to me? Yes, very much.
When you googled something and got five contradictory results, that told you the question was hard. A clean AI answer doesn't give you that signal. Coherence looks the same whether the answer is right or wrong.
The failure mode didn't get worse. It got quieter.
But, we still have the System 1, and survived and reached this stage because of it, because even a bad guess is better than the slowness of doing things right. It have its problems, but sometimes you must reach a compromise.
Large parts of the paper score very high probability of being written entirely by AI in gptzero.
I'm not sure if I could trust anything written in it.
Which is kind of duh? Of course. They have some cool language like calling the AI system 3 and calling taking advice 'cognitive surrender' but I'm not sure how this differs from asking your mate Bob and taking his advice?
It's also probably bad about something else important.
Current status: partially solved.
Problem: System 2 is supposed to be rational, but I found this to be far from the case. Massive unnecessary suffering.
Solution (WIP): Ask: What is the goal? What are my assumptions? Is there anything I am missing?
--
So, I repeatedly found myself getting into lots of trouble due to unquestioned assumptions. System 2 is supposed to be rational, but I found this to be far from the case.
So I tried inventing an "actually rational system" that I could "operate manually", or with a little help. I called it System 3, a system where you use a Thinking Tool to help you think more effectively.
Initial attempt was a "rational LLM prompt", but these mostly devolve into unhelpful nitpicking. (Maybe it's solvable, but I didn't get very far.)
Then I realized, wouldn't you get better results with a bunch of questions on pen and paper? Guided writing exercises?
So here are my attempts so far:
reflect.py - https://gist.github.com/a-n-d-a-i/d54bc03b0ceeb06b4cd61ed173...
unstuck.py - https://gist.github.com/a-n-d-a-i/d54bc03b0ceeb06b4cd61ed173...
--
I'm not sure what's a good way to get yourself "out of a rut" in terms of thinking about a problem. It seems like the longer you've thought about it, the less likely you are to explore beyond the confines of the "known" (i.e. your probably dodgy/incomplete assumptions).
I haven't solved System 3 yet, but a few months later found myself in an even more harrowing situation which could have been avoided if I had a System 3.
The solution turned out to be trivial, but I missed it for weeks... In this case, I had incorrectly named the project, and thus doomed it to limbo. Turns out naming things is just as important in real life as it is in programming!
So I joked "if being pedantic didn't solve the problem, you weren't being pedantic enough." But it's not a joke! It's about clear thinking. (The negative aspect of pedantry is inappropriate communication. But the positive aspect is "seeing the situation clearly", which is obviously the part you want to keep!)
I LOLed.