Hacker News Reader

Comment by nikisil80 | Hacker News Reader

Best reply in this entire thread, and I align with your thinking entirely. I also absolutely hate this idea amongst tech-oriented communities that because an AI can do some algebra and program an 8-bit video game quickly and without any mistakes, it's already overtaking humanity. Extrapolating from that idea to some future version of these models, they may be capable of solving grad school level physics problems and programming entire AAA video games, but again - that's not what _humanity_ is about. There is so much more to being human than fucking programming and science (and I'm saying this as an actual nuclear physicist). And so, just like you said, the AI arm's race is about getting it good at _known_ science/engineering, fields in which 'correctness' is very easy to validate. But most of human interaction exists in a grey zone.

Thanks for this.

View on HN

> that's not what _humanity_ is about

I've not spent too long thinking on the following, so I'm prepared for someone to say I'm totally wrong, but:

I feel like the services economy can be broadly broken down into: pleasure, progress and chores. Pleasure being poetry/literature, movies, hospitality, etc; progress being the examples you gave like science/engineering, mathematics; and chore being things humans need to coordinate or satisfy an obligation (accountants, lawyers, salesmen).

In this case, if we assume AI can deal with things not in the grey zone, then it can deal with 'progress' and many 'chores', which are massive chunks of human output. There's not much grey zone to them. (Well, there is, but there are many correct solutions; equivalent pieces of code that are acceptable, multiple versions of a tax return, each claiming different deductions, that would fly by the IRS, etc)

I'm not sure where construction and physical work goes into your categories. Process and chores maybe. But I think AI will struggle in the physical domain - validation is difficult and repeated experiments to train on are either too risky, too costly or potentially too damaging (i.e. in the real world failure is often not an option unlike software where test benches can allow controlled failure in a simulated env).

Neither, my categories only cover "services" (at least as Wikipedia would categorise things into this bracket: https://en.wikipedia.org/wiki/Service_economy).

I agree with you on construction and physical work.

I have considered this too. I frame it as problem solving. We are solving problems across all fields, from investing, to designing, construction, sales, entertainment, science, medicine, repair. What do you need when you are solving problems? You need to know the best action you can take in a situation. How is AI going to know all that? Some things are only tacicly known by key people, some things are guarded secrets (how do you make cutting edge chips, or innovative drugs?), some rely on experience that is not written down. Many of those problems have not even been fully explored, they are open field of trial and error.

AI progress depends not just on ideation speed, but on validation speed. And validation in some fields needs to pass through the physical world, which makes it expensive, slow, and rate limited. Hence I don't think AI can reach singularity. That would only be possible if validation was as easy to scale as ideation.

programming entire AAA video games

Even this is questionable, cause we're seeing it making forms and solving leetcodes, but no llm yet created a new approach, reduced existing unnecessary complexity (which we created mountains of), made something truly new in general. All they seem to do is rehash of millions of "mainstream" works, and AAA isn't mainstream. Cranking up the parameter count or the time of beating around the bush (aka cot) doesn't magically substitute for lack of a knowledge graph with thick enough edges, so creating a next-gen AAA video game is far out of scope of llm's abilities. They are stuck in 2020 office jobs and weekend open source tech, programming-wise.

"stuck" is a bit strong of a term. 6 months ago I remember preferring to write even Python code myself because Copilot would get most things wrong. My most successful usage of Copilot was getting it to write CRUD and tests. These days, I can give Claude Sonnet in Cursor's agent mode a high-level Rust programming task (e.g. write a certain macro that would allow a user to define X) and it'll modify across my codebase, and generally the thing just works.

At current rate of progress, I really do think in another 6 months they'll be pretty good at tackling technical debt and overcomplication, at least in codebases that have good unit/integration test coverage or are written in very strongly typed languages with a type-friendly structure. (Of course, those usually aren't the codebases needing significant refactoring, but I think AIs are decent at writing unit tests against existing code too.)

"They are stuck in 2020 office jobs and weekend open source tech, programming-wise."

You say that like it's nothing special! Honestly I'm still in awe at the ability of modern LLMs to do any kind of programming. It's weird how something that would have been science fiction 5 years ago is now normalised.

All true, but keep in mind the biggest boosters of LLMs have been explicitly selling it as a replacement for human intellectual labor--"don't learn to code anymore", "we need UBI", "muh agents" and the like.

OK but getting good at science/engineering is what matters because that's what gives AI and people who wield it power. Once AI is able to build chips and datacenters autonomously, that's when singularity starts. AI doesn't need to understand humans or act human-like to do those things.

I think what they mean is that the fundamental question is IF any intelligence can really break out of its confined area of expertise and control a substantial amount of the world just by excelling in highly verifiable domains. Because a lot of what humans need to do is decisions based on expertise and judgement that in systems follows no transparent rules.

I guess it’s the age old question if we really know what we are doing („experience“) or we just tumble through life and it works out because the overall system of humans interacting with each other is big enough. The current state of world politics makes be think it’s the latter.

I don't necessarily think you're wrong, and in general I do agree with you to an extent that this seems like self-centeted Computer Scientist/SWE hubris to think that automating programming is ~AGI.

HOWEVER there is a case to be made that software is an insanely powerful lever for many industries, especially AI. And if current AI gets good enough at software problems that it can improve its own infrastructure or even ideate new model architectures, then we would (in this hypothetical case), potentially reach an "intelligence explosion," which would (may) _actually_ yield a true, generalized intelligence.

So as a cynic, while I think the intermediary goal of many of these so-called-agi companies is just your usual SaaS automation slop because thats the easiest industry to disrupt and extract money from (and the people at these companies only really know how software works, as opposed to having knowledge of other things like chemistry, biology, etc), I also think that in theory, being a very fast and low cost programming agent is a bit more powerful than you think.

Reply on news.ycombinator.com