Comment by raghavtoshniwal | Hacker News Reader

Comment by raghavtoshniwal | Hacker News Reader

Parent Full thread

raghavtoshniwal

The article you linked is from Sept '24 and points to the ARC-AGI test as "evidence" that we're not getting close.

We're in Feb '25. ARC-AGI (at least the version they're referencing) already has been solved by AI at above average human level.

>everything points to incremental progress with signs of diminishing returns.

Seems like everything in just Dec '24/ Jan '25 points the other way. These models are already helping PhDs in novel research, they're already getting super human at coding (yes yes, they're not perfect and I'm sure someone on HN has this weird coding job that AI can't replace yet and they're very excited to shit over AI), but they've already replaced a lot of real software dev jobs.

Also aren't you contradicting yourself?

> everything points to incremental progress with signs of diminishing returns

> corporation replacing employees with AI

If we have incremental progress, how are corporations going to replace employees with AI?

> getting super human at coding

They're getting super-human at _competitive coding_, which is essentially identifying and writing algorithms. They _are not_ good at general coding, as demonstrated by their subpar scores at benchmarks like SWE-bench, and even those aren't particularly representative of what a real coding job is.

>subpar scores at benchmarks like SWE-bench

The last few models have remarkably improved on SWE-bench too. o3 scores 73%, this number was in the low teens 16 months ago. Willing to wager that SWE benchmark gets saturated before the end of 2025.

> aren't particularly representative of what a real coding job

I don't know about that, large swath of "real world" coding is writing plumbing and UIs for CRUD apps, they're getting really good at that as well. Anecdotally, engineers I know have gotten insanely productive with tools like Cursor.

to get the score they had on ARC AGI, they had to fine-tune o3 on ARC AGI. That is hardly a sign of general intelligence or emergent capability.

PhD novel research ? What is the novel research discovered by an LLM ever since the emergence of ChatGPT ? None. Despite all the knowledge these models accumulate in their weights they haven't been able to connect the dots and discover a lot of things humans haven't discovered, autonomously.

Replace which software engineering job ? They are useful, sure; good at benchmarks, yes; but not a drop in replacement of any software engineer.

Reply on news.ycombinator.com