Hacker News Reader

Comment by ttul | Hacker News Reader

If you ask Sol or Claude how much time it will take to implement a plan they just came up with, they usually advise a timeframe in the weeks or months - assuming, I suppose, that human programmers will be building it. And then you ask the model to just "do it" and it takes an hour or two. I always find this entertaining.

View on HN

Funny but shows how it doesn't have a useful world model

It does, assuming "world" is made up of humans (which it is, in its training data). It does however not have any model whatsoever of itself, apart from some superficial "I am a helpful assistant built by $X".

I find that the models have increasingly started throwing out made up amounts of time around, as if they wouldn't be aware that the user is already using Claude Code or Codex.

It's like every time they make a plan, there's something about things taking "a week or two", "month of focused work", or whatever.

This is something that would've been RL'd out a long time ago if it wasn't great for business.

A model isn't "aware" of anything. It's trained on things, and that training doesn't include time estimates for working with AI because that didn't exist yet.

There's pre and post training.

What I was going after with "aware", is that the actual people working at the companies, training the models, are aware that people aren't mostly going to be implementing the plan by hand, if they've already made the plan in Claude Code or Codex. As for a specific Claude / GPT instance, "aware" would definitely be the wrong word choice there, but the instance does have its stats and environment information in its context window, unless you specifically remove it.

Either way: Training does include estimates on working with the model, and adjustments of the model itself based on that. That's literally what RLHF is.

It's straightforward to have a portion in post training that aims specifically at the model being able to give better estimates on how long that model takes to complete a certain type of task.

In all honesty I don't think it's possible to put this into the model because it's not actually sufficiently understood yet by anyone or anything. AI capabilities are changing too fast, and so is the way AI is used (worst case, even including financial limitations that are now increasingly appearing.)

Every part of training a model have the timing data observed + saved on multiple different axis, it's one of the most inherent parts of the training process.

When the post training run is nearing its cutoff point, there's a massive amount of data on how long coding tasks take to complete by that model in the golden format of "task -> time task took to complete", separable to whatever amount of subtasks, in the same format. With the parts from the end of the dataset being useful for evaluating the finished model's capabilites, whether that data is then fed back into another step in post training or not.

Completely separate from even the actual training: If you have a model proactively giving estimates that are an order of magnitude wrong, you can already fix the worst of it as of this moment by just changing the system prompt. It's a dirty fix, but it's the type of fix that has been used by Anthropic and OpenAI since forever when a model is dishing out blatantly wrong outputs.

Not sure if I'm misunderstanding your point?

The misunderstanding might be that I'm thinking about how long it takes a human + an AI to get something done, not an AI by itself. My current understanding is that it's still not generally viable to have LLMs perform architecture/design tasks (it might work in the short run but degrades roughly exponentially in the long run, accruing serious tech debt). Most things I'd consider actual "projects" involve at least some level of design work as well, and that'd need to be included in the estimate.

(And also prompts will need to be refined, etc., unless you have amazing prompt skills it won't immediately deliver what you wanted. Even if it did, the work to put together the AI inputs also needs to be included.)

⇒ I don't think it's easily possible right now to give a time estimate for **the full picture of** a project to be implemented with AI assistance.

Reply on news.ycombinator.com