Hacker News Reader

Comment by codingisfreedom | Hacker News Reader

I’ve asked Astra to build me an app for a prototype I created quickly using Sonnet.

It’s been 2 days and it made no real progress on the actual app. It created docs, scripts, workflows, and it’s doing a bunch of reviewing on every PR.

I told it that I just need an MVP.

I’m pretty sure an average senior engineer would have finished that task much quicker, and guaranteed with more readable, higher-quality code. Meanwhile, I think I’ve easily crossed 100k tokens so far on nothing.

Funny world we’re living in that this is “SOTA” and “AGI”.

I’m genuinely curious what these OAI and A/ engineers are working on that they praise these models so much. I did not see any improvement since Opus 4.5.

Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.

View on HN

I don't know how to make apps or evaluate code, but with astra I having been making an iOS app on my own for the first time and it's going great. my app is not terribly complex but requires using bluetooth and other intricacies which I thought would be tough. but it's going really well. I'm not asking it to one-shot it though, I'm going feature by feature, testing and building up.

yes, at first it would run simulator tests on all font sizes but it stopped after I asked it not to do that until UI review

maybe sol would have done the same thing, idk. but I find the whole process to be really nice with astra. I use it on high unless it says something is impossible then i go max and ask it to find alternatives (happened once)

I've been making a macos app with opus 4.8-5 and at first it was great, everything materialized in a week, but when I started tuning stuff and fixing performance problems I have spent a very frustrating month refactoring code where I had to constantly catch llm red-handed and explain and sometimes push obvious ways how to make things work properly (a general knowledge from a completely different stack). In the process CLAUDE.md and memory grew exponentially explaining what it should and what it should never do.

were you letting it run for hours like the OP, or doing short tasks and reviewing/testing each one? every once in a while I also ask it to consolidate/summarize docs and stuff like that. we'll see what happens in a week though

I have started with generating very detailed "feature" spec and going over it many times until it looked good to me. Then I made it write an "architecture doc" and plan how to implement all which was about 15 parts. Then I was making it implement one part and then tested it and made it fix 20 things and then again, and consolidate docs too, so after each part it looked and worked well enough.

>Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.

If a person, or team of people, can build a demo quickly then it's good odds that they can build the real version (though, famously, not a guarantee). However, it turns out that a machine that can spit out 100 demos of whatever can't actually build the real thing.

Similarly, a chess engine rated to 1000 Elo doesn't play like a 1000 rated human being. The mistakes that each make to reach the equivalent level are different in size, frequency and kind. The thing that makes a human reach a good demo is very close to the skillset to reach the finished article. This isn't so for LLMs but we have yet to update our priors.

This resonated with me. "Developing ideas and artifacts using AI breaks our normal intuitions along many meaningful axes and we've yet to update" is a really clean idea.

For me it also produces totally overengineered tests that are tightly coupled to the implementation. For example testing existence of css classes (in a template based go prooject ...) instead of behaviour.

Can you recommend any model that doesn't do this?

Not GP, but IME it's not fixable by model selection, but being zealous about guiding output and vision, and pushing back on all the bad habits LLM in general has (eg verbose output as a band-aid for emergent intelligence). As soon as something is introduced into your codebase, it will continue being picked up into context until you remove it and any reference to it from any potential context entrypoint. If you don't any model will keep venturing down wrong/bad paths.

sadly not. I just run circles trying to remediate it after the fact

> I’m pretty sure an average senior engineer would have finished that task much quicker [...]

Maybe the lesson here is to not send a staff engineer in these cases ;)

Sonnet is not SOTA. Try with Fable or Astra.

>I think I’ve easily crossed 100k tokens so far on nothing

100k tokens? Is it just me or is that very low for an app build?

100k output is a good amount. Probably read and cache are way more, to the millions.

equals roughly 10k loc. If output is code only without rewrites.

Reply on news.ycombinator.com