Comment by hetspookjee | Hacker News Reader

Comment by hetspookjee | Hacker News Reader

Over the last 2 weeks (evenings only) I've spend a lot of time crafting the "perfect prompt" for claude code to one shot the project. I ended up with a rather small CLAUDE.md file that references 8 other MD files, ranging from project_architecture, models_spec, build_sequence, test_hierarchy, test_scenarios, and some other files.

It is a project for model based governance of Databricks Unity Catalog, with which I do have quite a bit of experience, but none of the tooling feels flexible enough.

Eventually I ended up with 3 different subagents that supported in the development of the actual planning files; a Databricks expert, a Pydantic expert, and a prompt expert.

The improvement on the markdown files was rather significant with the aid of these. Ranging from old pydantic versions and inconsistencies, to me having some misconceptions about unity catalog as well.

Yesterday eve I gave it a run and it ran for about 2 hours with me only approving some tool usage, and after that most of the tools + tests were done.

This approach is so different than I how used to do it, but I really do see a future in detailed technical writing and ensuring we're all on the same page. In a way I found it more productive than going into the code itself. A downside I found is that with code reading and working on it I really zone in. With a bunch of markdown docs I find it harder to stay focused.

Curious times!

I feel we're developing something like what made Test-Driven Development so strong: TTD forced you to sit down and design your system first, rather than making it all up on the fly. In the past we mapped the system while we were building the code for it.

This kind of AI-driven development feels very similar to that. By forcing you to sit down and map the territory you're planning to build in, the coding itself becomes secondary, just boilerplate to implement the design decision you've made. And AI is great at boilerplate!

That is exactly what this felt like indeed! I found a lot of interest in both refining the test strategy and test decisions, but when it started implementing some core functions were in fact lost in the process. This rather leaky memory still suprises me every now and then. Especially 'undoing' things is a big challenge as the (do not) kind of route versus the (do) route is so much more confusing for the LLM, it seems.

I feel TDD ended up fizzling out quite a bit in the industry, with some evangelists later admitting they'd taken to often writing the code first, then the tests.

To me it's always felt like waterfall in disguise and just didn't fit how I make programs. I feel it's just not a good way to build a complex system with unknown unknowns.

That the AI design process seems to rely on this same pattern feels off to me, and shows a weakness of developing this way.

It might not matter, admittedly. It could be that the flexibility of having the AI rearchitect a significant chunk of code on the fly works as a replacement to the flexibility of designing as you go.

TDD fizzled because not enough emphasis was put on writing high level tests which matched user stories and too much emphasis was put on it as a tool of design.

No, TDD failed because it assumed you could design a perfect systems before implementation.

It’s a totally waste of time to do TDD to only find out you made a bad design choice or discovered a conflicting problem.

What makes you think TDD assumes that? It sounds like the complete opposite of what TDD is about.

This is precisely the problem I alluded to which is solved by writing higher level tests with TDD that make fewer assumptions about your design.

TDD ought to let you make a bad design decision and then refactoring it while keeping the test as is.

Roughly how many test on average would you write before beginning implementation?

“TDD failed because it assumed you could design a perfect systems before implementation”

TDD states the opposite.

TDD is very hard to do right and takes a lot of discipline. If i hadn’t worked for a company that did 100% XP, i would not have either believed it could be so effective. (Best Enterprise software i’ve ever seen and written)

In a way, it is funny. You can practise XP with your AI as pair.

"Extreme programming" methodology said you should not do TDD if you don't already know how to implement the code. In that case you should instead experiment until you know, and then throw away the experiments and write the code test-first.

Maybe it should be done that way with AI: experiment with AI if you need to, then write a plan with AI, then let the AI do the implementation.

thats a great way to put it. the LLMs can't design things, thats way too above their capabilities. they can pretend to design things and even fool people, but they're jsut regurgitating other designs from their training data (and for a todo app, thats enough). but it we do the design for them, they're really really good at putting meat on that skeleton

>they're jsut regurgitating other designs from their training data

I'm deep into the crank/lone genius territory with my BitGrid project, doing things with code that nobody else would be silly enough to waste time on. If it's just copy/pasting code from some latent space, I have no idea where it's getting it.

When's the last time you wrote code to directly handle all the bits to multiply two floating point numbers? I've never done it.

> "TTD forced you to sit down and design your system first, rather than making it all up on the fly"

It's interesting because I remember having discussions with a colleague who was a fervent proponent of TDD where he said that with that approach you "just let the tests drive you" and "don't need to sit down and design your system first" (which I found a terrible idea).

Yeah, your colleagues opinion is kinda inline with how it was usually pitched - at least around 2010 when I learned about it to a degree that it wasn't just a fuzzy concept to me.

One of the goals I remember was to think from the outside to the inside: you first create a test which asserts your most outter API as a unit test with inputs as you want to use it.

Now you keep adding functionality until it passes, creating new tests whenever you make a new boundary/function/API

This supposedly makes it easier to design it well, because you don't have to keep everything in scope and instead only tackle one layer at a time - like an onion from the outside to the core. And you always design the APIs as they make sense, not as is technically easiest because you can just define it however you want, and then think about implementing it.

E.g. https://www.infoq.com/presentations/tdd-ten-years-later/

Mmmm… read the book on Extreme Programming.

The idea is that you let the code drive the system and do not optimize prematurely. Sometimes developers design parts that are not needed, and often not in the first phase.

It is a way to evolve a system unbiased. Of course there is a trade-off. If the refactoring/change is very expensive, like a database schema change in production, it is good to spent a lot of upfront design. Takes experience to see where you can relax and where you need to be careful.

Test-driven and prompt-driven development aside, I never understood why people (and groups) spend many hours (or 1000s, or 10000s of hours) building things when they don't really know what they're building.

(I've certainly seen it done though, with predicable result.)

Do you mean people that build something without a plan? Some people have an idea (or vision) but not a plan and they figure things out along the way. Other want to / need to plan everything ahead of time first.

In my anecdotal case: I behave like the former in some cases (crafting) and the latter in others (travel planning)

I wouldn't say one way is always better than the other.

Most people do not spend thousands of hours building something "not knowing what they're building."

On the contrary, in my experience it's much more important to "play" with a concept and see it working. Too many engineers think they're going to architect a perfect solution without ever getting code on the page.

A slapdash prototype is worth the weight of 100 tests and arch diagrams.

Note: I'm not saying the latter is not important. My comment is, it's ok (and encouraged) to do potentially throwaway work to understand the domain better.

> Most people do not spend thousands of hours building something "not knowing what they're building."

They sure do in my experience.

> On the contrary, in my experience it's much more important to "play" with a concept and see it working...

I agree with all that. That's the point: figure out what you're trying to do before building it. Of course you will not know everything up-front, and of course you would try things out to learn and progress, and, for anything that it's tiny, of course it makes sense to do this iteratively, working from the most pressing/important/risky points earlier.

Or, at least, it seems obvious to me.

This is what Naur, 1985, Programming as Theory Building is about!

https://pages.cs.wisc.edu/~remzi/Naur.pdf

The act of programming is building the theory of what the program does, so that you acquire new knowledge of doing things. It's not just text production.

>"[...] not any particular knowledge of facts, but the ability to do certain things, such as to make and appreciate jokes, to talk grammatically, or to fish."

Which is why re-building a program from scratch is so tempting: you've figured out the theory as you went along, now you can build the real thing.

Long after we are all gone and the scrum masters are a barely remembered historical curiosity, there shall remain, humble and eternal, the waterfall model.

A waterfall, frozen, in time?

Well waterfall is how we built the old world. Piece by piece, module by module, roads, bridges, buildings, boats.

I got intrigued by your comment, I couldn't wrap my head about a process just changing. Got AI to throw out this table, but I think its of interest:

Waterfall ~1970, Agile ~2001, Continuous (DevOps) ~2015, Autonomous Dev ~2030, Self-Evolving Systems ~2040, Goal-Directed Ecosystems ~2050+

What do you think about “goal-directed ecosystems” mapping to Mulder’s Collaborative Agent Maturity Model (CAMM)?

terribly interesting! Thank you

~1985 Iterative Development.

That waterfall was what everyone did before agile is a myth. AI produces the recent popular viewpoint as truth whether it is true or not.

That is exactly my issue. I am more districted while being more productive. It feels just wrong, but works for now. In the long run, I need to find a solution for this. What works best for now, is to let multiple agents run on multiple repos of the same project solving different tasks. This way, I stay somewhat focused, since I constantly need to approve things. Just like a Projekt Manager with a big team... Indeed curious times.

I agree I think this is the way

That's pretty novel. What framework is actually running the agents in your experiment?

Its just the auto generated sub-agents from Claude Code https://docs.anthropic.com/en/docs/claude-code/sub-agents

I plan to do a more detailed write down sometime next week or the week after when I've "finished" my 100% vibe coded website.

These days, I record product details, user journey, etc. with voice, and kick off the product technical details documentation process. Minimal CLAUDE.md. GitHub based workflow for software development process. I am struggling with generating good CI workflows, on it.

Here is my playbook: https://nocodo.com/playbook/

Reply on news.ycombinator.com