Hacker News Reader

> I automate nearly all my tests with AI

How exactly? Do you tell the agent "please write a test for this" or do you also feed it some form of spec to describe what the tested thing is expected to do? And do these tests ever fail?

Asking because the first option essentially just sets the bugs in stone.

Wouldn't it make sense to do it the other way around? You write the test, let the AI generate the code? The test essentially represents the spec and if the AI produces sth which passes all your tests but is still not what you want, then you have a test hole.

I'm not saying my approach is correct, keep that in mind.

I care more about the code than the tests. Tests are verification of my work. And yes, there is a risk of AI "navigating around" bugs, but I found that a lot of the time AI will actually spot a bug and suggest a fix. I also review each line to look for improvements.

Edit: to answer your question, I will typically ask it to test a specific test case or few test cases. Very rarely will I ask it to "add tests everywhere". Yes, these tests frequently fail and the agent will fix on 2nd+ iteration after it runs the tests.

One more thing to add is that a lot of the time agent will add a "dummy" test. I don't really accept those for coverage's sake.

Thanks for your responses!

A follow-up:

> I care more about the code than the tests.

Why is that? Your (product) code has tests. Your test (code) doesn't. So I often find that I need to pay at least as much attention to my tests to ensure quality.

I think you are correct in your assessment. Both are important. If you're gonna have garbage code tests, you're gonna have garbage quality.

I find tests easier to write. Your function(s) may be hundred lines long, but the test is usually setup, run, assert.

I don't have much experience beyond writing unit/integration tests, but individual test cases seem to be simpler than the code they test (linear, no branches).

This is why the quality of my code has improved since using AI.

I can iterate on entire approaches in the same amount of time it would have taken to explore a single concept before.

But AI is an amplifier of human intent- I want a code base that’s maintainable, scalable, etc., and that’s a different than YOLO vibe coding. Vibe engineering, maybe.

My core uses are 100% racing the model in yolo mode to find a bug. I win most of the time but occasionally it surprises me.

Then also switching arch approaches quickly when i find some code strategies that are not correctly ergonomic. Splitting of behaviors and other refactors are much lower cost now.

> No surprise, really. You can use AI to explore new horizons or propose an initial sketch, but for anything larger than small changes - you must do a rewrite. Not just a review. An actual rewrite. AI can do well adding a function, but you can't vibe code an app and get smarter.

Sometimes I wonder if people who make statements like this have ever actually casually browsed Twitter or reddit or even attempted a "large" application themselves with SOTA models.

You can definitely vibecode an app, but that doesn't mean that you can necessarily "get smarter"!

An example: I vibecoded myself a Toggl Track clone yesterday - it works amazingly but if I had to rewrite e.g. the PDF generation code by myself I wouldn't have a clue!

That's what I meant, it's either, or. Vibe coding definitely has a place for simple utilities or "in-house" tools that solve one problem. You can't vide code and learn (if you do, then it's not vibe coding as I define it).

Did I say that you can't vibe code an app? I browse reddit and have seen the same apps as you did, I also vibe code myself every now and then and know what happens when you let it loose.