Having functional behavior and implementation details separate can be really useful, even though it is typically a pain to keep them in sync.
I'm not saying that you can just take any code out there and call it your spec.
I even pointed out in my last paragraph that code not written as a spec is garbage for LLM driven development - precisely because you end up with unintended implementation details becoming your spec.
There are a lot of ways to address this, but ultimately it is down to how you structure your code, where you place your comments, and what you write into your agents.md in the modules and what kind of QA agents you configure to go through your code before reviewing the code changes/spec adjustments yourself
A more heavy handed approach could also be domain driven design, where your domain/core package becomes the spec, and the unspecified parts get extracted to less specified modules with less explicit structuring.