D-Machine

An important consideration somewhat missing in discussion in this thread: if we don't carefully document AI-assisted coding sessions, how can we ever hope to improve our use of AI coding tools?

This applies both to future AI tools and also experts, and experts instructing novices.

To some degree, the lack of documenting AI sessions is also at the core of much of the skepticism toward the value of AI coding in general: there are so many claims of successes / failures, but only a vanishingly small amount of actual detailed receipts.

Automating the documentation of some aspects of the sessions (skills + prompts, at least) is something both AI skeptics and proponents ought to be able to agree on.

EDIT: Heck, if you also automate documenting the time spent prompting and waiting for answers and/or code-gen, this would also go a long way to providing really concrete evidence for / against the various claims of productivity gains.

Try this argument in any field where the stakes are high (e.g. medical diagnostics) and see how far this gets you.

This is really basic decision-theory stuff, often the cost of an error is far greater than the benefit of correctness.

Yup. Realistically there will always be simple changes that AI can handle completely (docs, comments, and commit message), and other changes where some human input will be hugely valuable.

Until then, it makes sense to automatically include some distillation of the AI generation process, by default, IMO.

Something like "it is important to document core / crucial prompts somewhere" covers it. Whether this should be in git or elsewhere is trickier, but doing vibe-coding without documenting any aspect of the process is a recipe for disaster.

Also, how can we (or future AI models) hope to improve if there is only limited and summary documentation of AI usage?

> well, maybe some people do but most people I know re-write the same file multiple times before committing, or rebase/squash intermediate commits into more useful logical commits

Right, agreed on this, we want a distillation, not documentation of every step.

> For example, having some documentation for the intent behind a given series of commits and any assumptions made can be quite valuable in the future, but every single discussion between any two devs on a project as part of the commit history would be so much noise for very little gain. AI prompts and sessions seem to me to fall into that same bucket.

Yes, documenting every single discussion is a waste / too much to process, but I do think prompts at least are pretty crucial relative to sessions. Prompts basically are the core intentions / motivations (skills aside). It is hard to say whether we really want earlier / later prompts, given how much context changes based on the early prompts, but having no info about prompts or sessions is a definite negative in vibe-coding, where review is weak and good documentation, comments, and commit messages are only weakly incentivized.

> Some of these things might be useful to preserve some of the time either in the commit history or along side it

Right, along side is fine to me as well. Just something has to make up for the fact that vibe-coding only appears faster (currently) if you ignore the fact it is weakly-reviewed and almost certainly incurring technical debt. Documenting some basic aspects of the vibe-coding process is the most basic and easy way to reduce these long-term costs.

EDIT: Also, as I said, information about the prompts quickly reveals competence / incompetence, and is crucial for management / business in hiring, promotions, managing token budgets, etc. Oh, and of course, one of the main purposes of code review was to teach. Now, that teaching has to shift toward teaching better prompting and AI use. That gets a lot harder with no documentation of the session!

I mean, sure, a good, detailed commit message is perfectly fine to me in place of the prompts / a session distillation. But I am not holding my breath for vibe-coders to properly review their code and make such a commit message. But, if they, do, great! No need for prompt / session details.

> You have the source code though. That is the "reproducibility" bit you need.

I am talking about reproducing the (perhaps erroneous) logic or thinking or motivations in cases of bugs, not reproducing outputs perfectly. As you said, current LLM models are non-deterministic, so we can't have perfect reproducibility based on the prompts, but, when trying to fix a bug, having the basic prompts we can see if we run into similar issues given a bad prompt. This gives us information about whether the bad / bugged code was just a random spasm, or something reflecting bad / missing logic in the prompt.

> Is it important to know that when the foo file was refactored the developer chose to do it by hand vs letting the IDE do it with an auto-refactor command vs just doing a simple find and replace? Maybe it is for code review purposes, but for "reproducibility" I don't think it is.

I am really using "reproducibility" more abstractly here, and don't mean perfect reproducibility of the same code. I.e. consider this situation: "A developer said AI wrote this code according to these specs and prompt, which, according to all reviewers, shouldn't produce the errors and bad code we are seeing. Let's see if we can indeed reproduce similar code given their specs and prompt". The less evidence we have of the specifics of a session, the less reproducible their generated code is, in this sense.

First N prompts is a good / practical heuristic for something worth storing (whether N = 1 or greater).

I think this is the right analogy, contrary to some other very poor ones in this thread. Yes, it is rare to really look at commit messages, but it can be invaluable in some cases.

With vibe-coding, you risk having no documentation at all for the reasoning (AI comments and tests can be degenerate / useless), but the prompts, at bare minimum, reveal something about the reasoning / motivation.

Whether this needs to be in git or not is a side issue, but there is benefit to having this available.

If you also ensure the AI writes relevant (and correct) docs, and also code comments and commit message, then I agree there is not much need for extra info, e.g. prompts / session distillation. I am not sure that that is the case currently (though we might be getting there soon at least in some cases).

It should be a distillation of the session and/or the prompts, at bare minimum. No, it should not include e.g. research-type questions, but it should include prompts that the user wrote after reading the answers to those research-type questions, and perhaps some distillation of the links / references surfaced during the research.

Prompts probably should be distilled / summarized, especially if they are research-based prompts, but code-gen prompts should probably be saved verbatim.

Reproducibility is a thing, and though perfect reproducibility isn't desirable, something needs to make up for the fact that vibe-coding is highly inscrutable and hard to review. Making the summary of the session too vague / distilled makes it hard to iterate and improve when / if some bad prompts / assumptions are not documented in any way.

Obviously yes, at least if not the prompts in the session, some simple / automated distillation of those prompts. Code generated by AI is already clearly not going to be reviewed as carefully as code produced by humans, and intentions / assumptions will only be documented in AI-generated comments to some limited degree, completely contingent on the prompt(s).

Otherwise, when fixing a bug, you just risk starting from scratch and wasting time using the same prompts and/or assumptions that led to the issue in the first place.

Much of the reason code review was/is worth the time is because it can teach people to improve, and prevent future mistakes. Code review is not really about "correctness", beyond basic issues, because subtle logic errors are in general very hard to spot; that is covered by testing (or, unfortunately, deployment surprises).

With AI, at least as it is currently implemented, there is no learning, as such, so this removes much of the value of code review. But, if the goal is to prevent future mistakes, having some info about the prompts that led to the code at least brings some value back to the review process.

EDIT: Also, from a business standpoint, you still need to select for competent/incompetent prompters/AI users. It is hard to do so when you have no evidence of what the session looked like. Also, how can you teach juniors to improve their vibe-coding if you can't see anything about their sessions?

Nothing other to say than I appreciate you sharing these explicit details and insights here.

Insofar as they are testing very broken and simple "skills" (as evidenced by comments in this thread), and still find net significant net positives in some cases, negatives in others, and huge variability overall, I actually think this is a fun paper providing support that, at least if you count using skills here, you can, in general, very objectively and quantifiably be holding it right/wrong.

I.e. your use of skills resulting in differences of up to ~52 percentage points (or negative percentage points) in improvements (or degradations) in your percentage pass rate is a decent first-pass measure of the importance of skills here.

Everything about frontier AI companies relies on secrecy. No specific details about architectures, dispatching between different backbones, training details such as data acquisition, timelines, sources, amounts and/or costs, or almost anything that would allow anyone to replicate even the most basic aspects of anything they are doing. What is the cost of one more secret, in this scenario?

> What does out of distribution even mean?

There are in fact ways to directly quantify this, if you are training e.g. a self-supervised anomaly-detection model.

Even with modern models not trained in that manner, looking at e.g. cosine distances of embeddings of "novel" outputs could conceivably provide objective evidence for "out-of-distribution" results. Generally, the embeddings of out-of-distribution outputs will have a large cosine (or even Euclidean) distance from the typical embedding(s). Just, most "out-of-distribution" outputs will be nonsense / junk, so, searching for weird outputs isn't really helpful, in general, if your goal is useful creativity.

For sure we know modern LLMs and AIs are not constrained by anything particularly close to simple linear combinations, by virtue of their depth and non-linear activation functions.

But yes, it is not yet clear to what degree there can be (non-linear) extrapolation in the learned semantic spaces here.

The tricky part is that LLMs aren't just spewing outputs from the distribution (or "near" learned manifolds), but also extrapolating / interpolating (depending on how much you care about the semantics of these terms https://arxiv.org/abs/2110.09485).

There are genuine creative insights that come from connecting two known semantic spaces in a way that wasn't obvious before (e.g, novel isomorphism). It is very conceivable that LLMs could make this kind of connection, but we haven't really seen a dramatic form of this yet. This kind of connection can lead to deep, non-trivial insights, but whether or not it is "out-of-distribution" is harder to answer in this case.

It is especially glaring in this case because, when queried, it is clear that far too many of the most zealous proponents don't even understand the simplest basics of how these models actually work (e.g. tokenization, positional or other encoding schemes, linear algebra, pre-training, basic input/output shaping/dimensions, recursive application, training data sources, etc).

There are simple limitations that follow from these basic facts (or which follow with e.g. extreme but not 100% certainty), such that many experts openly state that e.g. LLMs have serious limitations, but, still, despite all this, you get some very extreme claims about capabilities, from supporters, that are extremely hard to reconcile with these basic and indisputable facts.

That, and the massive investment and financial incentives means that the counter-reaction is really quite rational (but still potentially unwarranted, in some/many practical cases).

Even more generally than verification, just being tied to a loss function that represent something we actually care about. E.g. compiler and test errors, LEAN verification in Aristotle, basic physics energy configs in AlphaFold, or win conditions in e.g. RL, such as in AlphaGo.

RLHF is an attempt to push LLMs pre-trained with a dopey reconstruction loss toward something we actually care about: imagine if we could find a pre-training criterion that actually cared about truth and/or plausibility in the first place!

A lot of AI worship is midwits gazing in awe at the mediocre accomplishments of the 130+ IQ.

(Still sane to be scared for the future).

> It's interesting to me that whenever a new breakthrough in AI use comes up, there's always a flood of people who come in to handwave away why this isn't actually a win for LLMs.

>> OP's original comment is something that is actually happening in a bunch of comments on this very thread

OPs original comment was obviously a general claim not tied to responses to this thread. As usual, you fail to understand even the basics of what you are talking about.

Downvotes eventually turn all online communities into echo chambers, definitely. It is only a matter of time for HN, and you can see it accelerating in the past 1-2 years (though mostly on AI stuff, and mostly in downvote behaviour - it still remains surprisingly resilient overall).

> Your framing is weirdly disingenuous

I am not surprised that you can't understand that the quote I am making is obviously parodying the OP as disingenuous. Given our previous interactions (https://news.ycombinator.com/item?id=46938446), it is clear you don't understand much things about AI and/or LLMs, or, perhaps, basic communication, at all.

"They're moving the goalposts" is increasingly the autistic shrieking of someone with no serious argument or connection to reality whatsoever.

No one cares about how "AGI" or whatever the fuck term or internet-argument goalpost you cared about X months ago was. Everyone cares about what current tech can do NOW, and under what conditions, and when it fails catastrophically. That is all that matters.

So, refining the conditions of an LLM win (or loss) is all that matters (not who wins or loses depending on some particular / historical refinement). Complaining that some people see some recent result as a loss (or win) is just completely failing to understand the actual game being played / what really matters here.

"It's interesting to me that whenever some new result in AI use comes up, there's always a flood of people who come in to gesticulate wildly that that the sky is falling and AGI is imminent. Like with the recent solutions GPT 5.2 has been able to find for Erdos problems, even though in almost all cases such solutions rely on poorly-known past publications, or significant expert user guidance and essential tools like Aristotle, which do non-AI formal verification - many users here (even in this very thread!) think they know more about this than Fields medalist Terence Tao, who maintains this list showing that, yes, though these are not interesting proofs to most modern mathematicians, LLMs are a major factor in a tiny minority of these mostly-not-very-interesting proofs: https://github.com/teorth/erdosproblems/wiki/AI-contribution..."

The thing about spin and AI hype (besides being trivially easy to write) is that is isn't even trying to be objective. It would help if a lot of these articles would more carefully lay out what is actually surprising, and what is not, given current tech and knowledge.

Only a fool would think we aren't potentially on the verge of something truly revolutionary here. But only a fool would also be certain that the revolution has already happened, or that e.g. AGI is necessarily imminent.

The reason HN has value is because you can actually see some specifics of the matter discussed, and, if you are lucky, an expert even might join in to qualify everything. But pointing out "how interesting that there are extremes to this" is just engagement bait.

It can be reasonable to be skeptical that advances on benchmarks may be only weakly or even negatively correlated with advances on real-world tasks. I.e. a huge jump on benchmarks might not be perceptible to 99% of users doing 99% of tasks, or some users might even note degradation on specific tasks. This is especially the case when there is some reason to believe most benchmarks are being gamed.

Real-world use is what matters, in the end. I'd be surprised if a change this large doesn't translate to something noticeable in general, but the skepticism is not unreasonable here.

> Because the gains from spending time improving the model overall outweigh the gains from spending time individually training on benchmarks.

This may not be the case if you just e.g. roll the benchmarks into the general training data, or make running on the benchmarks just another part of the testing pipeline. I.e. improving the model generally and benchmaxing could very conceivably just both be done at the same time, it needn't be one or the other.

I think the right take away is to ignore the specific percentages reported on these tests (they are almost certainly inflated / biased) and always assume cheating is going on. What matters is that (1) the most serious tests aren't saturated, and (2) scores are improving. I.e. even if there is cheating, we can presume this was always the case, and since models couldn't do as well before even when cheating, these are still real improvements.

And obviously what actually matters is performance on real-world tasks.

> which is why only "ARC-AGI Certified" results using a secret problem set really matter. The 84.6% is certified and that's a pretty big deal.

So, I'd agree if this was on the true fully private set, but Google themselves says they test on only the semi-private:

> ARC-AGI-2 results are sourced from the ARC Prize website and are ARC Prize Verified. The set reported is v2, semi-private (https://storage.googleapis.com/deepmind-media/gemini/gemini_...)

This also seems to contradict what ARC-AGI claims about what "Verified" means on their site.

> How Verified Scores Work: Official Verification: Only scores evaluated on our hidden test set through our official verification process will be recognized as verified performance scores on ARC-AGI (https://arcprize.org/blog/arc-prize-verified-program)

So, which is it? IMO you can trivially train / benchmax on the semi-private data, because it is still basically just public, you just have to jump through some hoops to get access. This is clearly an advance, but it seems to me reasonable to conclude this could be driven by some amount of benchmaxing.

EDIT: Hmm, okay, it seems their policy and wording is a bit contradictory. They do say (https://arcprize.org/policy):

"To uphold this trust, we follow strict confidentiality agreements. [...] We will work closely with model providers to ensure that no data from the Semi-Private Evaluation set is retained. This includes collaborating on best practices to prevent unintended data persistence. Our goal is to minimize any risk of data leakage while maintaining the integrity of our evaluation process."

But it surely is still trivial to just make a local copy of each question served from the API, without this being detected. It would violate the contract, but there are strong incentives to do this, so I guess is just comes down to how much one trusts the model providers here. I wouldn't trust them, given e.g. https://www.theverge.com/meta/645012/meta-llama-4-maverick-b.... It is just too easy to cheat without being caught here.

We CAN categorically say that no such token or cluster of tokens exists, because we know how LLMs and tokenizers work.

Current LLM implementations cannot delete output text, i.e. they cannot remove text from their context window. The recursive application is such that outputs are always expanding what is in the window, so there is no backtracking like humans can do, i.e. "this text was bad, ignore it and remove from context". That's part of why we got crazy loops / spirals like we did with the "show me the seahorse emoji" prompts.

Backtracking needs more than just a special token or cluster of tokens, but also for the LLM behaviour to be modified when it sees that token or token cluster. This must be manually coded in, it cannot be learned.