felix9527

We instrument production services with traces and structured logging. AI agents that touch the same codebase? Most people have zero visibility once the session ends. The scrollback is gone, the only record is whatever the agent chose to summarize about itself.

The study only looks at what lands in the PR. In my experience a single prompt can trigger 20+ tool calls, most of them reads and greps. The final diff is a tiny fraction of what actually happened. Hard to judge quality without seeing the process.

The article treats this as AI being stupid, but it's a design category error. Spotify is applying rules ("don't repeat artists," "mix genres") to a system whose behavior space is unbounded. Rules are a blacklist — there's always a gap.

Real DJs don't follow playlists. They work within constraints — energy, tempo, crowd — and let the set emerge. Better boundaries, not more rules.

Been running Claude Code with hooks recording every tool call for two weeks. ~25k actions across 116 sessions.

The thing that surprised me: 24 tool calls per prompt on average. "Just review everything" is not realistic at that ratio.

What worked for me was the dashcam approach — don't try to prevent, just have an independent record of what the agent actually did. Not the agent's summary, the actual sequence. Caught a few cases where the summary glossed over failed attempts and retries that mattered.

Exactly. The moment an agent touches prod, "we logged it" isn't enough — you need "here's the cryptographic proof of what happened, and you can verify it without trusting us."

Compliance teams (SOC 2, EU AI Act Article 12) will demand this. The nice part is RFC 6962 is already battle-tested at scale — Certificate Transparency processes billions of entries. Same math, different domain.

Sure — it's at https://github.com/PunkGo/punkgo-jack

It hooks into PostToolUse, PreToolUse, SessionStart/End, and UserPromptSubmit. Each event gets submitted to a local kernel that appends it to an RFC 6962 Merkle tree. You can then verify any event with an inclusion proof, or check log integrity between two checkpoints with a consistency proof.

The verify command works offline — just needs the checkpoint and tile hashes, no daemon required. There's also a Go implementation in examples/verify-go/ that independently verifies the same proofs, to show it's not tied to one language.

Would be interesting to explore composing nah's classification decisions with a verifiable log — every allow/deny gets a receipt too.

The core problem this article surfaces is forensic: once the agent session ends, the evidence is whatever the vendor chose to log. Terminal scrollback is lossy, session logs are vendor-controlled, and "undo" only works if you catch it in time.

Certificate Transparency (RFC 6962) solved a structurally identical problem for TLS certificates after the DigiNotar incident. The insight: commit every action to an append-only Merkle tree where any third party can verify inclusion proofs — without trusting the log operator.

Applied to agents: - Inclusion proof: "this specific action was recorded at position N, and the log hasn't been rewritten" - Consistency proof: "between checkpoint A and B, the log only grew — nothing was removed or altered"

This gives you verification-based accountability, not trust-based logging. The difference matters: a signed receipt proves the signer said something happened. A Merkle inclusion proof proves it's part of a complete, append-only sequence — deletions are structurally detectable.

The AI agent ecosystem is having its DigiNotar moment weekly. We have the cryptographic tools to fix it. The question is whether we'll wait for a catastrophic incident to force adoption, like we did with CT.

Interesting approach to the PreToolUse side. I've been building on the other end — PostToolUse hooks that commit every tool call to an append-only Merkle tree (RFC 6962 transparency log style).

  The two concerns are complementary: "nah" answers "should this action be allowed?" while a transparency log answers "can we prove what actually happened, after the fact?"

  For the adversarial cases people are raising (obfuscated commands, indirect execution) — even if a classifier misses something at pre-execution time, an append-only log with inclusion proofs means the action is still
  cryptographically recorded. You can't quietly delete the embarrassing entries later.

  The hooks ecosystem is becoming genuinely useful. PreToolUse for policy enforcement, PostToolUse for audit trail, SessionStart/End for lifecycle tracking. Would be great to see these compose — a guard that also commits
  its allow/deny decisions to a verifiable log.