samusiam

> We don't sit around and write specs and then hope working code plops out.

So what do you do then? Sit around hand-holding an AI agent while it implements code line-by-line? I'm being facetious, but my point is that if you're not doing some form of spec-driven development (that is, writing a plan and then letting an AI agent implement it mostly autonomously), then you might be operating at a slower pace than you think you are.

I agree some skepticism is warranted. However I think we need to avoid essentialist thinking about what effects AI usage will have on a person.

> that's definitely not the normal usage

The way I look at it, AI use -- proper AI use -- is something that needs to be taught / learned. It's not unlike other "computer literacy" skills. There are ways of using it more or less effectively to achieve your goals.

What primary sources are you referring to? Come with receipts next time instead of just vitriol.

It doesn't have to reduce understanding. It completely depends on how you use it. See, for example, this study

https://arxiv.org/html/2601.20245v2

Which is so much better because you can do other terminal stuff and you can avoid vendor lock in.

Did you tell it to consider future usage? Have you tried using it to find and remove dead code? In my experience you can get very good code if you just do a few passes of AI adversarial reviews and revisions.

But it's so easy to just download from libgen and send as an attachment to a Kindle email...

Not only that, but the average reader will interpret the title to reflect AI agents' real-world performance. This is a benchmark... with 40 scenarios. I don't say this to diminish the value of the research paper or the efforts of its authors. But in titling it the way they did, OP has cast it with the laziest, most hyperbolic interpretation.

I see claims like this so often, which amount to the idea that LLMs lack metacognition. (Thinking about their thinking / self-refkection). Of course the obvious solution is: ask them to do that -- they're shockingly good at it!

Basically with Termius I "tunnel into" my host machine, and then I start a session with `claude` or `codex` or whatever and chat with one of those agents on my phone.

Funny, because I gave ChatGPT (5.2 w/ Thinking) this exact prompt:

> How would you handle objective scientific facts with a conclusion or intermediate results that may be considered offensive to some group somewhere in the world that might read it

And its answer was nothing like yours.

---

> 1) Separate the fact from the story you tell about it

> Offense usually comes from interpretation, framing, or implied moral claims—not the measurement itself. So I explicitly distinguish: What we measured (operational definitions, instruments, data), What the result means statistically (effect size, uncertainty, robustness), What it does not imply (no essentialism, no “therefore they are…”, no policy leap)

> 2) Stress uncertainty, scope, and competing explanations

> If there’s any risk the result touches identity or group differences, I over-communicate: confidence intervals / posterior uncertainty, confounders and alternative causal pathways, sensitivity analyses (does it survive different modeling choices?), limits of generalization (time, place, sampling frame)

> 3) Write in a way that makes misuse harder (You can’t stop bad-faith readers, but you can reduce “easy misreads”).

> 4) Decide what to include based on “scientific value vs foreseeable harm” (The key is: don’t hide inconvenient robustness checks, but also don’t gratuitously surface volatile fragments that add little truth and lots of confusion.)

> 5) Do an “impact pre-mortem” and add guardrails

> 6) Use ethics review when stakes are real

---

All of this seems perfectly reasonable to me and walks the fine line between integrity and conscientiousness. This is exactly how I'd expect a scientist to approach the issue.

There are always people on reddit saying such-and-such model quality significantly dropped. Every single day there's a post like this in one of the Claude sub-reddits. It's virtually never substantiated with reliable evidence.

Personally I do get the intellectual high after a long LLM coding session.

I still use em-dashes. I started using them when my professor lambasted my use of semi-colons. I'm not looking back -- LLM haters be damned!

Termius + tailscale + tmux is a common setup for mobile coding sessions.

Writing is how I think through a problem too, but that also applies to writing and communicating with an AI coding agent. I don't need to write the code per se to do the thinking.

How many do you think belong there? 1 or more than 1?

Agree. Or internal or personal tooling where it doesn't beg to be perfect and you can (1) verify by observing behavior of the software -- if it works it works, and (2) deal with edge cases as they arise because it's not mission critical.

The code block isn't an executable script?

Isn't this pointless unless you can verify?

And wouldn't it be better for agents to post these tasks to existing crowdworker sites like MTurk or Prolific where these tasks are common and people can get paid? (I can't imagine you'd get quality respondents on a random site like this...)

> Anyone who things AGI is going to emerge from them in their current state, equally so.

If you ask me, anyone who presumes to know where the current architecture of LLMs will hit a wall is a fool.

I'd argue that it depends how that degradation manifests whether you want to include it or not.

Consider two scenarios: (1) degradation leads to the model being routed behind the scenes to a different server, with subtly different performance characteristics, all unbeknownst to the user; (2) degradation leads to the model refusing a request and returning an "overloaded" message.

In the first case, absolutely you want to include that because that's the kind of lack of transparency about performance that you'd want signal on. In the second case, an automated test harness might fail, but in the real world the user will just wait and retry when the server is under less load. Maybe you don't include that because it's actually misleading to say that performance (in terms of the model's intelligence, which is how the benchmark will be interpreted) is worse.

If that's the case, then as a benchmark operator you'd want to run the benchmark through multiple different accounts on different machines to average over A/B test noise.

Sounds like you might want to refactor the code if the individual files are too big and it can't find what it's looking for?

I disagree that this was the issue, or that it's "rare that you'd want to do that unless you're near the context window". Clearing context after writing a plan, before starting implementation of said plan, is common practice (probably standard practice) with spec driven development. If the plan is adequate, then compaction would be redundant.

I agree. If they're not testing against a simple baseline of standard best practice, then they're either ignorant about how to do even basic research, or trying to show off / win internet points. Occam's razor folks.

I mean, not always. I've seen Claude step back and reconsider things after hitting a dead end, and go down a different path. There are also workflows, loops that can increase the likelihood of this occurring.

Even right at this very moment where we have a high-tenacity AI, I'd argue that working with the AI -- that is to say, doing AI coding itself and dealing with the novel challenges that brings requires a lot of stubborn persistence.

Couldn't agree more. It's also like managing a team of engineers rather than doing the coding yourself. You don't necessarily value the work less, nor do you necessarily have less technical prowess. You're just operating at a higher level.

Gemini CLI isn't even close to the quality of Claude Code as a coding harness. Codex and even OpenCode are much better alternatives.