diwank

I'd recommend considering applying to the Thiel Fellowship.

https://thielfellowship.org/faq

Compressed it. Thanks for letting me know :)

In my experience, you’ll run out of Opus usage credits way too quickly on the Pro plan

yup but:

1. AIDEV-* is easier to visually distinguish, and

2. it is grep-able by the models, so they can "look around" the codebase in one glance

This isn’t entirely surprising. Language-model “reasoning” is basically the model internally exploring possibilities in token-space. These models are trained on enormous multilingual datasets and optimized purely for next-token prediction, not language purity. When reasoning traces or scratchpads are revealed directly (as OpenAI occasionally does with o-series models or DeepSeek-R1-zero), it’s common to see models slip into code-switching or even random language fragments, simply because it’s more token-efficient in their latent space.

For example, the DeepSeek team explicitly reported this behavior in their R1-zero paper, noting that purely unsupervised reasoning emerges naturally but brings some “language mixing” along. Interestingly, they found a small supervised fine-tuning (SFT) step with language-consistency rewards slightly improved readability, though it came with trade-offs (DeepSeek blog post).

My guess is OpenAI has typically used a smaller summarizer model to sanitize reasoning outputs before display (they mentioned summarization/filtering briefly at Dev Day), but perhaps lately they’ve started relaxing that step, causing more multilingual slips to leak through. It’d be great to get clarity from them directly on whether this is intentional experimentation or just a side-effect.

[1] DeepSeek-R1 paper that talks about poor readability and language mixing in R1-zero’s raw reasoning https://arxiv.org/abs/2501.12948

[2] OpenAI “Detecting misbehavior in frontier reasoning models” — explains use of a separate CoT “summarizer or sanitizer” before showing traces to end-users https://openai.com/index/chain-of-thought-monitoring/

Claude Code is a really very different experience. I’d say give it a shot but Opus 4 is a step change

There are a bunch of different options so it depends on the models you end up liking but the simplest place to start is to get Claude Max $100 tier and use Opus 4 (the others don’t really get you the full experience)

Thank you! aider is a different beast actually, I found its memory/context handling best in class. Somehow though, I ended up liking Claude Code the most because of its TUI but really a matter of personal preference and workflow

Have tried everything under the sun at the moment— broadly just two winners (and both have become my daily drivers for different use cases):

1. Claude Code with Opus 4

2. Cursor with Opus 4 or Gemini 2.5 Pro (Windsurf used to be an option but Anthropic has now cut them out)

3. (Coming up; still playing around) Claude Code’s GitHub Action

The risk, in my experience, is to reach a state where you as the dev is not easily able to understand/update the tests. That becomes very tricky

Yeah absolutely. I’m on their Max sub since last month and I still don’t fully get how many tokens I have etc.

Simplest I’d say is: get the $100/mo Max and then npm install -g @anthropic-ai/claude-code

Mostly just good faith during PR reviews. Plus other than Opus 4 models largely flub it and it shows

that didn't include the testing, that def took a lot longer but at least now my devs don't have an excuse for poorly written tests lol

yeah, the max subscription comes in two tiers, $100 gets you 5x the tokens than pro (which only has sonent) and the $200 gets you 20x. doing the math for tokens is kinda annoying and not straightforward atm. they also have a "hybrid" mode which uses opus until ~20% tokens for opus left then switches to sonnet

yeah, this might change in the future but I also found that since building features has become faster, asking devs to write the tests themselves sort of demands that they take responsibility of the code and the potential bugs

we tried a few different variations but tbh had universally bad results. for example, we use `ward` test runner in our python codebase, and claude sonnet (both 3.7 and 4) keep trying to force-switch it to pytest lol. every. single. time.

maybe we could either try this with opus 4 and hope that cheaper models catch up, or just drink the kool-aid and switch to pytest...

true but this does matter to a lot of enterprise customers that have to obey strict data provenance laws (for instance, there are no gpt-4.1 model endpoints hosted in India and hence, fin-tech companies cannot use those apis)

the cost per token for ~similar performance is dropping by a factor of 2 every 10-11 months at the moment, so I am not sure. that said, I think devs in less expensive parts of the world are actually picking up these tools the fastest (maybe from existential angst? idk)

absolutely! I think of these as new levers in the making, rather rusty, and def can often bite you in the behind but worth learning, and perhaps most importantly to help evolve them into useful tools rather than an excuse to ship sloppy engineering

I appreciate this :)

no, just used chatgpt to bootstrap the research :)

here's the original chat: https://chatgpt.com/share/6844eaae-07d0-8001-a7f7-e532d63bf8...

I also used bits from claude research but apparently if you use claude research, they don't let you create a share link -_-

I completely understand. Just to clarify, when I said it was ~40%, I didn’t mean the content was written by Claude/ChatGPT but that I took its help in deep research and writing the first drafts. The ideas, all of the code examples, the original CLAUDE.md files, the images, citations, etc are all mine.

Really appreciate the kind words! I did not intend the post to be too much about our company, just that it is the codebase I mostly hack on. :)

For sure. Another interesting trick I found to be surprisingly effective is to ask Claude Code to “Look around the codebase, and if something is confusing, or weird/counterintuitive — drop a AIDEV-QUESTION: … comment so I can document that bit of code and/or improve it”. We found some really gnarly things that had been forgotten in the codebase.

Same. I ran out on the $200 one too yesterday. It’s skyrocketed after Opus 4. Nothing else comes close

True but, in my experience, a few major pitfalls that happened:

1. We ran into really bad minefields when we tried to come back to manually edit the generated tests later on. Claude tended to mock everything because it didn’t have context about how we run services, build environments, etc.

2. And this was the worst, all of the devs on the team including me got realllyy lazy with testing. Bugs in production significantly increased.

> - Is there a more elegant way to organize the prompts/specifications for LLMs in a codebase? I feel like CLAUDE.md, SPEC.mds, and AIDEV comments would get messy quickly.

Yeah, the comments do start to pile up. I’m working on a vscode extension that automatically turns them into tiny visual indicators in the gutter instead.

> - What is the definition of "vibe-coding" these days? I thought it refers to the original Karpathy quote, like cowboy mode, where you accept all diffs and hardly look at code. But now it seems that "vibe-coding" is catch-all clickbait for any LLM workflow. (Tbf, this title "shipping real code with Claude" is fine)

Depends on who you ask ig. For me, hasn’t been a panacea, and I’ve often run into issues (3.7 sonnet and codex have had ~60% success for me but Opus 4 is actually v good)

> - Do you obfuscate any code before sending it to someone's LLM?

In this case, all of it was open source to begin with but good point to think about.

I don’t understand why not. I’m not a natural prose writer, but (I felt that) these ideas were worth putting out there.

I posted on HN largely to get feedback.

Why not? This way at least the idea gets out there. Otherwise I’d have never come around to writing this.

I’d say around ~40% me, the ideating, editing, citations, and images are all mine; rest Opus 4 :)

I typically try to also include the original Claude chat’s link in the post but it seems like Claude doesn’t allow sharing chats with deep research used in them.

See this series of posts for example, I have included the link right at the beginning: https://diwank.space/juleps-vision-levels-of-intelligence-pt...

I completely get the critique and I already talked about it earlier: https://news.ycombinator.com/item?id=44213823

Update: here’s an older chatgpt conversation while preparing this: https://chatgpt.com/share/6844eaae-07d0-8001-a7f7-e532d63bf8...