Hacker News Reader

Comment by achou | Hacker News Reader

Y'all are sleeping on custom lint rules.

Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you. Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that explains entire architectural principles, concurrency rules, etc. Stuff that is very tailored to your codebase and are far more precise than a generic compiler/lint error.

View on HN

I have examples of this here for eslint

https://github.com/shepherdjerred/scout-for-lol/tree/main/es...

Ex here to detect duplicated code: https://github.com/shepherdjerred/scout-for-lol/blob/main/es...

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it".

To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if "is variable XYZ unused?" or "does variable X oveshadow Y" or other more "esoteric" lint rules will not catch it. Likelihood is high that the LLM just hallucinated some false positive lint rule anyways giving you a false sense of security.

> static code analysis which by definition does not find runtime bugs

I'm not sure if there's some subtlety of language here, but from my experience of javascript linting, it can often prevent runtime problems caused by things like variable scoping, unhandled exceptions in promises, misuse of functions etc.

I've also caught security issues in Java with static analysis.

The usefulness of using static code analysis (strict type systems, linting) versus not using static code analysis is out of the question. Specifically JavaScript which does not have a strict type system benefits greatly from using static code analysis.

But the author claims that you can catch runtime bugs by letting the LLM create custom lint rules, which is hyperbole at least and wrong at most and giving developers a false sense of security at worst.

> But the author claims that you can catch runtime bugs

I think you misinterpreted OP:

Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it

Key word is prevent.

Catch or prevent - linting only covers a tiny (depending on programming language sometimes more sometimes less) subset of runtime problems. The whole back pressure discussion feels like AI coders found out about type systems and lint rules - but it doesn’t resolve the type problems we get in agentic coding. The only „agent“ responsible for code correctness (and thus adherence to feature specification) is the human instructing the agent, a better compiler or lint rule will not prevent massive logic bugs LLMs tend to create like tests testing functions that have been created by the LLM for the test to make it pass, broken logic flows, missing DI, recreating existing logic, creating useless code that’s not being used anywhere yet pollutes context windows - all the problems LLM based „vibe“ coding „shines“ with once you work on a sufficiently long running project.

Why do I care so much about this? Because the „I feel left behind“ crowd is being gaslighted by comments like the OPs.

Overall strict type systems and static code analysis have always been good for programming, and I‘m glad vibe coders are finding out about this as well - it just doesn’t fix the lack of intelligence LLMs have nor the responsibility of programmers to understand and improve the generated stochastic token output

OP isn't claiming all runtime bugs can be prevented with static lints suggested by LLMs but, if at least some can, I don't see how your comment is contributing. Yet another case of "your suggestion isn't perfect so I'll dismiss it" in Hacker News.

Why is this such a common occurrence here? Does this fallacy have a name?

EDIT: seems to be https://en.wikipedia.org/wiki/Nirvana_fallacy

Well, if you haven't noticed, LLM topics receive a particularly hostile reaction on HN.

My LLM has theorized that its success at answering trivia questions has left some people feeling threatened.

Static analysis certainly can find runtime bugs

Clever! Sharing my lightning test of this approach.

Context - I have a 200k+ LOC Python+React hobby project with a directory full of project-specific "guidelines for doing a good job" agent rules + skills.

Of course, agent rules are often ignored in whole or in part. So in practice those rules are often triggered in a review step pre-commit as a failsafe, rather than pulled in as context when the agent initially drafts the work.

I've only played for a few minutes, but converting some of these to custom lint rules looks quite promising!

Things like using my project's wrappers instead of direct calls to libs, preferences for logging/observability/testing, indicators of failure to follow optimistic update patterns, double-checking that frontend interface to specific capabilities are correctly guarded by owner/SKU access control…

Lots of use cases that aren't hard for an agent to accurately fix if pointed at directly, and now that pointing can happen inline to the agent work loop without intervention through normal lint cleanup, occurring earlier in the process (and faster) than is caught by tests. This doesn't replace testing or other best practices. It feels like an additive layer that speeds up agent iteration and improves implementation consistency.

Thanks for the tip!

Yeah, I have written multiple almost completely-vibecoded linters since Claude Code came out, and they provide very high value.

It’s kind of a best case scenario use-case - linters are generally small and easy to test.

It’s also worth noting that linters now effectively have automagical autofix - just run an agent with “fix the lints”. Again, one of the best case scenarios, with a very tight feedback loop for the agent, sparing you a large amount of boring work.

I just discovered https://megalinter.io/

I like this idea but I can’t think of a concrete example to ground it. Can anybody share a real example?

Max 200/300 LOC per file is pretty popular.

I got tired of brittle literals like `http://localhost:3000` and `postgres://…@127.0.0.1/...` creeping into my code, so I wrote a few ESLint rules that detect “hardcoded infrastructure” strings and ask the agent to find the constant in the codebase — not by guessing its name but by `grep`-ing for its value.

The detection is based on dumb string literal heuristics, but has proven rather effective. Example patterns:

const hardcodedInfrastructure = { url: /^https?:\/\/(localhost|127\.0\.0\.1|192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)(:\d+)?/i, dbUrl: /^(postgresql|postgres|mysql|mongodb):\/\/.*@(localhost|127\.0\.0\.1|192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)/i, localhost: /^localhost$/i, localhostPort: /^localhost:\d+$/i, };

Claude Code is obsessed with using single letter names for inline function parameters and as loop control variables. I don't like it and I think it is sloppy, so I told it to stop in CLAUDE.md. In my experience, Claude Code will respect CLAUDE.md around 70 % of the time, it seems to cherry pick areas that it will respect more and less often and of course it kept ignoring this instruction. So I told it to add a pre-commit hook and invoke the TypeScript compiler and analyze the AST for single-letter variable names and tank the pre-commit check when it detects one with an error message indicating the offending symbols' locations. Now it can be non-deterministic as much as it wants, but it will never commit this particular flair of slop again as the adherence is verified deterministically. I already have a few more rules in mind I want to codify this way to prevent it from reproducing patterns it was trained on that I don't like and consider low quality.

Nice. The big picture strategy is figuring out the right mix of deterministic and probabilistic programming elements and how to mix them. I see agents ignore clear instructions all the time thus we need to codify the important instructions into deterministic rules, which probably take the form of tests or linters.

I realized this recently and I've been creating a RuboCop plug-in[1] to automatically have the LLM code better match my personal style. I don't think it'll ever be perfect, but if it saves me from moving a few bits around or adding spacing I'd rather see then it's wroth. The fun part is I'm vibe coding it, since as long as the tests verify the rules then it doesn't really matter much how they work. As a result adding a new rule is pasting in LLM generated code followed by what I'd prefer it look like and asking it to add a rule.

[1]: https://github.com/tristandunn/rubocop-vibe/

Anthropic ships an official plugin to create linters for you based on your Claude Code history or instructions, it’s great. You can vibe code your lint rules per repo.

https://github.com/anthropics/claude-code/tree/main/plugins/...

I disagree. A ton of linting is cosmetic stuff, you need more specific guardrails, even though linting in general is helpful.

I agree, I have 'written' a handful of rubocop rules that are hyper specific to the codebase I work on. I never would have bothered before claude code. Stuff like using out custom logger correctly, or to not use Rails.env because we have our own (weird of course) env system.

Ha, I just had the LLM create my first custom eslint rule yesterday and was thinking that I should make more.

could you explain what this does ? like i ask it to write a testcase (pytest, etc). why is a linter better ?

Reply on news.ycombinator.com