Hacker News Reader

Comment by mmillin | Hacker News Reader

>Yet even with a lot of manual steering, that type of code does not come out of LLMs naturally, and even if the code comes out naturally like that, they will still attempt to handle now impossible errors.

This is something I’ve struggled to fight against in many PR reviews. Especially once already written, convincing someone that their excessive null checking is harmful is an uphill battle. Short of better modeling (and languages that allow for sum types to enable it), I haven’t been able to come up with a universally convincing argument against this kind of “shotgun parsing.”

Maybe it really just isn’t that big of a deal? But when actually reading through and refactoring a codebase I’ve always found it frustrating to manage these unnecessary checks. Sometimes they’re nearly impossible to delete safely once present without first adding some kind of logging or broad investigation.

View on HN

> convincing someone that their excessive null checking is harmful is an uphill battle.

The argument that seems to hit home more often than not is that optionals effectively “fork” the state space, the possible states your program can be in. And the larger the state space is, the harder it is to reason about the code and maintain it. That’s actually part of what make undesirable states un-representable means.

And AI code reviews encourage overly delusional defensive paranoia. triple null checking deep inside a function is technically a real risk, but in practice should never be hit because you've checked for nulls in every function that calls or could call the function in question and is thus not necessarily worth guarding against.

Are LLMs too dumb to understand the type system? Or is the type system too bad, to represent non-nullable?

Even in Python with its meh-ish optional typing system something that can be None is different from something that cannot be None.

How impossible are we talking?

I tend to be a fairly defensive programmer - maybe nothing currently sends this function a negative value, but how hard is it for a future code change to alter that assumption? I always figured a clear error was best. It lets even someone unfamiliar with the code know what assumptions are being made about the valid range of inputs, so they don't have to consider impossible outliers.

Obviously it isn’t totally impossible, but it becomes challenging to know if it’s required or not. It’s hardest when it isn’t just throwing an error but instead defaulting to something only half-sensible. For example replacing a negative number with 0 or overflowing rather than panicking.

When it comes to assumptions about the input, ideally model them in the type system. If you can’t, explicit checks and throws are OK in my book. But don’t check-and-hide any errors. You’ll be hard pressed to debug the issues they’ll cause down the road, since it will usually be far from the implementation that you see the impact.

I hate the "pick defaults out of a hat" approach that LLMs seem to take. I suspect this behavior may be a result of reenforcement learning on coding challenges - making the code worse by throwing in these assumptions (which in turn become part of your documented API) may get you an extra half percentage point on the code challenges.

Reply on news.ycombinator.com