Comment by gavinh | Hacker News Reader

Comment by gavinh | Hacker News Reader

> We may create codebases that are not merely hard to maintain by humans, but that assume machine participation as part of their maintenance model... People more and more merge code they cannot fully explain. People lose their ability to create issue reports or discuss things in chat, without augmenting or rephrasing their messages with the context provided by a clanker. Too many people increasingly rely on a machine to summarize or contextualize it. More and more do I encounter people who converse with me through the indirection of an LLM.

I experience this daily now. It find it discouraging and concerning.

I believe we're merging more code we can't fully explain because we are now relying on code review to build the mental model that was previously built by writing code and collaborative technical planning. I don't think code review is fit for this purpose. I do think we can extend code review with structured exercises, informed by pedagogy, that strike a better balance between friction and understanding. (I'm looking for help testing these exercises).

And this code is often full of security vulnerabilities. It's just hacks on top of hacks on top of hacks. You end up with 100K lines of code full of weird fallbacks, doing something which could have been done more reliably with just 1K lines of code.

I think author's comment about preferring systems which make invalid edge cases impossible rather than implementing fallbacks is hugely important. With the fallback approach; you end up implementing fallback on top of fallback on top of fallback... Each fallback seems to increase the amount of code exponentially and somehow it always creates new problems. This should almost be a 'General law of system design.'

Fallbacks reduce the risk of failure but make failures more complicated and harmful when they do happen.

As a software engineer, like the new coding environment which is being created by AI.

Big tech companies have created infinite work for me. The human developer has become a critical component of code execution. The human needs to always be present to handle the nearly infinite number of difficult unhandled exception cases which are guaranteed to occur from time to time.

The software engineer is no longer like a laborer, but more like a security guard who sits at his desk drinking coffee most of the time and only steps in on rare occasions when something goes wrong.

Ensure you check every PR with opens4.8 or fable - they catch every security issue upfront.

This isn't going to work because the LLM doesn't have enough context. Many security issues involve a failure mode which cuts across multiple parts of the code. A PR which seems perfectly valid on its own may be the missing piece which opens up a vulnerability. Each component may be fine on its own, but brought together, the system is vulnerable.

Think of a machine with interlocking gears; each gear may itself be perfect and may fit perfectly with each other, but then if a tiny pebble comes between them, the entire machine breaks. Maybe the problem here is that the final gear was too close to the ground and would catch stray pebbles kicked up by the wheel in front of it... The LLM couldn't know this unless it understood the full context in which the change occurred; not only the code, but the environment itself.

In a poorly designed codebase with hundreds of thousands of lines of code, it's impossible to have the full context of the code even. The architecture would lack proper separation of concerns to allow one to effectively establish an appropriate defense perimeter. In a poorly designed codebase, every part of the code can harbor a vulnerability.

It's like; if you don't have a proper access control layer which is automatically and declaratively enforced for all your endpoints, every endpoint will have to enforce security restrictions on their own; duplicating similar-looking code over and over. If one endpoint out of 1000 incorrectly enforces a security restriction, that could be a critical vulnerability.

Of course you don't just check the diff. Rather in your CI infra, it's important as part of every PR, it needs to be given the full repo to check if it introduces any issues. This works wonderfully on github, even with non SOTA tools like gemini-code-assist.

why do you think it's not possible to have full context of codebase? modern harnesses excel at finding all the right codepaths, even in a large codebase.

> they catch every security issue

No they don't! They catch a lot but certainly not all of them. I can't explain why but it 100% happens.

My experience with LLM's is that the "fallback" issue is probably one of the most serious issues. I have seen zero talk about it outside of this thread. I'm not sure its even being worked on. The llms have this terrible drift towards always making something happen even if that thing is not even related to the task at hand. "failure" in the sense of simply throwing an error/exception is something models seem highly resistant to.

I cant tell you at this point how many times I've seen them do something like

Fail > make up values > maybe log it > keep working silently with increasingly corrupt data.

Not to mention that effectively reviewing code is a much more difficult skill than writing it. Without a good mental map of how it affects other parts of the system, it’s basically a rubber stamping ceremony.

Github’s poor PR UI doesn’t help either, there’s limited tooling to navigate around the codebase not directly changed (but affected) to identify and highlight problems.

What's your product? I'm dying to see how the product developed to 100x human standards by agent swarms is. Must be amazing.

/s

https://pages.cs.wisc.edu/~remzi/Naur.pdf

I'll help if I can.

Thanks, I'd be happy to chat. gmh283@nyu.edu.

Reply on news.ycombinator.com