Hacker News Reader

Comment by glimshe | Hacker News Reader

Without reliable benchmarks, they are Mythos-like only in the sense that they accept text as input and produce text as output.

View on HN

I don't even look at benchmarks anymore. I just try different models as they're released on our large, proprietary, systems software codebases in real, shipping products or projects that will ship eventually. It's pretty clear which models help me do my job better or faster. I'm fortunate enough to have the token budget to use basically as much as I need, for now.

No need for benchmarks, evals, marketing, system cards or anything like that. I read the web for tips, practices and release announcements. My colleagues and I share our experiences with each other but beyond that, everything else is just noise.

This is the way. Not that big of budget here. But if there’s something promising, I just try that for a month or so. But even then… at this moment I’m using z.ai models and those do the job. No need for anything else. So I’m staying until there is something new, same affordability, but a lot better. (Using a coding plan)

I am not against AI but I do wonder how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who?

I “trust” OpenAI and Anthropic (somewhat) but to be honest I still feel only safe using it on code without any secret sauce whatsoever. Luckily that’s a lot of code, but still. I wonder how others are looking at this?

(FYI I feel the same about Github and we also don’t store our code there)

> how you guys handle the fact this leaks all your code and is stored forever on servers belonging to God knows who?

I don't.

My company doesn't host any code on GitHub, we have our own Git servers.

Regarding the privacy issues with OpenAI, Anthropic or anyone else, we engineers are just using the tools we've been authorized to use by the company. If there's a security issue, that would need to be worked out between the company and the model providers.

Would I use external model providers through an API for personal projects? For everything I'm doing now, probably. Could I see a day where I'm working on something too sensitive to be willing to give any data to these companies? Possibly.

Well if they are hyped like Mythos then we can add that to the list of “like Mythos”. Perhaps what’s missing is their CEO warning the world that their model is too unsafe to be released on the internet and someone must stop them before it’s too late.

They provide benchmarks in the paper https:// arxiv.org/abs/2606.21228

Think 'indepenent' was implied when glimshe wrote 'reliable' benchmarks. Needs to be on the usual leaderboards.

Reply on news.ycombinator.com