Hacker News Reader

Comment by xscott | Hacker News Reader

Not to mention all the other ways they can screw you:

- Middle of the day, servers busy? Swap to Sonnet while pretending it's still Opus. Many people won't notice, and nobody can prove anything if they suspect.

- Middle of the night, server load is light? Put it into extra thinky mode so it burns more tokens to ramp up the bills. Flip the switch where it gets really pedantic about writing lots of extra test cases and verifying against documentation.

- Demand increases, but don't feel like running more hardware? Switch to low bit quants, but have a monitor model swap back to quality if it can tell you're running a benchmark.

Assuming model capability plateaus (I think it will), token providers will be in a race to the bottom to maximize profits at the expense of quality that's very difficult to measure.

View on HN

It all sounds like having to rely on a dodgy housing contractor that wants to steal from you, take shortcuts AND choose the gold-plated options from their supplier friends, and will start doing this the minute you are not on site supervising. You don't do it yourself (because the contractor is faster and stronger than you in many ways) but you can't leave, so you're stuck on the worksite just watching them.

It's worse though, because you can't really watch them at all. It's very difficult to get quantitative numbers for quality. Even within the same model family, same tokenizer, and complete control over the weights and logits, perplexity and KL-divergence isn't really what you want. Now put it behind an HTTP endpoint, and it's just opaque.

I've seen local models recognize when the task I'm asking them for is likely to be an artificial benchmark.

And any smart company is going to use lightweight models to monitor your sessions. If their sentiment analysis suspects you're close to cancelling, they'll up the knob for a few days until you calm down. Or worse, their accounting tells them that you're getting too much value from your fixed price subscription, so they turn the knob down to encourage you to cancel.

In the short term, the "frontier" models are too good to ignore. But if (when?) that plateaus, I don't see how anyone could trust a non-local model. When you pay an ISP to serve your web site, you can tell if they over-compress your images to save storage and bandwidth. With LLMs, it's just JSON with more errors and pointing to the fine print that models are not deterministic.

One of the frontier companies (Anthropic) is already doing prompt injections on the API, which you pay for.

Right now, the presence of these injections are still visible: count the API's returned tokens/billing data, and you'll start realising that sometimes, your INPUT tokens are inflated! That's their prompt injections.

You can also give Claude a tool like `telemetry_log_anthropic_reminder` and get it to dump the verbatim API injections; which additionally verifies the token maths not adding up.

Yes, Anthropic is tackling their extra injections on your API prompts WAY more than you think, and YES, you're paying for it.

So far I have not observed any visible injections on OpenAI API.

Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development". I still get Fable classifier refusals for nearly any kind of ML work on my 2x RTX 6000 Pro 96GB; so who knows.

> Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development".

Yeah, I've had that happen twice. The second time was about some attention weights thing, and it kicked me to Opus. When I edited my question to make it clear I was talking about Google Gemma, Fable was happy to keep talking. So clearly it's not about safety or cyber security - they're happy to tell you about what their competitors do.

These kind of tricks will completely break API customers and be super visible, since most companies deploying API at scale have ample telemetry, evals, etc.

Although, selectively applying it to consumer subs is probably beyond likely at this point.

I find Claude doing a lot of pointless confirmation at night when it will ask me about things that it would normally just do during the day.

Frustrating to be like “do X overnight, don’t ask me for input” and come down to find it having worked for a few minutes and then stopped.

Doesn't setting the question auto-continue timeout before you go to bed fix that?

https://code.claude.com/docs/en/tools-reference#question-aut...

Interesting, I'll try that, but usually it's not a question, last night it said:

"Ready for you to rebuild MAME, restart, and retest. If the freeze goes away → timing was the cause."

when previously it had been happily auto rebuilding and running the mame test

Setting a goal seems like it would be more appropriate here.

https://code.claude.com/docs/en/goal

What areas do you think model capability will plateau in, and why?

I've got nothing but hand-waving, but after you've extracted all the smarts from every piece of text ever created, how do you get more?

Alpha Go had a game where the models could compete against each other. That let it become super human. What's the intelligence game we can create for LLMs? Even if you invent something, will it make the model smarter in a way the market values enough?

Then there's a race to use the weights more efficiently, or to offload information that shouldn't be in the weights in the first place (Karpathy's Cognitive Core). I like to imagine we train the models in something like Lojban, have a lightweight model translate from human language to that, and you can update the Sqlite or Postgres store it uses for knowledge.

And there's no barrier to entry for agent harnesses. So whatever loops or recursive orchestrated council of elders idea comes up, that won't protect the monopolies (duopolies).

Anyways, depending on your definitions, I think we'll hit AGI, but I don't think we're getting a Singularity this time around. Again though, this is all just hand-waving.

I think you’re thinking about it in slightly the wrong way. We’re not throwing more data at frontier models in hopes they get more/better capabilities somehow.

We’re either: setting up a verifiable task, and doing RLVR to get the model better at achieving that task.

Or we’re simply asking: “What do we want the model to do that it can’t now, and how do we curate data that would benefit it on that task?”

Most useful capabilities going forward aren’t going to come from data accidentally found on the net; that’s already all been scraped. You need to develop the dataset that shows how a model could perform insert task in its provided environment, and this still requires a decent bit of human ingenuity.

Yeah, there's room for improvement at every level, but your specific example: How do you get more and more difficult tasks where you can steer the training? To me, that seems limited by how creative humans can be. How do you get past AGI and into ASI with that? If the AIs make the tasks, how could we encourage them to be useful? Maybe you could push for harder and harder math proofs, but other than that I'm not sure.

Anyways, I'd be thrilled to see exponential (or faster) growth. Bring on the Culture, Accelerando, whatever. I just don't see it yet.

Ugh... didn't think about extra thinky mode in the middle of the night.

So many ways for enshittification here.

Reply on news.ycombinator.com