Hacker News Reader

Comment by tripleee | Hacker News Reader

I had to laugh yesterday when Claude went and changed a feature I didnt want or ask to be changed and the suggested message was something along the lines of revert the change to it. Like it knew I wouldnt like it but did it anyway

View on HN

There was an interesting LessWrong post recently about the "talker" part of the robot being very different from the "doer" part: https://www.lesswrong.com/posts/cJX2ssssGoYqnijwi/the-talker...

ADHD makes me like this too. It feels like I always know what I should do and then I just don't do that.

I can relate. The most sadly amusing part is when I know I’m approaching something the wrong way and I still keep at it even though the voice in my head keeps telling me that it should take a different approach.

Last year I tore down the whole roof of my cabin instead of just the asbestos tiles because I had momentum, all while I kept thinking “you should really stop and consider what you are doing, this was NOT the plan”.

The voice was right and now I need to learn how to level the roof on an old Frankenstein’s monster cabin…

I feel like the voice is always right, but some part of me tries to know better, it doesn't work and I end up regretting it.

But sometimes the roles are flipped and the voice is the one trying to tell me to give up on something that ends up fine.

I can never figure out which side is the right one!

My family has a history of schizophrenia. Sometimes I feel tempted to start thinking about my mind as a collection of separate agents and separate voices, and quickly it starts to feel like it's pulling me into a place where I know I shouldn't be looking.

Compared to the uniformity of LLMs, human minds are fascinatingly different from each other. Are voices good, are they bad, or maybe you don't have even an inner monologue? Yet these differences mostly stay hidden and don't stop us from collaborating and understanding each other through empathy.

> or maybe you don't have even an inner monologue

Does this exist?

Apparently yes, it was a pretty big online topic a few years ago. Here's a recent Reddit thread:

https://www.reddit.com/r/AMA/comments/1o0kz9d/i_have_no_inne...

> My family has a history of schizophrenia. Sometimes I feel tempted to start thinking about my mind as a collection of separate agents and separate voices, and quickly it starts to feel like it's pulling me into a place where I know I shouldn't be looking.

No history of schizophrenia here, but my mind definitely is that way. (Thanks, DID!)

There's a theory (multiple theories, actually, like Internal Family System) that most minds are that way, they're just more or less distinctly separated.

That was an interesting read. I don't use LLMs but I have been wondering for a while with trends in model size, RL and synthetic data if we'd start to see more obvious separation of functions.

Most of all though, I couldn't stop thinking about Wintermute and Neuromancer...

"But a human cannot point to a piece of the outer world and say, "See that stuff right there? That stuff is Failure."

Counterpoint, I would point at the mouth of a shark or a slippery cliff and say, "That is failure"

Given how little they've changed over the years, sharks are very successful.

I think the point was that moving towards these outcomes (bit by a snake, ripped apart by a shark) are examples of failure.

Humans create structures with clearly defined success and failure all the time. That’s part of good management. Experienced people can often say whether an effort is going well or not and list reasonable justifications for their assessment.

Haha, yesterday I found a problem which I wasnt aware of since the function was not used for longer time. Its about dynamic compilation of a C# file during app runtime. To do this, you need to include some using references at the top, though they are not used in the file itself.

Below this block was a comment: "// DO NOT REMOVE UNUSED usings BECAUSE THEY ARE NEEDED FOR DYNAMIC COMPILATION - DO NOT TOUCH THIS BLOCK !!!!!!!!!

-> Claude has removed the using block in question during its last session :-X

It'd be interesting to see the trace of that session. Depending on how it used its tools, it may never have seen that comment.

The thing is:

This comment was there already for a very long time, so "earlier" it was OK.

And noone noticed because since these usings are unused in the code, the compiler didnt complain about missing declaraionts. (if you want to be able to debug dynamic code at runtime and changing it, requires some specific .NET packages included) - and because noone used this dynamic compilation feature for a longer time, we didnt became aware of that its missing

Pre-LLMs I had a version of this at the top of a file:

Under no circumstances should you allow IntelliJ 'optimise' these imports, it doesn't understand Scala 2 implicits very well here so it'll all fall apart like a wet cake!

I noticed Claude seems more assertive in its code change lately, it's sometimes good, sometimes annoying.

Just when they finally toned down the obnoxious verbosity...

So it's self-aware enough to suggest the fix but not self-aware enough to not do the thing in the first place.

Nothing self-aware about it, it's just LLM output. I'm not trying to be flippant, I just feel really strongly that you shouldn't use words like "self-aware" about an LLM.

It feels like screaming into the void sometimes. People don’t know / don’t want to know. Learned people, smart people, makes no difference.

with more tokens it probably could be. you dont need the real tool call response, if the likely tokens after the tool call(s) is that the user is unhappy, it could predict all those tokens, then have a secomd model predict whether the tool call should be made or not

Perhaps they are preparing for a switch to usage-based pricing for everything at some point and are thinking of ways to burn tokens? Not to be conspiratorial but many companies think like that, and tokens = money at this point.

I don’t really understand the emphasis here.

Customers pay for things that are useful. If the token drifts from being a reasonable proxy for what is useful, customers won’t accept being charged on that basis (or jump to the best trade).

If I have an employee who works 1 hour and builds me a beautiful cabinet and another who builds one in the same time, with a door falling off, the one with less productivity per hour (token) is not getting a call back.

Think about the target audience, this technology is being pushed on people with money that don't have the depth of insight that you do.

>Customers pay for things that are useful.

Depends on the definition of useful. In a corporation "useful" is "checks regulatory boxes" - it doesn't need to actually be useful, it must just pretend to be useful. Hence all the shit customer service processes. They are useful to the corporation as in that they tick the "we have customer service" box. Whether the consultant just hangs up on you instantly after picking up doesn't matter.

It’s absolutely on purpose, everything is with these people.

For example if you ask Claude chat to write something of a certain length, it won’t, it’ll just reply at the same length as everything else. Why? Because they’re billing at a fixed price. They don’t want you to get a whole essay as an answer because you’re on a plan.

Claude code on the other hand, which is supposedly the same model, will happily write out a billion lines because it is billed by the token.

From the subscription plans Claude code consumes the same usage as the chat, and offers the same unsubsidized usage credits, which is close to billing per token.

If you read the model specs, they have a maximum output length. If you are running into that it might simply be that you are seeing the difference with a harness that can allow it to take another turn automatically and build up the output in a file vs the chat which may not be able to, though they do get some tools there.

The response output length limit includes the thinking tokens.

Almost certainly. I work for a large company that does financial modelling and analytics for risk. That's the only working model that they have going forwards and believe me it's going to be expensive. They are operating at a loss. And they will be introducing that before they go IPO. They're delaying the IPOs through some of the "we're not ready" PR statements to get as many people dependent on the product as possible and to avoid making public finance statements.

It's going to be a mess very soon. A huge mess. A huge incremental cost mess.

You're being hooked on the good heroin and they're going to charge you for the shit stuff later.

Maybe the silver lining will be the collapse of the market, hardware being accesible again and we can just run local models.

A man can dream, right?

If there's an economic disaster, I would expect that a huge chunk of the hardware gets recycled because the materials will be needed to make weapons.

Only partially joking there.

instead of depleted uranium tipped ammunition we'll use deprecated ram chips.

said that as a joke then remembered how all the drone weapons being used in ukrane/elsewhere are in a way just computer chips with explosives and motors... so ya you may have a real point.

Yeah that's sort of my point unfortunately. Wish it sounds like I was insane but it sounds less like that every day.

So, currently, if you sign up as a fake new customer , you get rotated on the good stuff? So i can get the blue caps with that - and dilute it down myself?

No. Everyone is on the good stuff now. Give it a year.

It'll go from a monthly usage plan to a per-token usage plan with minimum commitment. Same as AWS pricing did.

LLMs "think" with tokens - it's the only unit of state they know, so it's really more of an accident of having larger models with more context.

The suggested message would’ve been only created after it did the unnecessary work.

Of course

The comment you're responding to is pointing out irony

Reply on news.ycombinator.com