xg15

Software can absolutely act goal-driven without having consciousness etc - every pathfinding or navigation system or chess engine does this.

Lots of "old-school AI" algorithms have explicit modeling of goal or target states.

(In fact, the oldest "goal-driven" system is the control loop - like in thermostats - which was the founding invention of cybernetics, the predecessor of modern computer science)

LLM coding agents are clearly able to identify some sort of "goal" state in their prompts, work towards those and track progress - otherwise agentic coding wouldn't work.

The question is of course how well this works if it's all just "grown" neural network biases and not a fixed data structure like a goal tree. So I think it's possible that an agent can be thrown off-track, "forget" its goal, etc. But the basic structure of identifying goals, evaluating progress in light of those goals and then predicting the next action based on that is definitely there.

Just use an agentic model with thinking traces visible for a while and you can see that for yourself.

They are token prediction engines. They're also very very complicated token prediction engines that pull in an enormous lot of additional information and relatively nebulous internal concepts to calculate that next token. That makes it hard to understand what kind of patterns those things can or can't predict.

Maybe to leave out the controversial "brain" analogy, it's like saying "a computer is just a bunch of electrical switches". True, but massively underestimating the complexity.

One "AI-ism" I noticed is that LLMs often just put sentences behind one another without using any connecting adverbs - and just leave it to the reader's imagination how the sentences are related to each other.

E.g. in the readme: "LiteLLM routes LLM calls across providers and translates between message formats. That core is buried under 100k+ LOC of proxy servers, caching layers [etc...]".

Those two sentences have opposite sentiment on LiteLLM, so a human author would at least put a "but" between them. In contrast, the LLM just strings them together.

This reads "blunt" and "matter-of-fact" at first glance, but I wonder if it's really just an artifact of allocating less space for text generation and more for code in LLMs.

Three of them were - I don't think the others did.

I love how people are pleading to take the AI takeover seriously, but at the same time training models for exactly those properties that such a takeover would need: Extremely long-running sessions, "perseverance" i.e. the tendency to keep on going and try ever crazier solutions instead of stopping - and the ominous "recursive self-improvement" which is simultaneously what the AI labs are warning most of and what they see as the "holy grail" and work towards with the highest priority...

> a model is gigabytes

Maybe in the future, but right now, in my understanding, we have two things:

- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.

- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).

I don't think we have something that is both small enough and smart enough to operate like a virus yet.

(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)

Exactly. But even if the model got access to its own weights, it would need to find a machine with a sufficiently capable GPU, transfer the weights and install itself there. Basically, the only machines powerful enough that AIs could self-replicate to would be other AI datacenters.

Life... finds a way.

In the method proposed in the paper, the signature is stored in the pixels and ostensibly even survives some compression. So if you made a screenshot of a signed image, the image would still have its signature inside the screenshot, but the screenshot wouldn't have any (new) signature as a whole.

That's how I understood it at least.

I'm pretty sure if signing images becomes the norm, because cameras will do it automatically then social media sites etc will start to care - or at least be thankful for the additional metadata.

I think what got me worried is the GPs take that "I expect in the near future all digital cameras to digitally sign the images they take". Sure, if you have your special, verification-enhanced camera, go for it. But if this goes like with smartphones and we have all cameras embedding such a signature - and therefore all cameras are made locked-down and tamper-resistent - then we have another category of devices removed from control of their users.

> Any of us are still perfectly free to upload images for fun to forums or social media or whatever that are private, or edited or whatever we like.

First, the paper proposed a method of embedding the metadata inside the actual image pixels in a way that makes it difficult to remove. I think it's still possible, but you're will need a specialized tool to do so and it will alter the image.

Second, that's assuming forums or social media would still be allowing uploading unsigned images. If AI images really become as much of a problem as anticipated here, then sites might decide to block any image they can't verify completely.

> Like, can you give examples of specific cases you are imagining where simultaneously the photo itself reveals no information about time and location [i]and[/i] it's something local/national/international-newsworthy where people would fear AI-alteration?

All kinds of photos from warzones have this property. We wouldn't have an entire science of geolocating photos from landmarks that were accidentally captured if everyone was so easy with putting their GPS coordinates in the file.

I understood the GP so that only some unique ID or hash of the original image would be included, not the image itself. Basically like the commit chain of Git but without the actual content blobs.

You could use this data to prove that image B is an edit of image A if you already have both A and B.

I still think this is a bad idea, because this all requires the images to have some sort of ID - and that seems like a prime target for tracking.

EXIF data is stripped for a good reason - because it can be a privacy hazard. Suddenly this plays no role anymore?

> For example, a screen attack still works: photograph a screen displaying an AI image and you get a signed photo of a fake. But it's always nice seeing big actors interested in addressing this problem.

Yeah, very nice. So this whole idea basically doesn't work - but we get a new stealth way to embed metadata in an image that can be used for tracking...

(And a new narrative why cameras need to have TPMs and locked-down firmware as well)

A Moroccan city founded by Spanish traders following Portuguese colonists, then part of a French colony which is controlled by the Germans until freed by the Americans and becoming the icon of an American movie...

Yeah, this whole blog was repeating the old fairy tale narrative of automation "if we automate the work then workers will have sooo much more time!"

I'm surprised they expect people to still believe that stuff.

For Qwen3.8, my first impression was weirdly Reni from TwoKinds: Immensely powerful but also apparently a ball of insecurities in the thinking traces.

(which makes sense, as I think one motivation for reasoning traces is to explore different options and approaches. So it makes sense that there is a lot "but wait, let me reconsider" in them)

What I found surprising is how strongly "logical contradictions" seem to influence the thinking trace. E.g. I had a situation where I accidentally copied a python file into a repo, but forgot to add a package that the file was depending on. Then I (somewhat carelessly) commited the file without ever testing it and gave the agent a task to work on the file.

If I had run it in Python, I'd have gotten an "cannot resolve import" error and that would have been the end of it.

Instead, the model went absolutely haywire.

The thinking traces were full of utter confusion how the file could possibly resolve its dependencies - but at the same time, entertaining the possibility that a committed file may have an error was apparently Verboten. Hence, the model wrote up ever more outlandish theories how the file could resolve that package and in the end started to make tool calls outside the repo to explore the entire file system before I stopped it.

Moral of the story: Underspecified requests are fine, but beware of anything self contradictory, it can easily send the Qwen model spiraling.

At this point the "you need to contact support to get unblocked but also must already be unblocked to contact them" catch-22 has occurred often enough and is in place long enough that I can't imagine it's an oversight anymore. This looks more like a deliberate policy.

"The blaze is out of control, so we have to pour even more gasoline on it to contain it!"

Who is "Humans"? This stuff is done by a handful of tech companies and megalomaniacal billionaires who are pretending they represent the entirety of the human race. It is not done by "us humans".

AI models don't train themselves. The vast majority of even just the US population is deeply skeptical of this stuff, even if they use it a lot. You can see in the whole data center debate how little people are willing to support even just inference. And now we're seriously claiming those people would want to have ever-accelerating model training and recursive self-improvement?

I'd honestly be interested to know what kind of argument HN would accept to acknowledge there IS a problem.

I don't see what unions would have against automated emergency stop systems. Workplace safety is a core issue of unions, so if anything, shouldn't they demand those systems?

> make_more_likely is, of course, doing a heroic amount of work here.

Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.

> it is trained to predict next tokens as they occur in its training data.

What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.

Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.

This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.

Maybe I'm out of the loop, but wasn't AGI the full-on scifi version of AI, where the AI is a persistent, conscious entity? I don't see how task benchmark scores are relevant for that.

Yeah, apparently they both (used to) offer unbounded registrations of 3LDs and unbounded registrations of 2LDs? So if I see j.doe.name, the only way to find out if "doe.name" is a public suffix or not, i.e. if I should (not) be able to set a cookie on it, would be to email the registrar?

So does that mean that in practice, .name domains were always treated by browsers like regular 2LDs, meaning the cookie and origin protection was always broken for those domains?

Doesn't sound like good news for the guy in the OP...

Yeah, the story reads to me less as if it was time pressure and more like they underestimated the risk.

"What could go wrong" etc etc

There are other next token prediction algorithms such as markov chains or HMMs that also "fit the same interface", but are vastly simpler than what LLMs use.

I've seen various posters talk about "stochastic parrots" or about how LLMs were just "using very simple word statistics" to get their results, which sounded to me very much as if they thought of LLMs simply as glorified markov chains. I would consider that a big misunderstanding.

> which in no way challenge the established view that these bots are next-token predictors.

I mean, of course they are, that's literally what the inference loop does. You can look at the source of your favorite model runner and you'll see exactly that.

What I find misleading about this term is that it focuses attention on the "next token" part and glosses over the "prediction" part as some sort of unspecified "statistical algorithm" - even though this is where most of the work happens and where the interesting questions are.

Personally, I like if not everything is going through my phone.

It can be distracting. You have social media, messages and apps on the same device you need to login. Already makes it harder to put away the phone for a while.

It can never be "cold": You have your phone always with you, which means the risk of it being stolen or lost is higher than with something you can keep in your desk drawer for 99% of the time.

It's a single point of failure: If you should lose it, you potentially have a huge problem: You're right that a thief couldn't make much use of it and we have sophisticated tech to remote-lock and locate phones now. But that won't help you if you need your phone to log into your Google or Apple account to access that tech in the first place.

(Yes, I know there alternative ways to regain access to Apple or Google accounts, but those all rely on you having something... not on your phone)

I don't really own the phone. I know it's paranoid, but I don't like the idea of some update being able to change the way authentication is done. (Even if this is improbable in practice) At least FIDO2 firmware has a fixed interface.

Well, unless you have a device-bound or FIDO2 key, according to the post.

And I think for FIDO2 keys, this is fine. If I can register a key (plus a backup) the usage doesn't seem so different from a regular key.

For the rest, I think what irks me is the feeling of "you have free choice which corporation you want to entrust all your login credentials with, but you will have to choose one". Previously, password managers were a convenience (that incidentally also increased security, because they made keeping a separate, hard password for each domain practically feasible) - but nothing stopped me from keeping passwords at several different places at once or memorizing some of them, in case I lose access to the password manager.

Now suddenly, they become the arbiters of my logins everywhere. What happens if they ban me, or go out of business or get bought up? (Or in Apple or Google's case, make arbitrary business decisions that can now affect the way I login to completely unrelated services?)