Hacker News Reader

Comment by PeterStuer | Hacker News Reader

ChatGT makes exactly the same mistakes as your colleague that has no actual experience in a certain matter: they present what they reasonably think how something probably should be or work in absence of factual knowledge about the insane mess that was actually created in reality.

Just ask a technical colleague that has no deep experience in the specifics about how they think e.g. SharePoint, SAP or Microsoft Identities work. The architectures they will extrapolate from sane logic will be very far of the incredible craziness of the actual reality.

View on HN

I still think there’s a big difference. If asked to speculate on the architecture of Sharepoint, my response would be “I have no earthly idea.” If pressed further for an answer, my response would be “It’s probably some over-complicated mess and I don’t care enough about Sharepoint to spend any further time on this line of questioning.” I have yet to see ChatGPT just admit it doesn’t know, but in this case “I don’t know” is the most trustworthy answer I, as a technical person, could give.

It doesn't know it doesn't know.

It is fundamentally incapable of "knowing" anything. It is a statistical engine, with no internal representation of the world or understanding of the words it emits.

It clearly does have an internal representation of the world, implicitly encoded in its network weights. Quite an accurate one too, for most general knowledge, and one that can be updated in-context - it handily solves "blocks world[0]" style tasks. "Understanding" and "knowing" aren't helpful words, just focal points for pointless philosophical arguments.

[0] https://en.wikipedia.org/wiki/Blocks_world

That's not a representation of the world. It's simply a lossy encoding of its data. It's not semantically structured in the way that our thoughts largely are—it's merely syntactically structured.

What exactly is the difference? It's clearly managed to abstract the training data to a ludicrously deep degree, such that it's capable of solving semantically non-trivial problems it's never seen before. It can make metaphors, accurately predict the behavior of humans in complex social scenarios, and translate arbitrary passages between syntactically distinct languages while preserving nuance. That last task in particular is pretty much a slam dunk against any argument of the type you are making.

"Sufficiently advanced syntax is indistinguishable from semantics."

Sufficiently advanced syntax may be superficially indistinguishable from semantics, but we're not talking about output in this subthread: we're talking about an internal representation of the world.

Pure syntax, no matter how advanced, is insufficient to represent the world in any meaningful way. By definition, in fact, because pure syntax is divorced from meaning.

>Pure syntax, no matter how advanced, is insufficient to represent the world in any meaningful way. By definition, in fact, because pure syntax is divorced from meaning.

Then "by definition" LMMs transcend "pure syntax", because of all the examples of semantically interesting tasks they can do which you failed to engage with. It clearly has internal representations of abstract concepts. Your argument seems to be that you intuitively reject the possibility of complex emergent behavior from such networks because they're trained on "just words", and no amount of demonstrably intelligent emergent behavior will convince you otherwise.

There's nothing magic about meat brains. Both we and the LMMs learn a world model from a bunch of input data we correlate until it makes sense. There's no "meaning gland" we have that ChatGPT doesn't.

"That's not a representation of the world. It's simply a [representation of the world as viewed from its input data]" is an... interesting take.

They don't know it doesn't know

Not sure why this is being upvoted but it's completely wrong. ChatGPT will answer like a colleague that is a subject matter expert on whatever you're asking them. And in doing so goes above and beyond inventing minute details that are blatantly wrong to pretend they're an expert.

But the vast majority of folks have a concept of uncertainty and will either communicate (or know internally) that what they're saying is speculative. ChatGPT is like a pathological liar that pretends to be an expert and is often correct.

Someone with no experience may give you their best guess, but they'll also tell you that.

And if I ask a colleague of mine for their sources on a particular matter that they might not be sure about (or I find curious), they would not send me a list of totally made up journal articles and book titles like ChatGPT does.

While you are right, I just don't think this is a useful comparison. There are other context clues when talking to people. I tend to know whether my colleague is talking out of his ass, and most people will preface stuff by saying they don't really know.

We are trained to trust what computers are telling us, and ChatGPT doesn't 'qualify' what it's saying. I think if ChatGPT could preface what its saying with 'Well, I'm not super sure, but here is what I think' that would go a long way to solving this issue.

Yeah, you could call it a "human" characteristic

Right, but all this means is that the training data doesn't include thorough or an adequate representation of that knowledge, no?

I don't think so, the rules of chess are certainly in the training set yet GPT-4 can not realiably play legal moves, yet alone good ones.

So they're only as good as most humans in that context? No biggie...

Reply on news.ycombinator.com