Hacker News Reader

This is a great idea. Like, if someone asked me to count the number of B's in your paragraph, I'd yeet it through `grep -o 'B' file.txt | wc -l` or similar, why would I sit there counting it by hand?

As a human, if you give me a number on screen like 100000000, I can't be totally sure if that's 100 Million or 1 Billion without getting close and counting carefully. Should ought have my glasses. Mouse pointer helps some as an ersatz thousands-separator, but still.

Since we're giving them tools, especially for math, it makes way more sense to start giving them access to some of the finest tools ever. Make an MCP into Mathematica or Matlab and let the LLM write some math and have classical solvers actually deal with the results. Let the LLM write little bits of bash or python as its primary approach for dealing with these kinds of analytical questions.

It's like giving a kid a calculator...

If you have to build an MCP for every system you aren’t building intelligence in the first place.

You don't need specialised MCPs for this. In the past you could add "use python" to there chatgpt prompt and it would do the right thing. This is exactly the intelligent "use the right tool for the right thing" idea. Chatgpt just want trained to apply it in the right circumstances automatically.

Why? just cause? analogize it to the human brain.

Why does it matter? I don't care whether it's intelligent, I just need it to be useful. In order to be useful it needs to start fucking up less, stat. In current form it's borderline useless.

Fair criticism, but also this arguably would be preferable. For many use cases it would be strictly better, as you've built some sort of automated drone that can do lots of work but without preferences and personality.

I think a piece of software that can correctly decide what oracle to consult to get answers to questions you give it can be called intelligent, even if it itself doesn’t know any facts.

What if MCP servers were really the neurons we were looking for all along? /s

We have fingers and a keyboard / mouse, because that's the best thing we've come up with. If we could output binary to a bunch of program interfaces directly (perhaps via some neuralink type thing) we would surely take that option.

> As a human, if you give me a number on screen like 100000000, I can't be totally sure if that's 100 Million or 1 Billion without getting close and counting carefully.

I become mildly infuriated when computers show metrics (or any large number) without thousands separators.

Worse still, I often see systems that mix units, don’t right-align, and occasionally blend in a few numbers with decimals together with whole numbers! Then, update everything every second to make things extra spicy.

Right. So if we have these kinds of cognitive blindness and just accept it as status quo, and when necessary we make up for it with tooling... it's not unreasonable to assume that AI will also have to do this

You can’t just prompt your way out of a systemic flaw

What's the systematic flaw?

The fact that it can't count.

If a LLM can get IMO Gold but can’t count, that’s an issue.

This particular LLM did not get an IMO Gold.

I think the issue is that it was advertised as having PhD-level intelligence, while in fact it can't count the letters in a word.

That isn't a flaw though. Counting is orthogonal to the functioning of LLMs, which are merely completing patterns based on their training data and available context. If you want an LLM to count reliably, give it a tool.

Anything articulate (hence possibly convincing) which could be «merely [guessing]» should either be locked out of consequential questions, or fixed.

We're still on that's just how it works. The LLM isn't aware of any consequence, etc. All it does is complete patterns as trained. And the data contains many instances of articulate question answering.

It is for those using the LLM to be aware of its capabilities, or not - be allowed to - use it. Like a child unaware that running their finger on a sharp knife blade will lead to a bad slice; you don't dull the blade to keep the child safe, but keep the child from the knife until they can understand and respect its capabilities.

If your prototype of the «knife» is all blade and no handle, fix it and implement the handle.

If the creation is planned, you will have also thought of the handle; if it is a serendipity, you will have to plan the handle afterwards.

Pretty sure it doesn't matter to the child whether the knife has a handle or not. They'll eventually find a way to cut themself.

It matters to the adult - who is also an user.

LLMs do not deliver (they miss important qualities related to intelligence); they are here now; so they must be superseded.

There is no excuse: they must be fixed urgently.

LLMs deliver pretty well on their intended functionality: they predict next tokens given a token history and patterns in their training data. If you want to describe that as fully intelligent, that's your call, but I personally wouldn't. And adding functionality that isn't directly related to improving token prediction is just bad practice in an already very complex creation. LLM tools exist for that reason: they're the handles, sheaths, sharpeners, etc for the knife. Teach those adults who're getting themselves cut to hold the knife by the handle and use the other accessories that improve user experience.

> given a token history and patterns in their training data. If you want to describe that as fully intelligent

No, I would call (an easy interpretation of) that an implementation of unintelligence. Following patterns is what an hearsay machine does.

The architecture you describe at the "token prediction" level collides with an architecture in which ideas get related with better justifications than frequent co-occurrance. Given that the outputs will be similar in form, and that "dubious guessers" are now in place, we are now bound to hurry towards the "certified guessers".

> Following patterns is what an hearsay machine does.

That's also how the brain works, at least partially. Primary differences are it takes and processes (trains itself on) raw sensory data instead of character tokens, and it continually does so for every conscious moment from at least birth until death.

> how the brain works, at least partially

With the difference, which have us go back to the original point, that the human mind has a crucial property of going beyond "pattern-based" intuition and check mental items lucidly and consciously.

> and it continually does so

It also persistently evaluates consciously and "store" and "learn" (which must be noted because it is the second main thing that LLMs don't do, after the problem of going past intuition).

> check mental items lucidly and consciously

Capabilities that evolved over millennia. We don't even have a decent, universally-agreed upon definition for consciousness yet.

> "store" and "learn"

Actually there are tools for that. Again, the core LLM functionality is best left on its own, and augmented on the fly with various tools which can be easily specialized and upgraded independently of the model. Consider too that the brain itself has multiple sections dedicated to different kinds of processing, instead of anything just happening anywhere.

> Capabilities that evolved over millennia

Means nothing. Now they are urgent.

> consciousness

I meant "conscious" as the wake opposed to deliriousness, as the ability "to be sure about what you have in front of you with the degree of accomplished clarity and substantially unproblematic boundaries of definition",

not as that quality that intrigues (and obsesses) less pragmatic intellectuals in directions e.g. at the "Closer to Truth" channel.

When I ask somebody, it has to be sure to a high degree. When implementing a mind, the property of "lucid conscious check" is fundamental.

> tools for that

The "consciously check, then store and learn" is structural in the proper human mental process - a high level functioning, not just a module; i.e. "it's what we do".

Which means, the basic LLM architecture is missing important features that we need if we want to implement a developed interlocutor. And we need and want that.

> Now they are urgent.

Something being urgent doesn't mean there's a known viable pathway to an ideal implementation. The brain has billions of neurons working in tandem, a scale we're nowhere near to replicating, last I checked. And there are signs pointing to scale of neural interactions as a key factor in intelligent capabilities.

> When I ask somebody, it has to be sure to a high degree.

You're looking in the wrong place if you need surety. LLMs aren't "sure" about anything, and will never be. We going in circles at this point, but if you need certitude in anything, add tools to the LLM to increase surety. For some reason you seem to be pushing for a less optimal solution to a problem that already has a decent one.

BTW seems you may be - unconsciously? - crossing a person with an LLM there.

> structural in the proper human mental process

The brain is very modularized. It has sections/lobes which specialize in core life support functions, seeing, hearing, movement, reasoning, memory, etc. That's why brain surgeons can reliably know what capabilities may be affected by their actions in a given part of the brain. And all those functions are tools in some way to something else, for example reasoning would be pretty limited without memory.

And even at a larger scope, as humans we still use tools to achieve greater surety from our mental processes. That's why we have calculators, watches, cameras, etc. And why some would type "blueberry" into a tool with proven spell checking or autocorrect capabilities and eyeball the letters for a couple seconds to confirm the number of "b"s in it. The brain as a whole is still pretty fallible with all its capacity and capabilities.

> doesn't mean there's a known viable pathway [...] solution [...] BTW seems

It means it must be researched into with high commitment. // LLMs are an emergency patch at best (I would not call them «decent» - they are inherently crude). This is why I insist that they must be overcome with urgency already because they are now here (if a community needed wits and a lunatic appears, wits become more needed). // And no, I am not «crossing»: but people do that, hence I am stating an urgency.

We do not need to simulate the brain, we only ("only") need to implement intelligence. That means the opposite of stating hearsay: it means checking every potential uttering and storing results (and also reinforcing the ways that had the system achieve sophisticated thoughts and conclusions).

It is not given that LLMs cannot be part of such system. They surely have a lot of provisional uttering to criticize.

There's already a lot of research happening for the next thing. The AGI race has been on among not only companies, but also between the largest nations. Everybody's doing their best and then some.

It could very well be that the highest intelligent functionalities requires a closely brain-like substrate. We don't know yet, but we'll get there eventually. And it is very likely to be something emergent, not specifically programmed features as you seem to be insinuating with "... checking [...] and storing results..."

> as you seem to be insinuating with

The implementation details are not clear, not the goals.

I never said that the feature has to be coded explicitly. I said it has to be there.

OK. So it's just a matter of waiting for the desired capabilities to emerge in future models.

But they will probably thought models, not just language models.

The engineering will be different.

Possibly. I personally think it's the type of data and scale that're the primary differentiators. The use of characters is a fundamental flaw because characters are synthetic entities. Instead the models should be based on raw sensory data types, such as pixels and waveforms, and iterate from there on something close to the existing architecture.

You don't need to as long as you don't use LLMs like these in cases where incorrect output isn't of any consequence. If you're using LLMs to generate some placeholder bullshit to fill out a proof of concept website, you don't care if it claims strawberries have tails, you just need it to generate some vaguely coherent crap.

For things where factuality is even just a little important, you need to treat these things like asking a toddler that got their hands on a thesaurus and an encyclopaedia (that's a few years out of date): go through everything it produces and fact check any statement it makes that you're not confident about already.

Unfortunately, people seem to be mistaking LLMs for search engines more and more (no doubt thanks to attempts from LLM companies to make people think exactly that) so this will only get worse in the future. For now we can still catch these models out with simple examples, but as AI fuckups grow sparser, more people will think these things tell the actual truth.

Counterpoint: You absolutely can: https://claude.ai/share/d91f24f8-e174-41c9-bdbc-136ec10b7465

4 "u" because "b" felt like a bit of a cheat to count in that sentence.

> prompting LLMs with a warning like "You don't intuitively or automatically know many facts about...

We are not interested specifically in the inability to «know» about text: we are strongly interested in general in the ability to process ideas consciously, procedurally - and the inability to count suggests the general critical fault.

I often tell LLMs to ask questions if required, and that it is a skilled developer who is working along side me. That seems to help them be more collaborative rather than prescriptive.

I added something like that to my Claude project prompt and it can now magically solve fairly complex letter counting problems using the dashes method. For anything longer than a couple sentences, it's probably best to just have them write a REPL script.