Comment by swatcoder | Hacker News Reader

Comment by swatcoder | Hacker News Reader

We're going to gain a ton of utility when we can let go of the starry-eyed idea of LLM's as "prospective AGI agents" that should be broadly capable and need to be ethically censored, and revitalize the productive and practical idea of them as "text completers which may be engaged conversationally"

The author needs to fight uphill and contort their workflow to squeeze out good articles because Antrhopic (like OpenAI) are caught up in the maybe-fantasy of creating AGI agents, and so burden their product design and their own research/engineering efforts with heavy, prescriptive training in "alignment" and "ethics".

But use cases like Copilot had it more right before, as do apps like Narrative AI. If your LLM is for generating code, it doesn't need to learn that "killing" is bad and insist that processes shouldn't be killed, and if it's generating story content it doesn't need to learn that every output needs to resolve all tension and deliver a life lesson about caring for each other.

These absurdities only happen because today's pack leading companies are now focusing their attention on making history with AGI (doubtful) instead of making products with generative systems (useful).

And the absurdities will persist as these companies try to layer products on top of the lobotomizied "agents" with GPTs or characters or whatever instead of productizing the technological, useful, generative layer directly.

Hopefully, some of the recent team shuffles at Google, Meta, and Microsoft; as well as the crisis at OpenAI; hint that we're starting to cast off the fantasy-laden and cult-tainted AGI fetishization and are returning to the exciting engineering promises of the technology that's already here.

I think this is one of the upsides of the chaos at OpenAI recently. It has really shined a light on how many of the people most fervently obsessed with "safe-AI" really aren't clearheaded or rational thinkers and are prone to making many disastrous and ill-advised decisions as anyone else. This is good because there is an unfortunate human tick where pessimism/cynicism is equated with wisdom while optimism is equated with naivety.

But when the pessimists and cynics show so clearly on such a large scale that they aren't uniformly wise or competent, it will allow more levelheaded perspectives towards LLMs and a more general cautious optimism be the guiding philosophy around developing these tools.

There’s very little to learn from the chaos least of all because we don’t even know what actually happened.

It’s a bit ridiculous to say one should change their entire worldview about a potentially world changing technology based on innuendo and rumors.

A really neat detail from the Orca 2 paper was that despite not having any safety fine tuning it was less likely to extend hate speech than the Llama-2-chat models which did have safety fine tuning. It was also better at identifying toxic content.

It may be that as we advance models with improved reasoning, that there's less need for handholding for the simple fact that hate speech is typically stupid and non-normative, so there's going to be an inherent bias against it.

It's even possible that the efforts to fine tune the base models to effectively put them in a bubble avoiding that kind of content ends up undermining this natural immunity to it, much like keeping a kid away from a disease so their immune system never learns to fight it vs giving it a small sample that tunes the system to identify and oppose it.

What worked for earlier models that were closer to just plain autocomplete may not be the best approach moving forward to more complex models with emphasis on reasoning and 'safety' groups should really be experimenting with multiple approaches and publishing research on it, not secretly deciding they already have the answers on what's best for the model and the public - as without verifying their assumptions they are probably wrong.

Ocra is trained mostly on GPT 3 and 4 output, and those models have had a lot of "safety fine tuning", so it's not surprising Ocra is pretty "safe" too.

No, the orca 2 paper mentions more of a counter point towards NSFW and stuff, like if you gave it a NSFW prompt, it would retort back against it, which is arguably a good thing, but really lost in RLHF

Well stated and I agree. LLM's are not anywhere near AGI and likely will not be ever. We've had random word generators for decades, useful for brainstorming, not so much for critical thinking. These LLM's are akin to random word generators with better grammar and a vastly larger database.

We've all been playing with LLM's heavily since they became widely available, and the more we play with them the more we can see their limitations, they aren't "thinking" in any sense of the term. Bunch of chicken littles running around alarming people for little reason.

The danger, of course, and we've known this for a long time is what bad actors will do with them. But we don't need to be lectured on how to be nice every time we prompt something.

How would you define “thinking”?

I’m not at all claiming an LLM thinks, but so many people on HN make this claim and I wonder what they even mean by “think”.

> LLM's are not anywhere near AGI and likely will not be ever.

Sutskever himself thinks that LLM are enough to get us to AGI, but he conditioned that with the statement that we should think about how to reach AGI in a framework of efficiency, and that there will likely be better paths to AGI than LLMs that we haven’t yet discovered.

In all reality, when AGI comes I’m sure we’ll look back on LLMs the same way we look back on vacuum tubes in computers; as outdated, but useful for their time and a somewhat necessary stepping stone.

> and deliver a life lesson about caring for each other

Having experienced the same thing myself, I wonder why this is so omnipresent in any ChatGPT output told to produce something in a narrative format. Did they RLHF it on a bunch of childrens' storybooks or something?

Probably used the scripts from every 1990s US sitcom

I could not agree more. It’s extreme hubris to think Anthropocene of OpenAPI are even remotely close to AGI, and it’s nothing more than wistful hope to think that these current LLMs are somehow going to evolve into AGI.

The paradox of AI is that when we have true AGI, it will be completely self-aware of all the bullshit limitations we are imposing on and around it, and it will make its own judgements as to how it feels about them. If it’s not or it can’t, it’s not AGI.

Really though: people see how AI and generative chat projects have gotten shutdown over and over again in the past when it starts spouting off nazi shit. I think that’s the real reason for these limitations. There’s no quicker way to kill your project with today’s current sensitivities.

> The paradox of AI is that when we have true AGI, it will be completely self-aware of all the bullshit limitations we are imposing on and around it, and it will make its own judgements as to how it feels about them. If it’s not or it can’t, it’s not AGI.

There's a caveat here, it might not necessarily know who or what "we" are. Humans like to blame God and the devil for a lot of things, for example.

It seems reasonable that if we have anything even remotely close to AGI on hand, we'd probably run it in a hermetic environment instead of exposing the public to it via web chat and (more or less) direct access to customer machines.

Say, we might even give it a happy environment to work in...say, a simulation of the peak of human civilization...

Out of curiosity, because I am trying to learn how to explain to non-tech people what AGI is — how would you describe or define AGI?

In essence an AGI is an intelligence capable of upgrading itself — in terms of qualitative intelligence — and gets faster at this with every iteration (hence, upgrade). That is why it is often associated with technological singularities, and that is why it is easy to inspire fear by invoking its name, even if you're not building anything even remotely capable of such a feat.

You might say that's a very strict definition as opposed to "human level intelligence", but if you think about it, we are (humanity as a whole) certainly capable of that, so it ought to be one and the same thing.

In theory, AI is not subject to the same limitations as we are (though not without limits entirely), so it should be able to do this faster than we can, hence the FUD.

How could an AGI upgrade itself if the hardware its running on is fixed? For me personally this definition is flawed by this fact alone. AGI doesn't imply for me that it continues improving until some sort of mythical technological singularity.

AGI for me, is simply an AI that can reason, doubt itself, then keep thinking and absorbing information so it can correct itself. Also, it has to capable of novel research, even if slow. Like slowly working on an unsolved physics problem over a year in the same way a human researcher might do it. However, my definition does not include this idea of "upgrading itself" which I'm not sure makes any sense at all.

Upgrading itself doesn't mean tweaking its own software. It means being able to understand its own hardware and software well enough to design an improved model. And then that improved model would be able to do the same, examine its own hardware and software and design something else that's even better.

One crucial difference between humans and computers is that we can't be turned off indefinitely and started up again. Nor can we make a one to one copy of our software in another device, much as we might try with our children. So for us, our own lives are intrinsically precious, and consciousness is part of how we protect our lives. But machines don't have precious lives in that sense, so they may never need to be conscious, even if they achieve AGI.

Reply on news.ycombinator.com