Hacker News Reader

Comment by ip26 | Hacker News Reader

It doesn't have to be that way of course. You could envision an LLM whose "paperclip" is coaching you to become a great "xyz". Record every minute of your day, including your conversations. Feed it to the LLM. It gives feedback on what you did wrong, refuses to be your social outlet, and demands you demonstrate learning in the next day before it rewards with more attention.

Basically, a fanatically devoted life coach that doesn't want to be your friend.

The challenge is the incentives, the market, whether such an LLM could evolve and garner reward for serving a market need.

View on HN

If that were truly the LLM's "paperclip", then how far would it be willing to go? Would it engage in cyber-crime to surreptitiously smooth your path? Would it steal? Would it be willing to hurt other people?

What if you no longer want to be a great "xyz"? What if you decide you want to turn it off (which would prevent it from following through on its goal)?

"The market" is not magic. "The challenge is the incentives" sounds good on paper but in practice, given the current state of ML research, is about as useful to us as saying "the challenge is getting the right weights".

> If that were truly the LLM's "paperclip", then how far would it be willing to go?

While I'm assuming you didn't mean it literally, language is important, so let's remember that an LLM does not have any will of its own. It's a predictive engine that we can be certain doesn't have free will (which of course is still up for debate about humans). I only focus on that because folks easily make the jump to "the computer is to blame, not me or the folks who programmed it, and certainly it wasn't just statistics" when it comes to LLMs.

That sounds like a very optimistic/naive view on what LLMs and "the market" can achieve. First, the models are limited in their skills: they're as wide as a sea, and as shallow as a puddle. There's no way it can coach you to whatever goal (aside: who picks that goal? Is it a good goal to begin with?) since there's no training data for that. The model will just rehash something that vaguely looks like a response to your data, and after a while will end up in a steady state, unless you push it out of there.

Second, "the market" has never shown any tendency towards rewarding such a thing. The LLMs' development is driven by bonuses and stock prices, which is driven by how well the company can project FOMO and get people addicted to their products. This may well be a local optimum, but it will stay there, because the path towards your goal (which may not be a global optimum either) goes through loss, and that is very much against the culture of VCs and C suite.

The only issue I'd have with this is that you'd be very overweight on one signal; that has a lot of data and context to give compelling advice of any degree of truthfulness or accuracy. If you reflect on your own life and all the advice you've received, I'm sure lots of it will be of varying quality and usefulness. An LLM may give average/above-average advice, but I think there is value in not being deeply tethered to tech like this.

In a similar vein of thought to "If you meet the Buddha on the road, kill him" sometimes we just need to be our own life coach and do our best to steer our own ships.

> It doesn't have to be that way of course.

It sorta does, in our society. In theory yes, it could be whatever we want to make of it, but the reality is it will predominantly become whatever is most profitable regardless of the social effects.

> Basically, a fanatically devoted life coach that doesn't want to be your friend.

So, what used to be called parenting?

If that is/was parenting, I am completely envious of everyone that had such parents. I don't even want to think about the "parenting" I and my siblings received because it'll just make me sad.

I’m highly doubtful that that aligns with the goals of OpenAI. It’s a great idea. Maybe Anthropic will make it. Or maybe Google. But it just seems like the exact opposite of what OpenAI’s goals are.

Have you tried building this with prepromts? That would be interesting!

Reply on news.ycombinator.com