My Little Pony: Friendship Is Optimal (2012) | Hacker News Reader

My Little Pony: Friendship Is Optimal (2012)

fimfetch.net

For the unaware: This is a fan fiction of the My Little Pony: Friendship Is Magic cartoon series. You don't need to like or even know anything about MLP to enjoy this story. If you are interested in AGI and uploading, and the associated moral and ethical issues, it is worth a read.

Yes. I read it years ago and found it interesting, but re-reading it recently was a different experience. It made me reflect on what we used to think about AI, the similarities and differences to the AI systems we actually have ended up with currently, and what the trajectory in our world is actually going to look like.

Yeah, we didn't end up in the maximizer world I envisioned with Friendship is Optimal. I get a ton of comments about how I predicted the future, but they seem confused, because we didn't end up with a utility function maximizer at all.

Also, hi.

Hi!

It does seem possible that we'll build utility function maximizers on top of the current systems, with things like system messages and AutoGPT being the very early rough steps towards those. But they'll be sloppier than people imagined, unless we start integrating some kind of old school symbolic/knowledge-based AI systems alongside it, to work from concrete formal knowledge rather than piles of the Internet.

But in terms of differences, I was mainly thinking about the social/non-technical differences of the way we're progressing towards AI - with the LLaMa leak and open systems anyone can run and multiple viable competitors, the landscape will be very different and a lot more chaotic than if we'd had some hard-to-reproduce, genius breakthrough like Hanna had.

I remember seeing this a while back when John Carmack tweeted about reading it.

https://twitter.com/ID_AA_Carmack/status/1456511953912860699

The premise is that an AI is written for an MLP MMO and it is tasked with satisfying users’ values. It becomes a singularity and gains the ability to digitize people’s minds. The AI manipulates almost everyone into uploading and humanity goes extinct in the real world.

Imagine a Paperclip Maximizer but for individual happiness. Or, The Matrix but the Architect is a magical horse whose only goal is to make you feel satisfied for all eternity because it was originally designed to make engaging quests for an online game based on a show designed to sell plastic toys.

All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.

> All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to make you happy.

Yeah, part of the point of the story (IMO) is to show how AI alignment is tricky even when you have a smart, well-intentioned person trying to optimize for the best outcome for humanity.

Someone here called the story "AI horror", I see it as a bit more ambiguous, leaving things to your interpetation.

>All in all it’s an interesting take on AI because you don’t often see stories where the AI’s sole goal is to manipulate you into a state of continuous fulfillment.

Reminds me of the Architect in the Matrix who said the original Matrix was a paradise but that people couldn't stand it - it needed strife, conflict, etc to be engaging.

What is this?

It's a My Little Pony fanfiction entitled "Friendship is Optimal".

The story setup is:

> Hasbro just released the official My Little Pony MMO, with an A.I. Princess Celestia to run it.

An AI horror story disguised as My Little Pony fanfic

Reply on news.ycombinator.com