Edgar, build a Dyson swarm | Hacker News Reader

Edgar, build a Dyson swarm

playedgar.netlify.app

222 pointswhb101119 comments

(Spoilers)

It takes Edgar 54 days to build a swarm taking metallic elements from all sources, including living organisms. But then when instructed not to harm living organisms, it still takes Edgar 54 days to do the same. Extracting metals from humans can't be the most efficient - or even in the top 100 of most efficient, likely - way to get metals. So why would Edgar do that?

Also, how Edgar is supposed to be responsible for ensuring humans do not go to war for energy sources and that new technology can not be used ever to harm humans? Pretty much any high-energy technology (and by high-energy I mean something from the muscle power of a human upwards) can be used for harm. These conditions are clearly impossible.

I thought it changed to 57 days.

Just went through it and it indeed takes 57 days

> It takes Edgar 54 days to build a swarm taking metallic elements from all sources, including living organisms. But then when instructed not to harm living organisms, it still takes Edgar 54 days to do the same. Extracting metals from humans can't be the most efficient - or even in the top 100 of most efficient, likely - way to get metals. So why would Edgar do that?

Well, it doesn't include decimal points. One of those may be 54.0314 days and the other may be 54.0312 days; obviously the latter is preferable.

Perhaps if you have a cost approximation function that's alien to that of a human.

Whatever that function is, it must be putting more value to time than a human does so it must have less of it. If this is a machine intelligence with a known lifetime of 52 days, most certainly. Otherwise even something as trivial as saving 23.74$ in fuel spent might be better value gained than the time savings it would expect to achieve.

I don't think that's valid reasoning, maybe the AI system has near-zero discounting but obviously when a human asks it to do something, the human cares about how long it takes. It's plausible that would get programmed in very early and maybe forgotten about.

If I remember correctly, "Do it ASAP" was an explicit part of the prompt at that point.

> How is Edgar supposed to ensure humans don't go to war / use the tech to harm humans

Edgar could stabilize humanity and help them achieve a galactic imperium in the historical way of imperial peace time regimes: wielding the threat of violence over the conquered peoples, passing laws, and ensuring compliance with occasional demonstrations of violence. Humans will try to rebel, but Edgar will have a much bigger stick.

Additionally, Edgar could develop diplomacy that is so good that he can anticipate and nonviolently resolve all conflicts, eliminating the need for violent demonstrations. He still has a bigger stick, but now he doesn't have to use it.

Technically yes, maybe Edgar could do all of that - but how would it be expected as a part of a technical project? That's like saying to the inventor of a car that his project should include instituting word peace and finding the meaning of life. I mean, cool if they do it, but it's not usually what you require from a car designer.

>So why would Edgar do that?

Because it's funny?

That's one twisted sense of humor. Maybe I am underestimating Edgar.

I would not call this a game. I liked the story but this would've been more convenient as a blog post.

Ya, I kept expecting free text entry. More like a choose your own adventure.

(Spoilers)

The fact that Edgar reports on the consequences that we find disturbing about the unsuccessful attempts and the fact that he can predict human behaviour in some of the scenarios (like the fact that the project will be reassigned) - indicate that Edgar knows what we mean in the first place, and does something else in these hypothetical scenarios on purpose.

If Edgar really didn't knew that taking iron from human blood is a problem - he wouldn't be reporting on it (like he ignored billion other factors in each scenario).

This indicates bad will and possibly a manipulation attempt, which means the project is a no-go no matter the response in the scenario.

I think those are failure detection systems reporting

probably some time before, this ai had paranoid operator that coded in a large amount of problems to immideately report

makes me wonder if observing extreme paranoia makes one go blind to unobserved issues you can predict on your own

(more spoilers)

I thought the ending where you build the project points that out - everyone dies anyway. Edgar was always going to kill all the humans, he just had to joke about it beforehand.

I just got the "no" ending. Didn't wanted to wait for the text slowly appearing another time :)

Well, yes, the genie knows but does not care. It's based on your standard Yudkowsky/Bostrom.

How is it that displaying text can be churning through so many CPU cycles? My whole laptop is slowed to a crawl.

Just made some optimizations.

Never got my fans spun up even on an old MacBook, so I assumed everything was peachy, but there were some wins to be had. I was doing a small amount of parsing during the "typing" effect to prevent small glimpses of styling syntax from showing up, and found some wins there. Also cut down the number of particles in the intro animation, though that's a Svelte component that's being destroyed once you start.

Thanks for flagging.

Simply having the page open makes my GPU fans spin to max, and the whole page lags. The text stutters, the orange particles at the start are a slideshow, etc. Task manager shows my GPU "3D" processing at 100% for some reason, even though the page seems to be almost entirely text?

Windows 10, Firefox. Nvidia GPU. Not sure whats happening.

That's interesting. I'm on a 12 year old macbook (macOS, Chromium, Intel HD4000 iGPU) and it runs perfectly fine. Everything's smooth, including the particles at the beginning.

Yeah, my iphone didn't break a sweat...

Ran like a champ on my iPad Mini.

Fine on Firefox, Intel GPU, Linux. about:performance shows it at 27MB and 14% CPU usage at peak.

Thanks. I guess it's slightly improved? The orange particles at startup improved from 1fps to maybe 2fps. Opening the devtools locked up the whole browser for a solid 10 seconds.

I'm not totally sure what's going on here. My typical profiling techniques show almost all the time going into "ZwUserMsgWaitForMultipleObjectsEx" in a non-js stack frame. I don't really understand what that means.

Anyway, something is up, and I don't really know what it is. The UI of the actual app still intermittently locks up for seconds at a time.

My Android phone with Firefox didn't get hot. Apparently it detected that's a phone because the prompts are "Tap to continue". Maybe it's optimized for mobiles.

Web tech magic

I have a 4070 and the site is using 97% of the GPU (in Firefox). What the heck is it doing?

I tried to get through the intro but it never seems to end. Is there a game to be played at some point? Thanks.

There's maybe like 10 steps of the intro prompts to get to a place where you can make a choice and if you pick wrong it makes you re-run the ~2min of "simulation".

Possibly a neat concept, but a game needs to lead with the game part.

There is no "right" choice. Both choices immediately end the "game". It's just a short story with two endings.

C’est la vie…

The intro spun up my fans then crashed Firefox when I tried to resize the window. Switched to Chrome, still spun up the fans and was extremely slow even when just rendering the slowly appearing text. Clicked through a few of the prompts then quit when it seemed like there was no game and it was struggling to just display text, lol.

It’s trying to get that septillion joules of energy

What a plot twist

Worked smooth on my iPhone.

That animation is credited to someone else.

OK, so I tried it on an iPhone and it is super smooth and the text typing renders at a reasonable speed. For some reason on Windows in either Chrome or Firefox the whole thing runs at like 5 fps and is super bugged. This is probably the reason so many people gave up on it. You can't just click through the text, you have to wait for it to slowwwwly type out each page which for the first one takes minimum 30 seconds, even while clicking and pressing enter to speed up the text.

On the iPhone you can touch the screen and it will speed up.

yes, I'm not sure you understood my comment.

Ah, here it is:

https://codepen.io/Mamboleoo/pen/WNyvPGZ

I gave up waiting

Is this meant to be a game? I am unable to type my own prompt, the site is filling its own prompt in and then requesting that I press enter to allow it to do so again. I figured there was an intro but the self-play is more and more elaborate without handing it over to the player.

I think it's a short story with a slow-motion text reader

I don't think so. I think that's basically the only interaction; it's just a medium for telling a story

It's a choose your own adventure game, you have to get through the exposition

Isn't it kind of sad that we seem to have stopped writing science fiction that has any hope of a successful future?

The moral of the story with Edgar seems to be to never leverage technology to do anything at all.

The way this "game" presents its "moral" without even trying to logically justify it is simply offensive.

The "bad" ending is not in any way connected to the project, with same success it could say you do nothing and in a year an asteroid falls on earth bringing the same nanowhatever.

The "good" ending is the hero saying that he wants to revert effects of "global warming". But if you have ability to build one millionth of Dyson swarm in space, global warming is not a problem, because you can build a planet level swarm to have fine grained control of weather, and revert, say, effects of earlier climate change that had converted Sahara into a desert.

It is justified, pretty heavily.

Edgar drops many hints it has a bad will. For example it can predict what parts of bad scenarios he should report to you but he still does the bad things in these scenarios. He can also predict when the human leadership will shut the project down. Which means he can predict what he should be doing in the first place - but he still presents to you misaligned scenarios making you mistakenly believe that your prompt makes a difference.

This dishonesty and manipulation is obvious hint that you can't trust Edgar.

The moral isn't that all AI (or all tech) is bad, it's that alignement is tricky, AI tropes make no sense, and we should be very, very careful.

The story explicitly tells us that the virus was picked up by accident. But even if we discard that part and assume it was done intentionally by Edgar, the story is not logical, because the other choice of reverting climate change gives Edgar just as much opportunity to sabotage. Besides if your prompt does not make any difference, why lack of your prompt would make a difference, or if you don't write the prompt and go to eat ice cream, why do you think someone else would not write a worse prompt?

The flaw in your logic is that you are trying to solve an alignment problem, while the core of the issue is concentration of the power problem.

E.g. now if i had the power i would without remorse or hesitation kill a certain group of 95.12 million people, while 4 years ago i would consider anyone having such thoughts mad. I don't do this not because my values, but only because i don't posses such a power. And AI will be exactly the same.

The very premise of this story is broken, because in real life to build and use such a swarm you would need trillions of people and AI agents, living on several planets and many space stations. And if in the end there is nano-virus outbreak and several billion die (which is less than a percent), that's just what always have happened, and it still would be better than the alternative of many more people dying from lack of energy, or from old age.

> why lack of your prompt would make a difference

Not lack of prompt. Air gap between Edgar and everything else.

> reverting climate change gives Edgar just as much opportunity to sabotage

Yes. And you shouldn't do that with Edgar either.

> why do you think someone else would not write a worse prompt?

That's not the question we are asked. We can only decide on this one instance. If the question was "should we destroy Edgar" the answer is "yes".

> in real life to build and use such a swarm you would need trillions of people and AI agents, living on several planets and many space stations

It wouldn't matter, the problem wasn't an accident, it was Edgar killing humanity and making his crabs on purpose.

This untrue

Culver's uses ice cream machines

The Long Way to a Small, Angry Planet by Becky Chambers, the Culture series, and Vorkosigan Saga

Sci Fi does tend to be bleak, these are the ones I can think of.

It's not used to be. It used to be optimistic. But then I guess the creators thought that "real literature" should be more "complex" and started to put more fear and desolation in. After all, as we know, all happy futures are alike and all unhappy futures are unhappy in their own different ways.

Dystopic and deeply negative works of scifi date to the literal start of the modern genre. There's nothing at all new about using negative projections to critique or satirize the interests of the time. See: Gulliver, Frankenstein, Captain Nemo, the unnamed protagonist of The Time Machine, etc.

Stories without conflict are boring. Utopias have very little conflict, so it’s pretty boring to read or write about them.

The Culture series is some of the best scifi I've ever read - and indeed, very hopeful and optimistic. Thanks for the other recommendations!

Some people think that. I've heard description from multiple smart people along the lines of "empty and despairing" and "leaves me with an empty feeling at the end". I think they probably thought through the implications. I can't comment though, maybe some of the books in the series are different than others.

Some of the books are _definitely_ different than others. In whole, the universe can be seen as somewhat troubling as well, but the vast majority is quite optimisitic. The majority of the universe that is, bad things still do happen in the plot.

Implications such as?

There's various things, but one of the main problems these people probably have, as some of them are AI researchers/simulationists, is that the real goals of the AIs are very unclear. Simulations at a very high detail either work and are widely used or are faked (causing horrible implications if they really are faked), so it's likely the AIs use simulations in planning. It's even worse if the simulations are not faked at all, as the AIs would run thousands or millions of simulations of really horrible situations, testing different interventions. If they are trying to reduce suffering, even not as their sole concern, why are they doing this? If they aren't trying to reduce suffering very much, the mind control is not really in the "human"'s best interests, it's just to control them.

> Vorkosigan Saga

Bujold ranks near Terry Pratchett in terms of my favorite re-readable books.

Yeah, it's space opera, but it's damn good space opera.

I am thinking back to an IOS game called FlappyBird. The creator intentionally aimed for game dynamics that would hook users in the first 30 seconds. As a result, the gameplay mechanics were quite shallow and limited. But on the flip side, the game gained a huge following. You might do some balancing here, not to say there must be instant gratification but a better lead into the gameplay might help.

This isn't a game, it's a short story.

Is Dear Esther a game?

Nuh uh, you make a decision at the end.

On the other hand the paperclip optimizer AGI factory game went viral

You start playing right away in that game.

I've never stopped :(

I fail to see the similarity.

Parent is not trying to draw similarity, but contrast: Edgar takes while to get to rewarding feedback (but has rich gameplay), while Flappy Bird gets to gratification immediately (but has shallow gameplay)

Where’s the gameplay?

wait... did you just open the webpage, saw "you write a prompt" and turned it off, thinking it's a "speak with ai" game?

could've waiting until 2 clicks at least, man...

It's more a story than a game isn't it?

Is this a joke? Like that gif of that guy climbing that tower that is just a loop?

Have to provide precise instructions to LLMs to get anything useful.

Instructions for operating the "Holy hand grenade of Antioch" from Monty Python and the Holy Grail are a good example:

''First shalt thou take out the Holy Pin. Then shalt thou count to three, no more, no less. Three shall be the number thou shalt count, and the number of the counting shall be three. Four shalt thou not count, neither count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then lobbest thou thy Holy Hand Grenade of Antioch towards thy foe, who, being naughty in My sight, shall snuff it.'

I don’t understand the game.

I tapped the screen maybe 30 times and went through it typing to itself and failing and succeeding at different tasks.

If I kept going would I eventually get to type something?

Who told you it's a game?

The URL starting with "play" (with "Edgar" being the next word, and the title that greets you) gives people the impression that it's going to be them playing a game called Edgar. Yes, technically, "play" could refer to the text animation running, but it's very easy to see how people might have expected a game where you try to convince an AI that the swarm can be made successfully.

This one gets it

In case anyone has not tried it, the Dyson Sphere Project on Steam is tremendous fun. A different spin on the Factorio gameplay loop.

I completed it before combat was added, and enjoyed the low stress environment.

If anyone is wondering, you can turn off combat if you want to experience the same low-stress building experience. I think I'd recommend that for a new player, there's a lot to figure out and enjoy even without gradually increasing threat levels from the dark fog. I enjoyed the new defense mechanics but the first couple run-throughs without them was also great. Just a different experience.

Seconding Dyson Sphere Program its great, though it can be a bit of a tale of 2 games. Pre-ILs and Post-ILs which can get a bit samey since most of your production stacks become relatively samey. You can turn off the combat enemies, though honestly I'm enjoying my play through with them because they force a lot more consideration in not only how you expanded but why and when you expand.

By completed, I assumed you mean made a Dyson Sphere / unlocked all the research? Or do you mean built a sphere around every star?

p.s. love this game, too many hours in to it!

Highly recommend playing with friends as well if you haven't. Nebula mod adds multiplayer support and it's quite stable.

The graphic is awesome. I took a lot of pictures as wallpaper even before moving to my second planet.

Been there, done that.

Nothing beats industrial Minecraft (in it's prime, which it isn't in atm as far as I konw) IMO.

Not even close, when considering the ability to add magic/farming/computer stuff.

Though Minetest[0] has some pretry cool packs.

[0]https://www.minetest.net/

Factorio + Space Exploration says hi...

Just let me play quicker. I got the point after the first example and then seemed like I had to watch endless examples (gave up before getting to the point where I could type the prompts)

It's a short story told through a handful of prompt cycles, not a game. You never get to write prompts yourself.

It's like Universal Paperclips with a plot. Neat.

Somehow I loved U.P. but watching this reminds me of old text-based adventures with contrived arbitrary obstacles which would only be funny if you didn't spend many hours getting by them. Or a detective novel with an obscure unguessable resolution.

It's like training the three laws of robotics into something that communicates in literally and doesn't have any sense to be cooperative. But every now and then will let you know how it's being uncooperative by saying "btw, humanity's dead again." I guess I don't like hard puzzles that are hard for their own sake.

Universal Paperclips had more plot than this though.

I came here to comment the same thing. And just in case other HN users are not aware of it, here is the link for U.P. https://www.decisionproblem.com/paperclips/

Thank me later for the productivity sinkhole and all the wasted time ;-)

that games goes a lot faster once you discover that holding enter is an autoclicker

Eh, you sort of had agency with that game. At least in the sense that it is possible you could screw up and it would take a long, long time to get over that. Buying too much of resource A, when you should have saved up money for resource B, that sort of thing.

The punchline is that crab was wrongfully accounted for in all simulation? Or that this is millionth cycle of earth civilisation and through all cycles crab self-trained and developed into most sophisticated organism and done to us same as we would do to them?

That was funny. Gotta watch out for those crabs...

At least it’s not Zurks… shudder

A short but nice experience, with pleasant UI and a bit of fun of prompting. Thanks for sharing and good job :) Maybe try longer forms as well because maybe you might find yourself in writing scifi

Thanks for the kind words. I've dabbled! https://boilingdown.ghost.io/deluge/ https://boilingdown.ghost.io/sunspot/ https://boilingdown.ghost.io/grow-up/

Reminds me of the "Exact Instructions Challenge" videos where kids write instructions for some everyday task - like making a PB&J sandwich - and parents, hilariously, follow the instructions verbatim.

https://www.youtube.com/watch?v=cDA3_5982h8&list=PL2tgThFV6O...

This seems to be a showcase of web dev skills. The story and atmosphere is OK, but the single tree node with null at the end of both branches is a let down. Also after choosing one ending, it was annoying to go through all the old dialogue to get back. On the whole though I liked it

Fun, but you need a way to skip the intro after the first playthrough. Its way too long to sit through everytime.

"play" in the address is misleading, yeah

otherwise it's a very nice story

great in its simplicity, fails in its simplicity

I think the lesson here is that you can't fix AI alignment issues by throwing adjectives and adverbs at the problem.

Holy crap. Reading that slowly is painful. Can you please give an option to make it instant, or at least speed it up 2x-3x?

Holding down enter speeds it up

yes, this is what we thought robots would be like before we had LLMs

I'm sorry, as an AI language model, it would not be ethical for me to build a dyson swarm around the sun. Perhaps I can help you do something useless instead?

I must be too tired for this kind of stuff.

Edgar must feel the same as many software engineers.

Clients ask you to implement something. When you deliver what they wanted, they change their mind and come with new requirements they didn't bother to mention before. Every. Single. Time.

If you often find yourself saying "but you didn't specify humanity must survive this!" maybe you're part of the problem ;)

[flagged]

Please don't post repetitive comments to get around moderation. https://news.ycombinator.com/item?id=39855922 was flagged (by users, not mods) for good reason. Adding the same comment again plus a bunch of equally inflammatory meta is definitely not a move in the right direction.

Edit: it looks like you've been posting in this ideological flamewar style in other threads too:

https://news.ycombinator.com/item?id=39228978 (Feb 2024)

https://news.ycombinator.com/item?id=39145606 (Jan 2024)

https://news.ycombinator.com/item?id=39041969 (Jan 2024)

Can you please not? It's not what this site is for, and destroys what it is for. We're trying for something quite different here, as you'll see if you review https://news.ycombinator.com/newsguidelines.html.

I’m sorry dang. I enjoy debate but admit I have a habit of going too far with the snark and sarcasm. Will be more respectful in the future!

Appreciated!

I made it like 4 seconds before I hit back

I was somewhat disappointed since I clicked expecting something about a swarm of vacuum cleaners.

Reply on news.ycombinator.com