I've been wondering how to argue within their frame for a while, and here's what I've come up with: Is the likelihood that aliens exist, are unfriendly, and AGI will help us beat them higher or lower than the likelihood that the AGI itself that we develop is unfriendly to us and wants to FOOM us? Show your work.
If we were going slower maybe it would seem more theoretical. But there are multiple Manhattan-Project-level or (often, much) larger efforts ongoing to explicitly create software and robotic hardware that makes decisions and takes actions without any human in the loop.
We don't need some kind of 10000 IQ god intelligence if a glitch token causes the majority of the labor force to suddenly and collectively engage in sabotage.
If any of this can make decisions in a way that a human can, then I would start to question what human decision-making really amounts to.
Even the oracle version is already really dangerous in the wrong hands. A totalitarian government doesn't need to have someone listening to a few specific dissidents if they can have an AI transcriber write down every word of every phone conversation in the country, for example. And while it's certainly not error-proof, asking an LLM to do something like "go through this list of conversations and flag anything that sounds like anti-government sentiment" is going to get plenty of hits, too.
We already have widespread humans making their own decisions autonomously that aren’t in the best interest of humans, and we’re all still here.
I think philosophy is a noble pursuit, but it's worth noting how often people drew very broad conclusions, and then acted on them, from not very much data. Consider the dozens of theories of the constitution of the world from the time of the Greek thinkers (even the atomic theory doesn't look very much at all like atoms as we now understand them), or the myriad examples of political philosophies that ran up against people simply not acting the way the philosophy needed them to act to cohere.
The investigation of possibility is laudable, but a healthy and regular dose of evidence is important.
And in fact even for myself - perception is not necessarily valid evidence since perception can be distorted. If I am in a compelling VR game I might be more than willing to swear to the fact that I'm flying (if I wasn't otherwise aware of the situation) - while you simply look at me acting a fool standing still while vigorously flapping my arms.
The programming analogy to this kind of philosophy is writing design docs (or building a class hierarchy of abstracts) without ever writing implementation. Lots of work, but why should anyone outside the room care?
I also think this is for the best. If one looks at the evidence of the skies it's indisputable that humanity lies at the center of the cosmos, with everything revolving around us - which, in turn, naturally leads into religious narratives around Earth. It's only thanks to these weirdos that adopted quite absurd sounding (for the time) systems of views and values, completely without meaningful evidence at first, that we were able to gradually expand, and correct, our understanding of the universe.
And this doesn't just apply to the ancient past. Einstein's observation that the speed of light stays fixed while the rate of passage of time itself is variable, to enable the former, sounds so utterly and absolutely absurd. In many ways the real challenge in breaking through to relativity wasn't the math or anything like that (which, in fact, Einstein himself lacked when first developing the concept) but accepting that a concept that sounds so wrong might actually be right.
The difference, to my mind, between science and philosophy is that the philosophy of relativity can exist independent of such confirming evidence. But the scientific theory cannot.
It's always important, when dabbling in philosophy, to keep track of when one has wandered so far out on the branch of reason that one is no longer supported by the trunk of evidence.
""""Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing.”"""
... and, of course, the philosopher can argue that we are all of us tumbling through a Pachinko-machine of parallel universes, and it is only our constrained perception that suggests Man is dead; in a parallel universe, Man is just fine and living their days in the grey universe that we, through some as-yet-unphilosophized-but-don't-worry-we're-thinking-hard-about-it process, cannot exist in.
... but does that matter over-much if the observable is that ignoring zebra crossings gets you crossed out of this universe?
“Philosophy is poor at revealing truths but excellent at revealing falsehoods (or at least unsupported arguments)” was the main lesson I took from informally studying it.
the cynics, though, made a good life but that’s not because they had a better philosophy. it’s because cynicism is base/primitive logic available to the brute as well as the civilized man.
To summarize one of her points from memory, she basically lays into Rousseau about having a lot of opinions on domestic life for someone who clearly doesn't even know how to set foot in a kitchen.
> The mugger argues back that for any low but strictly greater than 0 probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet.
This is a basic logic error. It ignores the very obvious fact that increasing the reward amount decreases the probability that it will be returned.
E.g. if the probability of the reward R being returned is (0.5/R) we get "a low by strictly greater than 0 probability", and for that probability there is a (different) finite reward that would make it rational to take the bet, but it's not R.
This is even simpler and more stupid than the "proofs" that 0=1. It does not change my opinion that philosophers (and lesswrong) are idiots.
The rationalist community both frequently makes this reasoning error and is aware they frequently make this error, and coined this term to refer to the category of reasoning mistake.
That's not at all what the Wikipedia article for it says. It presents it as an interesting paradox with several potential (and incorrect!) "remedies" rather than a category of basic logical errors.
It’s a mental tease, not a manual on how to react when faced with a mugger who forgot his weapon at home but has an interesting proposition to make.
Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”.
It’s what the comment here [0] says. If you try to analyze everything purely rationally it will lead to dark corners of misunderstanding and madness.
So what? It's still presented as if it's a interesting problem that needs to be "remedied", when in fact it's just a basic maths mistake.
If I said "ooo look at this paradox: 1 + 1 = 2, but if I add another one then we get 1 + 1 + 1 = 2, which is clearly false! I call this IshKebab's mugging.", you would rightly say "that is dumb; go away" rather than write a Wikipedia article about the "paradox" and "remedies".
> Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”.
It absolutely wouldn't. I don't know how anyone with any morals could claim that.
There are people who die waiting for organ donors, and a single donor could match multiple people. We do not find an appropriate donor and harvest them. This is the trolley problem, applied.
My actual words quoted below give one extra detail that makes all the difference, one that I see people silently dropped in a rush to reply. The words were aimed at someone taking these problems in a too literal sense, as extra evidence that they are not to be taken as such but as food for though that has real life applicability.
> the trolley problem isn’t really about real humans in that situation
I don't think that matches the trolley problem particularly well for all sorts of reasons. But anyway your point is irrelevant - his claim was that the trolley problem isn't about real humans, not that people would pull the lever.
Edit: never mind, I reread your comment and I think you were also agreeing with that.
Is it though? Let's look at the comment [0] written 8h before your reply:
> the trolley problem isn’t really about real humans in that situation
As in "don't take things absolutely literally like you were doing, because you'll absolutely be wrong". You found a way to compound the mistake by dropping the critical information then taking absolutely literally what was left.
The reason is that murdering people for organs has massive second-order effects: public fear, the desire to avoid medical care if harvesting is done in those contexts, disproportionate targeting of the organ harvesting onto the least fortunate, etc.
The correct answer in case of it being about real people, of course, is to switch immediately after the front bogey makes it through. This way, the trolley will derail and make a sharp turn before it runs over anyone, and stop.
The passengers will get shaken, but I don’t remember fatalities being reported when such things happened for real with real trams.
An Eastern European solution is to get a few buddies to cut the electrical wire that powers the trolleys and sell it for scrap metal, which works on all electrical trolleys. (After the trolley stops, it can be scavenged for parts you can sell as scrap metal, too.)
Made me chuckle. Funny 'cause it's true. About the trolley problem, if taken literally (people on tracks, etc.) pulling the lever exposes you to liability: you operated a mechanism you weren't authorized to use and for which you had no prior training, and you decided to kill one innocent person that was previously safe from harm.
Giving CPR is a very tame/safe version of the trolley problem and in some countries you're still liable for what happens after if you do it. Same when donating food to someone who might starve. Giving help has become a very spiny issue. But consciously harming someone when giving help in real life is a real minefield.
P.S. These philosophical problems are meant to force a decision from the options given. So assume the the problem is just a multiple choice one, 2 answers. You don't get to write a third.
I know about it. And yet I refuse to play the game. The problem is that even philosophers should be able to acknowledge that in the real universe, no box should be too big to prevent from thinking outside of it.
Otherwise we get people who conflate map with the territory, like what this whole comment thread is about.
Points 1 and 2 only explain how they are able to erroneously justify their absurd beliefs, they don't explain why they hold those beliefs.
Points 3 through 5 are the heart of the matter; egotistical and charismatic (to some types of people) leaders, open minded, freethinking and somewhat weird or marginalized people searching for meaning plus a way for them all to congregate around some shared interests.
TLDR: perfect conditions for one or more cults to form.