xamuel

"Reward Is Enough" is an interesting paper, less for its content, more for its place in the zeitgeist or something. It has very little in the way of novel "theorem proving" but it has sparked quite a lot of discussion. It's a paper which manages to be really great in some sort of subtle way, just because it inspires so much conversation.

I'll plug my own response, "Can reinforcement learning learn itself? A reply to 'Reward is enough'", in which I ask: ok, if creating good enough RL agents can lead to AGI, and if AGI can do anything humans can, and if humans can write good RL agents, doesn't that seem to imply it should be possible to use an RL environment to incentivize RL agents to write good RL agents? https://philpapers.org/archive/ALECRL.pdf

Thanks, that's amazing! I wish there was more of a central forum for this stuff. Do you know of any similar channels for FF6, Chrono Trigger, and other golden-age JRPGs?

>new things are discovered almost every month - new game glitches, mechanics

Is there a good place to follow these discoveries? The site you linked seems to mainly just be mods, and when I search for glitch discussion I mostly just find generic wiki's listing the same old glitches known for decades.

The film you're thinking of is Sullivan's Travels (1941), a real fantastic film I'd highly recommend. The American Film Institute ranked it the #61 greatest American film of all time.

Yes, people completely misunderstand what the golden rectangle is about and why it's important. Its importance lies in the fact that it's canonical, a rectangle motivated intrinsically, independent of anyone's preferences (conscious or unconscious) or any experiment: the same golden rectangle would naturally be discovered by aliens on the other side of the universe, or even in alternate universes with totally different laws of physics. People looking for golden ratios in seashells and things like that haven't just missed the point, they've missed it by 180°, fired their arrows away from the target.

(Of course, there's an even simpler canonical rectangle: the square. But that one's too trivial to be interesting.)

That's one of the reasons it's so nice to use reinforcement learning agents as guinea pigs. I can "pip install" a bunch of industry-standard RL agents and put them through the most bizarre scenarios, scenarios it would never be possible to put a human through. No ethical issues involved, since these RL agents are nothing more than a bunch of matrix multiplications.

I think part of the problem might be people too often try to come up with some grand unifying theory all at once, without standing on the shoulders of giants. Sort of like trying to write a gigantic codebase all in one sitting (no test suite, no syntax highlighting) and then sending it off for code review without even running it.

This preprint my co-authors and I are working on might offer something of a babystep, not toward consciousness itself but toward the simpler problem of self-reflection, and more specifically of measuring self-reflection in reinforcement learning agents: https://philpapers.org/archive/ALEEET-3.pdf

The nightmare-inducing 1984 film "Threads" portrays a nuclear holocaust that begins with exactly these kind of small tactical nukes. <Spoilers>The nuclear conflict begins with the USSR using a nuclear-tipped SAM missile to wipe out a US bomber squadron. The US retaliates with a tactical nuke to wipe out a Russian base. After that, things get ugly fast.</Spoilers>

Augustine does explicitly reference Aristotle's "Categories" in his Confessions, I believe. Not sure what other material of Aristotle's was available to Augustine, I'm not a scholar on the subject. Of course, even to us today, many of Aristotle's works are STILL lost, so...

>Aquinas reconciled some Aristotelian and Neoplatonic views with the Catholic Church

I wonder if you might be mistaking Aquinas for Augustine? Augustine introduced huge amounts of Neoplatonism into Christianity very early---little more than a century after the original Neoplatonist, Plotinus himself. It's also worth noting that Plotinus was a disciple of the mysterious Ammonius Saccas, one of whose other disciples was none other than Origen, a very influential early Christian writer (albeit later considered a heretic).

Thanks, that's one of the best critiques I've ever heard of my paper.

One minor correction first: you're absolutely right that AGI is about comprehending the environment, not about perfectly solving all environments (the latter is mathematically impossible even with strong noncomputable oracles etc). I'm not sure why people so often come away from my paper thinking I'm saying AGI is supposed to solve all those environments, I never say anything like that. If I could go back in time, I'd make that clearer in the paper. No, it's about the AGI simply being able to comprehend the environments, like you say. And the thesis in the paper is that shoehorning general environments into real-valued-reward environments is a lossy process.

For the rest of your argument, you make a lot of good points. I would ask, what do you say in response to, e.g., Alan Turing who asks us to imagine Turing machines having infinite tape and running for all eternity? Obviously that too is impossible in the finite universe we live in. That's sort of the divide we disagree on. I'm talking about idealized AGI. If we consider human beings, humans have finite lifetimes so any particular human being's entire lifetime of actions could simply be recorded in a finite tape recording. But does that mean said finite tape recording is intelligent? In the idealized world, I would want to say it's a basic axiom that no finite tape recording of a human can be intelligent. But now we're deep in philosophical woods.

I like your point about musicians etc creating further and further derivatives of known successful work as opposed to starting anew :) I guess in terms of my paper, the real question is, if you confronted these derivative musicians with the grand new work that transcends them all, would they recognize it as such, or would they (like an AGI confused by rewards shoe-horned into real numbers) mistake it for something mediocre? Now we are deep in psychological woods!

>Is there some other way that we are do it beside our biological reward system?

Seems to me that's what this whole paper we're discussing is about. If you're already convinced that there is no other way, then you're basically already agreeing with the paper, "Rewards are enough".

>I'm asking what's a behaviour you can't replicate using a reward system based on real numbers

So glad you asked! I can give an answer which people will love who take the necessary time to understand it. It's complicated, you might have to re-read it a few times and really ponder it. It's about automatic code generation (though it might not look like it at first).

Definition 1: Define the "Intuitive Ordinal Notations" (IONs) to be the smallest set P of computer programs such that for every computer program p, if all the things p outputs are IONs, then p is an ION.

See https://github.com/semitrivial/IONs for some ION examples in python.

Definition 2: Inductively associate an ordinal |p| with every ION p as follows: |p| is defined to be smallest ordinal which is bigger than every ordinal |q| such that q is an output of p. Say that p "notates" |p|.

Finally, to answer your question, I want the AGI to write programs which are IONs notating large ordinals, accompanied by arguments convincing me they really are IONs. An easy way to incentivize this with RL would be as follows. If the AGI writes an ION p and an argument that convinces me it's an ION, I will grant the AGI reward |p|. If the AGI does anything else (including if its argument does not convince me), then I'll give it reward 0.

You can't correctly incentivize this behavior using reals. The computable ordinals are too non-Archimedean to do so.

>There are no infinite rewards in biology and yet mathematicians seem to do just fine answering these sorts of questions

This is only a problem if you're already assuming we do everything based on our biological reward systems, and in the current context that would be circular reasoning.

Imagine the treasury creates a "superdollar", a product which, if you have one, you can use to create any number of dollars you want, whenever you want, as many times as you want. Obviously a superdollar is more valuable than any finite number of dollars, and humans/mathematicians/AGIs would treat it accordingly, regardless of the finiteness of our biological reward systems.

Here's an example. Suppose there are two buttons, A and B. If you press A for the nth time, then you get reward n. If you press B for the nth time, then you get reward 0 if n is not a power of 2, or reward omega (the first infinite ordinal number) if n is a power of 2.

If the above rewards are shoehorned into real numbers---for example, by replacing omega with 9999 or something---then an RL agent would misunderstand the environment and would eventually be misled into thinking that pressing A yields more average reward.

And yet computers have no problem symbolically representing non-rational numbers like sqrt(2), pi, etc. Neither is there any inherent reason why they cannot symbolically represent various levels of infinity, nor why those would be incomprehensible to AGIs (even if the universe is discrete). You're right that only countably many numbers can be represented, but nevertheless even countable subsets of extended number systems can exhibit structural properties that the reals do not exhibit.

I thought it was a fun position paper, if not exactly groundbreaking.

They did avoid one common pitfall at least. They are (intentionally?) vague about which number systems the rewards can come from, apparently leaving it open whether the rewards need be real-valued or whether they can be, say, hyperreals, surreals, computable ordinals, etc. This avoids a trap I've written about elsewhere [1]: traditionally, RL rewards are limited to be real-valued (usually rational-valued). I argue that RL with real-valued rewards is NOT enough to reach AGI, because the real numbers have a constrained structure making them not flexible enough to express certain goals which an AGI should nevertheless have no problem comprehending (whether or not the AGI can actually solve them---that's a different question). In other words: if real-valued RL is enough for AGI, but real-valued RL is strictly less expressive than more general RL, then what is more general RL good enough for? "Artificial Better-Than-General Intelligence"?

Note, however, that almost all [2] practical RL agent technology (certainly any based on neural nets or backprop) very fundamentally assumes real-valued rewards. So if it is true that "RL is enough" but also that "real-valued RL is not enough", then the bad news is all that progress on real-valued RL is not guaranteed to help us reach AGI.

[1] "The Archimedean trap: Why traditional reinforcement learning will probably not yield AGI", JAGI 2020, https://philpapers.org/archive/ALETAT-12.pdf

[2] A notable exception is preference-based RL

When I said "the reader", I meant other readers besides me. I am not a knowledgeable man, in fact just about all I know is how little I know. But as you are one of those other readers besides me, maybe you know what courage is, if so then please teach me :)

The reader presumably already knows what courage is, and does not need to be taught that. Rather, Socrates/Plato probably intended to guide us toward the eternal Form of courage. Laches' approach would indeed be better for teaching a neural net, but neural nets are not Socrates'/Plato's intended audience.

If I were a billionaire, I'd finance a project where this kind of technology is used on witnesses of historical events (WWII battles, marching with MLK, lost rock concerts, etc). Ask them to visualize the event, and record their visualizations. Obviously any individual witness would be unreliable, but data recorded from multiple witnesses could be valuable.

Time is running out. I don't know how many WWII veterans will still be alive by the time I'm a billionaire.

Sherlock Jr. is unbelievably deep and profound. For Keaton's more lightheaded humor, I recommend The Goat: https://www.youtube.com/watch?v=GXjhVZPYYRU

I think you might be mixing up surreals with hyperreals.

>You are replacing the usual meaning of AGI (a machine that can execute a wide variety of tasks in human to superhuman fashion) with a far stronger one (a machine that can complete arbitrary tasks perfectly).

No, I never make any assumption about AGIs being capable of completing arbitrary tasks perfectly. Indeed, that would be quite impossible, no agent could do that. The point is rather that the traditional RL agent cannot even comprehend environments with non-Archimedean rewards, but a genuine AGI would be able to comprehend them.

Yes, it's instructive to recall that the Pythagorean Cult believed all numbers were rational (and this was a religious sort of belief). There's a legend (albeit probably not historically accurate) that when a member of the cult proved that sqrt(2) was irrational, they put him to death... https://en.wikipedia.org/wiki/Hippasus

I think they'll be important in AGI research for one thing. Take reinforcement learning (RL) for example: the rewards are arbitrarily constrained to be real numbers (or rational numbers). But why? There is a rich variety of RL environments one can construct using non-real number reward systems, and there's nothing special about the reals (or rationals) that suggests any connection to RL. The real numbers, recall, are the unique complete ordered field. What do complete ordered fields intrinsically have to do with reinforcement learning? Nothing, as far as I can tell (if there were some such connection, it would certainly be interesting to professors of real analysis).

As you point out, surreal numbers do a great job of tying together many different number systems, and that's inherently appropriate for studying generalized reinforcement learning where rewards come from many different number systems.

I wrote a whole paper on this topic: https://philpapers.org/archive/ALETAT-12.pdf

Conway is often remembered for his Game of Life, but IMHO that is trivia in comparison with his surreal number system [1]. I predict that while surreal numbers today are an under-appreciated toy, in coming centuries they will gradually become more and more important, eventually surpassing even what we currently call real numbers.

[1] https://en.wikipedia.org/wiki/Surreal_number

It is a common misconception that "Thou shalt not lie" is a divine commandment from God. You're probably thinking of "Thou shalt not bear false witness against your neighbor" (Exodus 20:16), which is more specific.

Great. So because of our glorious War On Drugs, now drugs are being hidden all over the place where curious little kids are liable to find them. Can we please legalize yesterday?

Not sure what you mean exactly. Note, the set P of intuitive ordinal notations is not computably enumerable. Proof: if X were a computer program which enumerated all the intuitive ordinal notations and nothing else, then X itself would (by definition) be an intuitive ordinal notation. It would notate an ordinal number bigger than all ordinal numbers that can be notated by intuitive ordinal notations. Absurd. (Of course, to make this proof fully rigorous I'd have to give the formal definition of which ordinal an ION notates, which I left out above. But you can probably fill in those details yourself.)

As an example to illustrate how this initially-simple-looking project spans the whole range of all possible difficulty levels: suppose you had a source-code for a superhuman flawless obedient honest AGI, call it HAL. From that source-code, you could derive a source code X for: "Print all the computer programs which HAL would list if HAL were commanded: 'Enumerate every computer program which you know to be an intuitive ordinal notation'". Since we're assuming HAL is obedient and honest, all of X's outputs will be intuitive ordinal notations, so X itself is an intuitive ordinal notation: you could say it notates "the ordinal of HAL". So this innocent-looking exercise in fact touches the most ambitious possible projects, even "Design a perfect superhuman AGI" :-)