rsf

For the avoidance of doubt, I neither work for Anthropic nor do I have a horse in this race in any other way. I didn't even know about the Anthropic compiler when I wrote the blog post.

> The OP should try with -march=native so the compiler can use vector instructions.

I just tried "-O3 -march=znver5" as well as "-O3 -march=native" and it didn't seem to make any difference.

> Sure, the code is strange, but it is not necessarily inefficient.

Out of the 6 pieces of Assembly code in the article, 2 of them are definitely inefficient - specifically, the 2 clang ones that contain irrelevant writes to the stack. Even if a CPU was smart enough to ignore those instructions with no performance penalty (which in itself is doubtful), at the very least those instructions take up space in memory/caches unnecessarily.

The gcc output when arraySize is 3 is almost certain to be inefficient as well, when you look at portions such as:

        mov     eax, 1
        test    eax, eax
        sete    al
        ret

All this code is doing is to set eax to 0 and then returning. This could be done by simply replacing it with "xor eax, eax ; ret" or "mov eax, 0 ; ret" if there's a reason to avoid "xor" - there's already a mov there. The code as present also has the side effect of changing the CPU's flags, but this side effect can't be relied on as we return immediately, and flag values are not part of the returned values with this ABI.

So yes, in general benchmarking is the only way to be sure. But when you look at the specifics of the generated code, we can see that at best 4 of the 6 snippets of Assembly code are optimal, and the actual number of optimal snippets is probably lower than 4 (my best guess is 2 here).

All that said, I might benchmark everything later on and post a new article about it.

> Also worth mentioning in passing: if you are not compiling with --march=native, all your code is being optimized for some prehistoric ancient least-common-denominator Intel processor, probably a 1990's-era 486, that nobody actually has anymore that has god-only-knows what inadequacies in its execution pipeline. So make sure you are.

Yep: See https://news.ycombinator.com/item?id=46978577

> The claim that the code is inefficient is really not substantiated well in this blog post.

I didn't run benchmarks, but in the case of clang writing zeros to memory (which are never used thereafter), there's no way that particular code is optimal.

For the gcc output, it seems unlikely that the three versions are all optimal, given the inconsistent strategies used. In particular, the code that sets the output value to 0 or 1 in the size = 3 version is highly unlikely to be optimal in my opinion. I'd be amazed if it is!

Your point that unintuitive code is sometimes actually optimal is well taken though :)

Interestingly it's actually not a flaw, the key appears after a while if you're not in the room (and don't have one):

https://news.ycombinator.com/item?id=17460392

I assume the agent somehow found this out and developed the behavior of going in and out of the room until the key shows up (which, with enough agent randomness it apparently will).

> In addition, the agent learns to exploit a flaw in the emulator to make a key re-appear at minute 4:25 of the video

After a bit of debugging, this appears to be a very intentional feature in the game rather than a flaw. That key appears after a while if you're not in the room (and don't have one).

Based on this disassembly: http://www.bjars.com/source/Montezuma.asm

Here's the relevant code with some annotations added:

https://goo.gl/VUDr9F

I'm not sure if this is a previously known feature in the game (a quick google search does not reveal much). It would be quite interesting if the RL agent was the first to find it!

PS: If you launch MAME with the "-debug" option and press CTRL+M you can see the whole memory (atari 2600 only has 128 bytes!!) while playing the game. If you keep an eye on the byte at 0xEA you will know when the key is about to pop up. Alternatively you can speed things along by changing it yourself to a value just below 0x3F.