cwinter

yes! the very last example in the post shows what happens if you truncate the mantissa to just the 4 most significant bits

Well if there was ever an appropriate time to plug my dark fantasy machine learning poem this is it: https://clemenswinter.com/2021/03/24/conjuring-a-codecraft-m...

The bot has actually never seen Destroyer's big drones during training even once, so I found it somewhat surprising that it even works as well as it does!

Completely agree that adding something like the "League" used by AlphaStar would be one of the top priorities if you wanted to push this project further. I don't think CodeCraft is sufficiently complex to really allow for several very distinct strategies in the same way as StarCraft II, but I would still expect training against a larger pool of more diverse agents to increase robustness quite a bit.

Thank you for the kind words! I am also quite excited about the new points in game design space that RL will unlock and am planning write another blogpost on that topic.

I quite like https://karpathy.github.io/2016/05/31/rl/ as an introduction to some of the ideas behind modern RL. Beyond that, I just recently found out about https://github.com/andyljones/reinforcement-learning-discord... which lists a lot of other high-quality resources.

CodeCraft is a programming game which you can "play" by writing a Scala/Java program that controls the game units. It's not actively developed anymore but still functional: http://codecraftgame.org/

Yes, excellent points. No doubt there is still much room for optimization.

The constraining factor in this case is CPU throughput. The CPU cannot execute instructions fast enough to keep up with the data being read in from memory.

Of course you are right that this is not the whole picture, it is possible to be bottlenecked on neither memory bandwidth nor CPU throughput.

With respect to memory, two additional considerations are read amplification and latency.

Read amplification: Read amplification basically means that you are reading data but not actually using it. Suppose we have a table stored in row format with four columns C1, C2, C3, C4 which store 8 byte integers. Row format means that the values for the columns are interleaved, so the memory layout would look like this, where cn refers to some value for column Cn:

  0x0000 c1 c2 c3 c4 c1 c2 c3 c4
  0x0040 c1 c2 c3 c4 c1 c2 c3 c4
  ...

One property of conventional memory is that we can only read memory in blocks of 64bytes (the size of a cache line). So if we have a loop that sums of all of the values in C1, we will actually also load all of the values for columns C2, C3, C4. This means we have a read amplification of factor 4, and our usable bandwidth will be a quarter of the theoretical maximum.

Latency: Suppose we also don't have any read amplification and make use of all values for each loaded cache line. Then after we finish processing a cache line, and load the next cache line, we might have to wait a long time (100s of cycles) before the new cache line actually arrives! So then we would actually be stalled because of memory latency during some parts of program execution.

One of the big advantages of using columnar layout (which stores all values for a column sequentially) is that it all but eliminates read amplification. Similarly, by accessing data sequentially we make it possible for the CPU to predict what data will be accessed in the future, and have it loaded into the cache before we even request it.

Of course there's a number of reasons why things might not work out quite so perfectly in practice, and those may why performance in this benchmark starts to degrade before the (usable) memory bandwidth reaches the theoretical maximum memory bandwidth. But as you can see it still possible to get very close, so it's safe to assume that queries which read much less data than the maximum bandwidth are constrained on CPU.

CodeCombat is aimed at people with no, or very little, programming experience. My game will be targeted towards people who already have at least basic programming knowledge, and are looking for a less restrictive environment. In that respect, it will resemble CodeCombat's zero sum mode[1]. I am hoping that the focus on slightly more experienced programmers will allow me to create a game with more strategic depth.

[1] http://codecombat.com/play/ladder/zero-sum

Hmm, that actually is a bit more accurate, considering it links to a blogpost. Thank you for the suggestion.

@mods: could you change the title on hn as well?

You raise a really good point about thinking through the game mechanics before diving into the coding. As it happens, I have actually spent a lot of time on this and even implemented something similar some years ago. I probably could made that clearer in the blogpost, but I mainly wanted to give a concise overview my goals and plans for this project. My next updates should have a bit more substance :)

> Looks good, looking forward to it!

Thank you :)

> Why isn't Clojure supported?

I have never used Clojure myself, so I don't know too much about it. Can it easily call into Java libraries? In that case, Clojure should work as well.