It is in retrospect a pretty banal observation. If cipher design is about achieving a certain security level at the lowest cost per byte, it sort of stands to reason that you'd want to design them around those features of a CPU on which the market is going to put the most pressure to improve.

In fairness, some of this design philosophy --- which Bernstein has held for a very long time! --- gets a lot clearer with the advent of modern ARX ciphers, which dispense with a lot of black magic in earlier block cipher designs.

A really good paper to read here is the original Salsa20 paper, which explains operation-by-operation why Bernstein chose each component of the cipher.

Eh. Remember that a lot of these guys think about hardware implementations, so they're fixated on ASICs, not GPUs.

At some point the inequality gets too far out of whack and it's time to reconsider your values. Same reason we keep switching back and forth between dominant network architectures every 8 years. Local storage gets too fast or too slow relative to network overhead and everyone wants to move the data. Then we catch up and they move it back.

There can often be a strong disconnect between theory and implementation. I don't believe you can be truly effective unless you can get your hands dirty with both. Although if you have to choose, I'd rather know a bit of theory, and a whole lot about reality, than vice versa. Constants factors matter in big O.