The one that drives me insane is that while editing a cell formula, that using the arrow keys move cells (and update [read: wreck]) the formula rather than moving the cursor within the formula you're editing.
To me it is a feature, not a bug. But yeah, F2 is fairly easy to use once you get used to it. Its like using VIM, the shortcuts seem annoying to the uninitiated, but once you get understand how everything works, the power can really speed things up and you appreciate it.
I really wish the default is the other way around. When I do want to add other cells, more than half the time there is a better way than using arrow keys.
In many cases you are just not fully up to speed with the features in windows / excel.