Yep, I completely agree with you. The other advantage non-qwerty layouts have is that you can't cheat by looking at the keys, which greatly serves to push you in the right direction.

Side note: After typing exclusively on dvorak for ~10 years now I actually think that it's not the best layout to maximize potential typing speed. (I've never gotten as fast with dvorak as I used to be with qwerty.) Dvorak focuses too much on single key locations and alternating hands. I think to get the maximum speed a layout needs to focus a lot more on runs, making sure the most common n-grams are typeable with out-to-in or in-to-out runs of adjacent fingers in the same hand. I've thought about working on this, but quite a bit of work would be needed to develop a new and more accurate cost model and find a layout that optimizes it--I've got too many other competing interests. But I'd love to see it done.