This is the crux of it in my opinion. 90% of my time using pandas seems to go on data formatting and cleaning to get info into the perfect form for pandas to do a relatively simple set of operations on. It gives we computational speed in exchange for a ton of fiddly tweaking.
The biggest thing for me is that data frames are collections of series, and not collections of rows. That just seems wrong on such a fundamental level to me.