- Random pivot (ie Median of 1) has 3.39N comparisons
- Median of 3 has 2.75N
If you want the fastest possible algorithm you should follow
http://dx.doi.org/10.1109/TSP.2012.2197394
which proposes to choose the median of N^{2/3}1/(4pi)^{1/3} elements as the pivot and then also uses a different strategy to choose the second pivot (close to the median yet "safe").
Another fast algorithm is Floyd & Rivest's SELECT (yes the same crypto Rivest) which is also asymptotically optimal and implementations exist. (Paper title: "Expected time bounds for selection")