It's always the fastest, but whether or not it's the fairest depends on how you define "fair." The simplest example I can use to illustrate this is the case where there is 1 server, and 2 people. One person will take 60 minutes to process, and the other, 1 minute. The 60 minute person gets in line a millisecond before the 1 minute person. Is it fair for the person who will take 1 minute to wait for the slow 60 minute guy?

If you define fairness as the average user-wait, I think fastest first is actually optimal. This is still the case with one-queue systems: If you let the 10-minute person go first, you'll have blocked 1/10th of throughput for everyone behind him, and thus increase their wait time. Fastest job first is generally not a bad way to handle queues; it works great for things where "jobs completed per unit of time" is of importance, like chores or fixing bugs (of equal or similar priority).

Besides average user-wait, you could start inventing your own units for fairness depending on the problem. Maybe its average user-wait per processing minute, or something like that, in which case the solution might be more complicated (or might not be, I'm not too concerned with it).