You need to compute the ratio between the number of successful experiments and the number of total experiments.

If what you want to know is the probability of having the 5 guys paired with 3 men and 2 women (which is what I think the grandfather meant), the answer is

C(4,3) x C(10,2) / C(14,5) = about 9%

(number of ways to pick 3 men out of 4 x number of ways to pick 2 women out of 10 / number of ways to pick 5 people out of 14, /regardless of the order/)

If you want to know the probability of having the pairs made exactly in the order the grandfather mentioned, it's

P(4,3) x P(10,2) / P(14,5) = about 0.9%

(as above, but considering the order)

where C(M,N) and P(M,N) are the number of combinations and permutations of N items out of M, respectively.

(edit to change multiplication symbol, asterisk doesn't show)

You need to divide by 3! and 2!. One way to look at it is that every selection of five from the 4M and 10F is equally likely. There are \binom{14}{5} (i.e. 14 choose 5) such selections. Of these, \binom{4}{3} * \binom{10}{2} have exactly 3M and 2F. Therefore, the probability is \binom{4}{3} * \binom{10}{2} / \binom{14}{5}, or (4 * 3 * 2 * 10 * 9) 5! / (3! 2! (14 * 13 * 12 * 11 * 10)) = 10 * 9/(13 * 11 * 7) = 0.08991...

The complexity of the formula seems to match the complexity of the problem: three lists of decreasing integers matching the three groups, multiplied/divided as appropriate.

The hard bit is having the intuition to see that this is right, which is just what Bill Dubuque's answer goes on about in the linked article.

Is there any difference between x and * operations?