The problem is that you don't actually have a clear measure of who you're rejecting. Every company uses a similar argument to support their (very different in every case) interview process. The fact that it allowed you to build a great team does not mean that you could not have built a better team by giving other options as well.