You cannot choose to breed with someone who is already dead. Even if you succeed in breeding with someone (alive), if their genes are not very good your children won't live to breed.
Thus if intelligence is a factor at all in survival (and it seems like it should be) your ancestors may not have had a choice but to select for intelligent.