But there are plenty of schools (my alma mater CMU is one such example) that have mind share but a very small endowment. CMU has made a point of building its endowment over the last few years -- there was a single $250M donation, a few others in the $50-100M range, and calls for much more.
Endowments are important to the long-term financial health of an educational system. They help schools borrow money at very cheap rates to fund large infrastructure projects that otherwise would need to be built by donor / grant money alone. A school like CMU with a small endowment ends up having sky-high tuition to compensate (indeed, CMU has the highest tuition in the country across nearly every degree program they offer) and limited scholarship opportunities. They also have to accept a greater percentage of foreign nationals who otherwise might not qualify (and who can usually pay the tuition out of pocket no problem, since the wealthy across the world want at least one degree from a US school on their resume).
But essentially, the endowment is a nest egg that the school uses to ensure its long-term survival and academic independence. If you feel your alma mater made you the person you are today, and you want to encourage that to continue into the future, a large endowment gift goes a long way to ensuring its continued existence, independence and prosperity. Agree that there are a few schools that don't really need more (Harvard being the prime example) but the vast majority of schools really do need that much money sitting in the bank. You're donating to a school of thought more than anything else.