This sound bite always comes up. The simple fact of the matter is that for these institutions, the endowment is essentially composed of gifts, and returns on investment of gifts. Many/most gifts are restricted, meaning that the donors gave them to be used in specific ways - like to fund a specific professorial chair, or for a particular library or whatever - and it's just not an option to use them to fund tuition or for other purposes.
Data point: Harvard's endowment has over 14,000 individual funds and the majority of them are restricted.