You're probably right with respect to growth.
Other commenters are going to glibly point out that you need taxes to pay for basic research or whatever. Yeah, I think they miss the point too.
If the country wanted to, it could not raise a single tax dollar and 10x its research spending. A country isn't a household, its balance sheet can have debt permanently, etc. etc.
No, I think taxes matter for justice and equality. And the whole reason cost effectiveness entered our debate is that rich people, like you're correctly pointing out, have opinions, and tax policy is a consequence of that.
But if overall rich people are aligned against higher taxes, which is the case in history most of the time, they're effectively aligned against equality and justice.
And save me the downvotes if you don't understand how property taxes, for example, that differ between neighborhoods, can be about justice as much as they are about funding school districts.
Or that the healthcare mandate was absolutely about justice, and that calling it a tax was a workaround precisely for the reasons I described: that we've become obsessed with positivist or legalistic (instead of normative) dialogue, so it was the only way to get ACA passed.
I love tax policy on HN, because it really brings out of the woodwork people who do care about what's good for the country, and people who don't.