This is a very uncommon idea within EA. There are some people who support it ("patient philanthropy") but almost everyone within EA either thinks you should be working / donating to help people/animals (most "neartermists") or you should be working / donating to help reduce extinction risk (most "longtermists").
I think the closest there is to a mainstream EA view on this is https://80000hours.org/articles/harmful-career/ , which argues that "it's a mistake to pursue a career in which the direct effects of the work are seriously harmful, even if the overall benefits of that work seem greater than the harms".
I'm opened to having my mind changed, but i'm going to need more than that.
https://www.forbes.com/sites/sarahemerson/2023/02/28/effecti...
(They are now selling said castle.)
I don't think EA is immune from grifters or people with bad intentions. It is after all a movement made up of humans. I think the question to be answered is does the philosophy encourage it in a way that other philosophies around philanthropy don't.