It is theoretically possible as a single male, but there are many more restrictions - in practice its very very rare. For example, see these requirements for a very large international adoption agency: [1]. In my experience there is, as OP describes, a strong stigma against single males, not just that they are considered perverts, but also that they are considered incapable of taking on this responsibility alone, not nurturing enough, too career-focused, etc.

[1] https://www.holtinternational.org/adoption/ See "Quick overview of parental eligibility requirements" section