There are ways to monetize open source. GitHub could make the repository/PR code open source and host the repo management/hooks/actions/etc code for enterprise.
There are hidden costs to going open source as well as expected ones. Could you imagine the number of PRs, issues and discussions over trivial shit the GitHub userbase would create against an open GH repo? Nightmare. Not to mention code cleanliness expectations and buildability expectations and so on.
Of course they "could do this" or "do it that way," but the fact that they don't should tell you their priorities lie elsewhere, and that's fine. Closed source isn't evil and we have other open source git hosting platforms.
It’s popular because it’s free.
Gitlab has always seemed a little clunky to me from the perspective of running a smaller operation.