I got 800 on the 1980s-era math SATs, came in third in the Portland OR area in a math contest in high school, and did OK at Caltech (not in a math major), but I'm no Terry Tao, and I very much doubt I'd've been anything very special in a good math undergrad program. Some years after graduation, I found it challenging but doable to get my mind around a fair fraction of an abstract-algebra-for-math-sophomores textbook, including a reasonable amount of group theory (enough to formalize a significant amount of the proof of Solow theorem as an exercise in HOL Light, and also various parts of the basics of how to get to the famous result on impossibility of a closed-form solution for roots of a quintic).
From what I've seen of real analysis and measure theory (a real analysis course in grad school motivated by practical path integral Monte Carlo calculations, plus various skimming of texts over the years), it'd be similarly manageable to self-learn it.
One problem is that some math topics tend to be poorly treated for self-learning, not because they are insanely difficult but because the author seems never to have stepped back and carefully figured out how to express what is going on in a precise self-contained way, just relying (I guess) on a lot of informal backup from a teaching assistant explaining things behind the scenes. On a small scale, some important bit of notation or terminology can be left undefined, which is usually not too bad with modern search engines but was a potential PITA before that. On a larger scale, I found the treatment of basic category theory in several introductory abstract algebra texts seemed prone to this kind of sloppiness, not taking adequate care to ground definitions and concepts in terms of definitions and concepts that a self-studying student could be expected to know, and that's harder to solve with a search engine, tending to lead into a tangle of much more category theory and abstraction than one needs to know for the purpose at hand. My impression is that mathematicians are worse at this than they need to be, in particular worse than physicists: various things in quantum mechanics seem as nontrivial and slippery as category theory to me, but the physicists seem to be better at introducing it and grounding it. (Admittedly, though, physicists can ground it in a series of motivating concrete experiments, which is an aid to keeping their arguments straight which the mathematicians have to do without.)
I have been much more motivated to study CS-related and machine-learning-related stuff than pure math, and I have been about as motivated to self-study other things (like electronics and history) as pure math, so I have probably put only a handful of man-months into math over the years. If I had put several man-years into it, it seems possible that I could have made progress at a useful fraction of the speed of progress I'd expect from taking college math courses in the usual way.
I think it would be particularly manageable to get up to speed on particular applications by self-study: not an overview of group theory in the abstract, but learning the part of group theory needed to understand the famous proof about roots of the quintic, or something hairier like (some manageable-size fraction of) the proof of the classification of finite simple groups. Still not easy, likely a level harder than teaching oneself programming, but not an incredible intellectual tour de force.
"Myself, only after 5 years of mathematics I'm somehow comfortable to study subjects by myself, and it's still hard."
Serious math seems to be reasonably difficult, self-study or not. Even people taking college courses in the ordinary way are seldom able to coast, right?