I did. It wasn't easy and I had the support of a large team of people to fall back on.

10 years ago I was an early employee of a risk analytics firm. I got pushed into a quant roll because I had an engineering background and could solve PDE's.

My first meeting I remember the head of the quant team hand-waved over a bunch of math that took me two weeks to go through on my own.

I had to spend 2-3 nights a week for 1 year going over my old statistics texts, Differential equations text and Knuth's Concrete Mathematics before I moved onto Stochastic Calculus.

It would have been almost impossible without having a team of people who were willing to help me get through sticking points. That was 8-10 years ago. I still spend atleast 1 night a week learning new math, though now its mostly non-linear regression and chaos theory.

And to be honest I still get alot of quants looking down on me, "A guy who dropped out of his masters, trying to claim he's our equal?". That does bother me a bit, and it means I probably couldn't get a quant job at Goldman, but I'm very proud of where I am now and how I got there!

TL/DR It can be done, but it gets done over the course of years, not months or weeks. You won't get there without applying what you learned to real world applications, reading the text's aren't enough.

I guess I sort of did. I never took mathematics seriously, have a GED instead of a HS diploma, and subsequently started college in remedial HS-level algebra. But now I have a PhD in applied mathematics and work in the overlapping gray area of research that exists between the mathematics, computer science, and systems engineering disciplines.

I did have some very good (and very patient) instructors early on. But at some point when I had the chance to read and understand the basics of discrete math and intermediate calculus in-between my semesters (undergraduate was a bit broken up for me, I was a deployed military reservist), I guess I found it interesting enough at that point to go deeper and change my major.

What are the best ways for a non-genius, normal person to learn advanced mathematics?

I took a Masters in Mathematics with the Open University. It's not quite as brutal as simply telling you which textbook to read and see you in nine months for the exam (repeat five times), but it's not far off.

It's not quite entirely by yourself, as in the webpage linked, but it worked like this:

  1) Get sent problem sheets and a list of what chapters
     in a textbook to work through.
  2) Read textbook.
  3) Solve the problems in the textbook.
  4) Watch a one-hour live webcast to you (and a dozen 
     other people) four times.
  5) Send off completed problem sheets; if you make the 
     pass mark, your prize is a seat in the exam.
  6) Optionally pay for a weekend of direct instruction to 
     you (and a dozen others) in person somewhere (I did
     this 3 times out of six, I think).
  7) Sit three hour exam in supervised exam hall.
  8) Go home, open next textbook.
  9) Repeat four more times, and then write a dissertation.

As in the linked webpage; it's just you and the textbook, for nine months. Not the same as in the linked webpage; there is a tutor you can eMail. This was often not as useful as you might think and some years I eMailed my tutor fewer than five times.

Also not the same; a formal, full-on three hours exam in a formal exam hall. This sets the bar pretty high and is a really good indicator to yourself. I think this makes a real difference; someone else validating that yes, you really do know what you're talking about (or not, as the case may be - I think that of everyone who starts this, less than fifty percent pass the first exam, and if I had to guess, I'd say that it's not so much that the maths is hard; making yourself study it to the level required month after month is hard).

I estimated that the time I spent on this was roughly equivalent to working full-time for 6-8 weeks per year. Interestingly, if I'd been given an actual 8 week block, I don't think I could have done it. I couldn't spend 8 hours a day reading a maths textbook and get the same from it as if I'd done it in four two-hour blocks spread over a week.

Coming back from an exam, sitting down at the table, putting all the notes and papers and books from the previous subject to one side and then sliding the virginal textbook and brand new notes and problem sheets onto the table was brutal. It can be done, but you have to want it. I found that in terms of taking in new knowledge, I couldn't do more than a couple of hours with the textbook at a time. For practising and solving problems, I could sometimes sit down at lunchtime on Saturday to just have a go at one quickly and stand up again eight hours later, table covered in notes and the problem at hand having been thoroughly explored and answered (typically, sadly, spread over several pieces of paper which I would come back to in the future and condense into a smaller paper space).

The study habit it forced into me is enormously valuable in itself. I can now pick up a high-level maths text book and simply start learning. I won't understand much of it, but I know that if I keep at it, I will. Grind, grind, grind. Not just mathematics, either. Someone gave me a Learn Japanese book for Christmas and now it's in me. I ground through it, and now I've got another one and I'm going through that. It's as if the subject doesn't really matter, so much as the studying.

One famous, although admittedly not particularly recent, example of a mathematical autodidact is Srinivasa Ramanujan (https://en.wikipedia.org/wiki/Srinivasa_Ramanujan).

I don't know if my journey can be called self study. I keep taking classes at community college when I can. I am not a fan of the emphasis on technique over understanding concepts but staying on track is hard when I try to do it all alone!

If you're looking for historical examples:

Stefan Banach, Israel Gelfand, Edward Witten...