>You should forget about all of that high school math and start directly with university math. It's actually easier in that it really starts from the beginning.
So, I think being able to do the problems in high school math is important; it's certainly expected for the college level books I've read. The books for highschoolers are completely useless, as far as I can tell, except as sources of exercises (and khan academy is way better for exercises than any book if you ask me) - My favorite books for high school math are Lang's "Basic Mathematics" and maybe "Short Calculus" which don't seem to be aimed at the high schooler, even though it covers math that the people I wish to associate myself with did in highschool. This might be what you mean about College math starting from the beginning? But my impression is that most college-level books presume you "speak the language" - presume that even if perhaps you can't prove all of high school math, you have memorized the notation and the formulas and know what formulae to apply to what situation.
I compare high school math, as conventionally taught, to the way a semi-literate reads; memorization without understanding. I Personally remain unconvinced that this has no value, though of course, it's clearly a lot less valuable than having real understanding through proofs and derivations. But value or no, most of the college-level math material seems to expect you to have spent some years memorizing and applying without understanding before you get to proofs and the other interesting stuff.
Personally, I've started working through the old memorization-and-drill because I can - though I am still functionally a layabout, I have worlds more willpower and motivation than when I was 16. Also, I've discovered spaced repetition flashcards, and between the two of those things, for the first time in my life, I can actually memorize things like the multiplication tables.
Right now, I'm looking at reading and comparing that with math; I started reading after I was expected to, and it was really quite difficult for me at first. But then I reached a certain level of proficiency, and after that? it became extremely easy, and really the foundation of all my further success. (and I am moderately successful, professionally. I've had a reasonable career as a UNIX SysAdmin/SRE/Programmer, though my competencies lean to the left of that list.)
My current working theory is that mathematics as a language is a similar sort of thing, and you certainly have to memorize the words and symbols... and memorizing the more important phrases will make you a better communicator, even before you gain true understanding of the depth of said phrase.
I guess the other thing is that when I first started learning to read... I would read books well above my level and actually understand very little of them; but I pretty quickly started filling in the holes. Perhaps I ought to utilize that strategy with math; I have plenty of good math books, but I tend to read until I find something I don't understand, then I stop and chew on it (or get help from a friend) - which is obviously way slower than just reading through even if I only understand 20%;