But it's absolutely commendable how Andrew Ng takes the topic to such an understandable level that a clever high schooler who knows a little about programming could take the course. There's even extra videos serving as a crash-course to linear algebra and octave programming in the first week.
So he really manages to make the topic accessible to a very large audience.
Do you know what kind of math is needed other than linear algebra.
Also being able to take derivatives helps in a couple of places, but is not necessary.
* Pattern Recognition and Machine Learning by Christopher M. Bishop
* The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani and Jerome Friedman
Both are very intensive, perhaps to a fault. But they are good references and are good to at least skim through after you have baseline machine learning knowledge. At this stage you should be able to read almost any machine learning paper and actually understand it.
Edit: Andrew Ng's Coursera course is CS229A (http://cs229a.stanford.edu/), not really watered down.