Regarding All of Statistics, the author states that the 'reader is assumed to know calculus and a little linear algebra'. It's a good book, though think you'd have a tough time with only 'a bit of algebra'.