I wonder how good you can get at maths just by casually reading books. You need to work on problems for hours and hours to get a grasp on the theories. Programming is different in the sense that it's something people routinely do as a hobby because it's quite fun and addictive. But maths? maybe if you have already strong foundations you can pick up a new topic and develop your culture. But I doubt one can get these foundations without actually graduating in maths as it's an extremely strong commitment.