Exactly. Max/MSP is great and I really want to like it, but despite making several attempts at using it in my practice, it hasn't stuck yet. Max doesn't have enough abstractions to make in depth development enjoyable IMHO. After a certain point I get bogged down and long for a regular programming language with an IDE. I'd probably enjoy Max more if I was comfortable in C.
Max's std lib is rich and it would hurt to start from scratch in another environment, but you don't need that many DSP modules to start building interesting patches. An LFO, filter, ADSR and an oscillator get's you a long way towards a usable synth. And there are many open source libraries to draw from.