Within the software, the hardest parts were understanding and controlling aliasing and getting a virtual ladder filter to perform well.
I already knew how to play piano, had a little music theory, and had spent a few hours playing with other people's analog synths.
Even if you're trying hard to be avant garde and break new sonic ground or whatever, it's hard to imagine that you won't have some significant overlap with the problems addressed by synth builders to date. That background will also be useful when trying to understand everything from published papers to existing synths built in pretty much any existing way, from analogue modular, to "rack-style" digital like Logic and Reason, to programmatic work in Max, Pure Data, SuperCollider, or even direct-coded in a traditional programming language, etc.