Mods are great, but they generally lack cohesion. I agree with OP. I wish that instead of passive upgrades and only a couple tower/ammo variants, the game had a deeper tower defense aspect.
It'd be nice to have more enemy and tower variants, with certain types of towers working better against certain types of enemies - or towers that debuff enemies instead of hurting them. Maybe there could be structures that attract nearby enemies so it'd be easier to direct where attacks hit.
I think stuff like that would keep the tower defense stuff from loosing relevancy in the late game while staying fun.