Do it. Build a couple, then build a simple opamp mixer (it's a single opamp and a handful of resistors, and a couple of capacitors across the power supply), and then - get adventurous - a PT2399 delay kit.

Then take a look on https://yusynth.net at some of the VCF designs, and build one of them.

You won't have a synthesizer, you'll have some crazy homebrew drone machine that you can make scary movie sountracks with.

Definitely do it.

Obligatory link to Forest Mims' book: https://archive.org/details/555-designs