As someone who's interfaced with the data brokers professionally, I can tell you that the ultimate starting point for their dataset is usually a state voter registration database. A lot of states pretty openly sell their data, like from the DMV, but every data broker is starting by figuring out a way to get that voter registration data- some states just give it, in some states (like say Maryland) you have to at least pretend to be an election researcher, etc. I'm sure for more restrictive states it gets passed around on the grey market.
But if you're anti-data broker (I certainly am!), it starts with regulating your own state