> There would likely be a lot of unaffiliated candidates if the number in the House was drastically and suddenly increased.
No, there wouldn't. The electoral forces that drive duopoly operate the same almost independently of district size, changing the number of single-member House districts with first-past-the-post elections won't reduce duopoly.
Changing the electoral method might (and doesn't require changing the number of districts.)