The reason I think this would help on a number of levels is as follows 1) Gerrymandering becomes more difficult if there are a lot more districts. 2) influence peddling and lobbying is diluted if there are more representatives. 3) the voter can have more influence and contact with their representative if their congress person represents a smaller sized constituency.
I think arguments around the size of the building (the capital) are silly, who cares where they meet, it can be in a stadium, or even better why bother having them all in Washington DC. it can all be done with telepresence in todays world and then you as a citizen could go down to say the court house and sit in the audience as your representatives interact via telepresence.
I get it that perhaps all this seems crazy or whatever, but none of it would require any changes to the constitution. It could be done by the House itself with no consent from the Senate or President.