Right, this is exactly what I'm asking. But why is the answer automatically "yes"? Why can't websites have a disclaimer saying "this company does not sell or ship to entities in the Eastern District of Texas"?
The general rule is that a federal court only has personal jurisdiction where a state court of general jurisdiction would have it. So both the constitutional due process requirements (International Shoe, et. al.) and the state long arm statute come into play. (This assumes the patent statutes don't have unique PJ rules, they might.)
All that said, without having actually looked into the case law, I'd imagine you couldn't defeat jurisdiction by not doing business in the forum because the Calder effects test would be construed to apply to infringing a patent owned by an entity in the forum.