>None of them were displacing American workers -- we would have hired just as many (or more) qualified American workers if they had been available too.
Did you try increasing salaries until you could find applicants. I'm virtually certain that you did not.
What you mean is that you couldn't find American workers at the price you were willing to pay, and because you weren't willing to pay the market price, you found another market.
If you could hire 20 people on H-1Bs but you only have enough money to hire 10 local people. You have displaced 10 American workers.