You've put forward a standard defense of the program. It generally goes like this: there is a meaningful shortage, salaries for talented people are extremely high, the reason Americans are avoiding these jobs is lack of good education, not rational, market based preference for other options, these workers create jobs, they don't take them away.
Interestingly, everyone agrees with you, until you start talking numbers. Even Norman Matloff (google him if you don't already know) supports 10,000-15,000 visas for talented people.
The problems start at scale, which is when you have to start asking if the visas are displacing or deterring rather than simply enriching. Just keep in mind, almost nobody is actually arguing for no visas.
I think the best, most sober argument that displacement is occurring comes from the RAND institute (which is historically devoted to truly rational analysis of public policy questions). Their study was at the graduate level, but they concluded that the American aversion to STEM fields is a rational, market based response to poor pay and prospects relative to the other things well educated people can do provided they already have US citizenship (or a green card, something that allows free movement within the labor force).
http://www.rand.org/pubs/issue_papers/IP241.html
Here's a quote from one of the study's authors:
"The diminishing share of degrees awarded to U.S. citizens suggests that science and engineering careers "are becoming less attractive to U.S. citizens," said RAND, "or, alternatively, that U.S. citizens encounter more competition (from foreigners) in applying for a limited number of desirable spots" at research institutions, where foreigners have helped reduce research costs by holding down wages."
http://usatoday30.usatoday.com/tech/science/2009-07-08-scien...
No question the US workforce is enriched in many ways by high skilled people from overseas. But based on the objective research, I think it's pretty clear that we've done this at a scale, and implemented it in such a way, that we're causing meaningful displacement and deterrence for US citizens in STEM fields, especially at the higher (graduate) level.
In the long run, I think it would be terribly destructive to close to door to top talent from overseas. The part that frustrates me is that the FWD.us types don't seem to appreciate the reality and danger this kind of displacement and deterrence could have on the US. We need a healthy pipeline of domestic labor, and we need to keep this in balance. If you kill off the domestic workforce, you can't revive it on a dime if (and I'd say when) they day comes when we can't just wave the magic wand and get people from overseas.