In terms of immigrants as a percentage of population the US is nothing special. Canada has a lot more (closer to 19% instead of 14%). Many European countries have statistics similar to that of the US (Sweden has a bit more, the UK a bit less, etc).

But I digress. The problem with the existing system is that it is insane and a waste of resources. One has to get a lawyer and wait for a long time with no certainty in the outcome while dealing with an extremely opaque bureaucracy where anything can go wrong at any time.

This has nothing at all to do with volume. Indeed, a simpler more streamlined and less opaque system would help with volume.

Say a system like the Canadian or the UK one. There's a point scale. You can compute the number of points you get ahead of time, now there's a simple web form actually. If you cross the threshold you will get in. There are known wait times, you can just call the embassy. There are points for things the country needs (particular jobs), for certain qualifications (degrees, etc), for language proficiency, some regional tweaks, family, and having an employment offer.

And extremely importantly. You get permanent resident status (green card), not an H1B. What Americans don't realize is that you want to hand out green cards. H1B lower both your salaries and ours. The H1B restricts immigrant mobility, can't move to a better job and raise the average salary, and encourages people to go home with all of their newly gained knowledge and money.

I'll give you a personal example of the difference between a sane and an insane system:

As an 11 year old I filled out all of our Canadian immigration paperwork (my parents checked it but it was correct and they didn't change it). We knew we would get in based on our points. The embassy told us the timeframe in which we should expect our paperwork to go through. It went through a bit early. We moved to Canada.

Now for the US. As a near-30-year old with a US PhD working at a top research institution I have to pay a specialized law firm several thousand dollars, spend weeks getting paperwork, bugging people in several countries to write absurd letters, building a case, etc. All of this to basically the same thing. And in the end, who knows what will happen because there are no standards, no appeal, no one to discuss anything with. Oh, and I have no idea what the processing time is.

So no. It is not an issue of "why can't they just let me in". It's a system that hurts your salary by restricting my mobility, hurts me by making me pay lawyers needlessly, hurts the image of the US by creating disgruntled people, and hurts the economy by routing business and increasingly prestigious conferences elsewhere. It just makes no sense.