I fly in and out of the country all the time (I believe I've returned the the US five times this year alone). I only ever have a backpack and it's not huge and I never check any luggage. I'm not dressed in business clothes, nor do I look particularly affluent. My passport is fully of "dangerous" countries (Colombia, Guatemala, China, Russia...) and is as beat up an scruffy looking as I am.
I have never once has trouble returning to the US.
In fact, I flew back from Mexico (several years ago, but post 9/11) without anything but a driver's license and even then didn't have any problems other than answering some reasonable questions.
I've heard anecdotally that it's more difficult for foreigners entering the US, but that's to be expected. Pretty much every country on the planet is harder for non-citizens to enter.
Oh, and for the record, I'm in Japan right now (for the first time) and their CBP was far more serious than anywhere else I've been and took longer to process everyone than anywhere I've been except Dulles (fuck that airport). I was finger printed and photographed (yes, yes, I know the US does this for non-citizens too) and this is highly atypical.