That's true in many ways. The actual site's UX isn't bad per se because it's pretty easy to find what you want, buy it, manage the order, etc. And yes, that's what "it's all about" at the end of the day.

I think I was caught in a little bit of Silicon Valley myopic thinking since their site isn't shiny and that's what is mistakenly considered as interchangeable with good UI design these days. That being said, Amazon.com the site does still need to improve UX in a couple key areas. Compared to more contemporary e-commerce sites, the ability to compare items and make sense of ratings (i.e. the flaws of the 5 star rating system) is where they need to make the most improvement IMHO.

It may have been an overstatement to say "their site leaves much to be desired" in all fairness.