As you say, the independent stores found a sweet spot. Compare the success of independent bookshops to the big chains like Barnes & Noble. The former are inherently browsable; you know you'll find something interesting, even if you have no idea what it'll be. And they draw you in with events like book-signings.

Mostly, I think the big retail industry stopped innovating the (cough) shopping "experience." They put more attention on helping people find things they are actively looking for ("I need a blue left-hand widget") than encouraging serendipity.

Both things can exist in parallel. I find myself comparing in-person retail shopping to sports. You absolutely can get a more comfortable sportball experience sitting on your couch, with cheaper beer and immediate analysis. But in baseball, for instance, the teams have done a lot to encourage people to attend games... not because one is better than the other, but because they are interestingly different.