Many people do hold that opinion of fine dining restaurants, yes, but normal people do go to them is only because there aren't enough snobs to sustain them.
I am not sure snobs is the right word. It is just a different experience.
I like opera and I like metal but while both are music, it is a completely different live musical experience.