To speak sensibly about what is good, we have to define what we mean by 'good' in this context. You can define 'good' as 'popular' if you like, but I don't think that corresponds with how most people use the word, and nor is it particularly useful. (We already have a word for popular.)
This doesn't mean that 'good' = 'lauded by highbrow critics', either. But it's okay to have a concept of literary quality that isn't reducible to popularity or pure entertainment value; it's okay to value things like psychological insight, or beautiful complexity, or the capacity of a book to broaden its readers' circle of empathy.
We will never all agree on which qualities matter most or which books best exemplify them. Employing these unavoidably fuzzy and subjective concepts certainly opens the door for pretentious charlatans to barge in and do their thing. We should adopt a sceptical stance toward any and all cultural gatekeepers, and exercise our own judgment. But there's still no need to flatten our concept of literary quality down into something objectively measurable.