The committee for the literature prize has contained some prestigious literary figures themselves, so it is to a slight degree a judgement of peers. Secondly, the membership of the committee has historically represented a high society that many authors have been members of themselves, or who hoped to have that high society among their audience. Go through the Nobel ceremony archives and you can see some writers getting quite emotional that they are there among the committee and the invited guests.