You start by allowing people to rate the ratings. For instance, flag individual titles or tags 'not interested' such as on Steam. Even just a way to hide individual titles would make a system like Netflix much better to use, and bring product customers are more likely to pay for to the foreground. And then you can feed the data to the recommendation engines, which might start to learn about what demographics are using the system rather than relying on assumptions. My personal belief (as someone with zero actual experience here), is that dislikes and disinterest would be much better for generating recommendations over likes and interest. 'likes' just gives you what is popular in your familiar genres. 'dislikes' expresses your tastes.
If you allow people to rate the ratings, you come back to the original issue that people are unreliable raters.
Dislikes over likes does sound interesting though.
Yes - it's a difficult problem. It must be possible to build a better statistical model of each rater, in order to weight their opinions. A first step would be to normalise the rating distribution of each person (e.g. by average and standard deviation). I wouldn't use a rating for a particular book, unless the rater had some minimum number of ratings, or a book had very few ratings.