Rotten Tomatoes' system has a lot of positives and I see a ton of signal in Rotten Tomatoes scores. It only works with a critical mass of people rating a given piece and I don't think anyone could get that critical mass for books aside from Goodreads.
I had an idea of a rating system where people would have to create a ranked list of the movies/books. Their rating for a given piece would be based on where it is in their list, linearly. Then ratings would be relative to other content. I think this would be much harder to get off the ground than a Rotten Tomatoes system.