The idea that only 4+ star rating books are worth reading is laughable. Reading only very popular books is a recipe to read only books that fit into your own preconceived worldview.
On a related note, for binary (good/bad) reviews, we can the temper the results by assuming a beta(1,1)-distributed prior and then updating [1]. The expectation (which can thought of as an "adjusted average rating") ends up being:
E[x] = (good + 1)/(good + bad + 2)
This adjusts for situations where the number of reviews is small, in which a good average rating can be misleading.
[1] https://www.johndcook.com/blog/2011/09/27/bayesian-amazon/
"If Newton's Principia were published today, it would have 4 stars on Amazon. There would be one cluster of 5 star reviews by people saying it had revolutionized their thinking, and another cluster of 1 star reviews by people complaining it was pointless and hard to read."
and then someone linking to actual reviews on Amazon (avg. rating 4.1).
There are so many books available to me that I do use Goodreads ratings as one heuristic for choosing what to read. It would take a lot for me to read a 2 star book (like a very strong recommendation from someone I trust), not because I am sure it wouldn't be worth my time, but because there are plenty of more highly rated books that are more likely to be worth it.
The author even qualifies the rule: "Unfortunately, Goodreads ratings are often not unbiased.."
... under the assumption that everybody who rated the book shares your preconceived worldview. Assume that not to be the case on Amazon.com (or similarly large review platforms).