I knew someone growing up who simply could not eat venison for this reason. For him, it was a reminder of poverty; it was the only meat that his dad could get, and they ate it every day because they could barely afford mac and cheese.
I found this really interesting, as I always thought of venison as one of those foods you pretty much only got at a really high-end restaurant or from a (rare) friend who hunted. Different culture.