The problem is actually cholesterol produced by the body using other substances. 80% or more of the cholesterol in your body is produced by your body, no more than 20% of it comes from your diet unless your diet is particularly unbalanced in that manner. The reason cholesterol in people's diet got the blame is that there is often a strong correlation between the foods containing substances that do cause increased cholesterol in the body and foods containing significant amounts of cholesterol, and a similar correlation between certain lifestyle options that are risk factors for heart problems generally and eating those foods.
So blindly reducing cholesterol in your diet is not useful and may lead you into a false sense of security - you could still be taking in the other stuff and living your life in a way that will encourage the body to produce too much cholesterol itself.