I would have to go dig up the source for this, but I believe that genetics play a role on how your body handles dietary cholestrol. For many it's not a problem, but for some it is.
So for some individuals with a super clean, low cholesterol diet, adding dietary cholesterol would significantly impact their serum cholesterol. But for many (arguably most), it won't make much of a difference, which is one of the reasons DC has moved down in importance in dietary guidelines.