Living in a society is sometimes a short term negative to a particular individual. If everyone solely optimized for their own situation, society would fail.

Why should I have to be OK to have a sidewalk on my property when it lowers my property value? Why should I have to agree to have utilities be buried in my yard if it might lead to a loss of use if said utilities need to be repaired? Why should I need to be OK with housing that will give people a place to live if it means I have to endure more traffic?