Anyone who says that is, frankly, unobservant.
We have government bail outs for people who live in areas that frequently flood due to things that have nothing to do with climate change, such as normal rain and hurricanes. Many of these areas flood almost predictably, yet we keep providing government support to keep people living there, or to help them move to someplace that floods less.
There is no way that something similar won't happen to people when climate change induced sea level rises starts making some coastal property unlivable.
Also, a good fraction of the population live in areas where there is a good chance that something will destroy their house over the next 30 or 40 years. Besides the aforementioned floods, there are earthquakes, wild fires, and tornadoes.
The only real difference between those and gradual sea level rise destroying your home is that with sea level rise afterwards you have to go live somewhere else instead of rebuilding where the old house was. No one claims that people who live in states that can have earthquakes or tornadoes do not really believe that earthquakes or tornadoes happen.