They might have significant others depending on UBI - parents, children, friends. So they would not like cutting basic income, even if it recovers slightly more money for them.

Compare this to situation with poor: Most of middle class don'r really have poor significant others in their life, making cutting benefit programs easier.