Try to set enabled=False in ~/.config/user-dirs.conf

https://www.unix.com/man-page/centos/5/user-dirs.conf/