Most of the time, you can control where XDG puts its litter, cf. https://wiki.archlinux.org/title/XDG_user_directories
Just note that XDG_DESKTOP_DIR and XDG_DOWNLOAD_DIR can not point to the same directory or chromium will disregard your config.
P.S. Reader, if you can commit to chromium without much hassle, check this and fix: https://wiki.archlinux.org/title/Talk:XDG_user_directories