Chrome and a number of other software insists on creating some of these directories (e.g. Downloads) too.
However you can fix that by setting XDG_DESKTOP_DIR and XDG_DOWNLOAD_DIR in ${XDG_CONFIG_HOME}/user-dirs.dir to something else (e.g. your home directory).
% cat ~/.config/user-dirs.dirs
XDG_DESKTOP_DIR="$HOME"
XDG_DOWNLOAD_DIR="$HOME"
% ls -ld ~/Desktop
drwxr-xr-x 2 mt mt 4096 Oct 22 18:04 /home/mt/Desktop/
I have no idea what keeps creating these. It's pretty annoying as I have a lot of intentionally short-named things in my ~ for easy shell usage, and it really sticks out.
inotify doesn't support reporting the PID of the process that triggered the event, but it seems fanotify does. Maybe I should set up a watcher for this.
systemctl enable --now xdg-user-dirs-update.service
(fish in this example but would be similar for other shells)
sed -nE 's/^([^=#]+)=(.*)/set -gx \1 \2/gp' <"$XDG_CONFIG_HOME/user-dirs.dirs" | source