Have you considered doing something like this:

  cargo build --target-dir ~/build-artifacts/$(basename $(pwd))

Then you can have something like this in your ~/.config/user-tmpfiles.d/clean_build.conf

  d /home/user/build-artifacts - - - 1d -

You can then do cleanup by calling

  systemd-tmpfiles --user

Or you can use the tmpfiles timer

  systemctl enable --user --now systemd-tmpfiles-clean.timer