But that's exactly it - if any of the filenames have spaces, that will fail, since xargs will try to delete each part separately. From the man page:

Because Unix filenames can contain blanks and newlines, this default behaviour is often problematic; filenames containing blanks and/or newlines are incorrectly processed by xargs.