Then you'd return 202 on a request to delete any invalid element, which really doesn't sound right.
> 200 with some sort of response to distinguish (e.g. { changed: boolean }).
Sounds even worse, and much like APIs that returns 200 with a payload saying {error:"not found"}
It depends on your use-case if you care about this or not. If you do care you will have to handle it either way somehow.
But if you want an idempotent API then a success is more appropriate IMO than an error.
The key is the perspective of what success is: it’s the state where the resource does not exist, or is the action having deleted the resource. But 200/202 is perfectly capable of expressing the distinction in the latter case as well.