Recently listened is an automatic list like a log, not user curated. I would never expect to be able to delete things from that list.
I don't need an audit log for my audio. What I do need however is to remove something so it doesn't influence the algorithm. I may not remember every time I play music for a kid to turn on the right mode.
You're never going to win against a black box algorithm. Just let Spotify do it's thing or go back to hosting your own music if you don't like it.
That would also then require them to invalidate caches and rerun scripts to rebuild those lists and recommendations, which costs extra compute.
Invalidate the recommendation cache
You mean like, when a new song is added to the list???