My conclusion is that Yahoo's spam filters just suck in general.
Have you checked out CFL? If users mark sender’s messages as spam, it can impact that sender’s deliverability. The CFL can help avoid these recipients by understanding spam reports.
More best practices for deliverability: https://senders.yahooinc.com/best-practices/
Since a while now, yahoo are delaying emails for 12 h to 48 h. I have 0 issue anywhere else than yahoo.
Not a single email sent by this server has been flagged has spam according to their own support staff.
I have no idea what to do to fix those delays and it seems they have no idea either...
I'm trying to get their support to help me. I'll update here if I have any useful answer from them.