Okay and you would say that if you optimised all of these factors you would end up with a latency around 300ms?

I just set my Ableton Live to 300ms and it was actually o.k. I think the reason is that a lot of people don't actually 'play' their instruments these days - at least in electronic music.

Instead, they program their drums by putting midi notes on the grid and then listening to the result. The same with synths etc. So when I work this way, the 300ms latency are actually bearable. Of course it would be different if I used drum pads to play my drums 'live'. But honestly I don't know many people who do that and when I watch tutorials on YouTube also almost no one is doing that. A lot of electronic music producers 'play' their instruments with their mouse button.