I need to re-read the papers but I think the real problem isn't even speculative execution but allowing speculative cache changes.
The notion that "gadgets" didn't even need to return properly was both amusing and eye opening for me. It doesn't matter because the result will be flushed anyway! ;-)