Unfortunately, that is incorrect understanding. In the context of stuff like this, "untrusted code" can be so many different things, it isn't just being able to run an executable... The issue with these vulnerabilities is the processor "pre-runs/prefetches" certain things for performance, which would likely be run in the future, so that it has those results cached, or to increase the amount of instructions in a CPU cycle. The issue is that many of these things aren't direct executable code, but more like access patterns.
For example, a speculative execution vulnerability might exist when there is a string of data in memory that has a memcp pattern, with memory addresses that are valid. To speed up the execution of programs, the processor might proactively read the memory from that memcp pattern because it is sending a command to the memory controller already.
Many of the mitigation techniques are to flush the caches/buffers and be careful about memory reads, which are the precise things that actually take the performance hits. And one of the problems is since they are access patterns, the vulnerabilities can be the html renderer compiled with an application, an SVG rendering library or a mail client.
For example, lets say there are mitigations built into the JS library... if you build a specifically crafted SVG, you could create similar access patterns to get around it. Its a huge cat and mouse game if there aren't OS protections and it only takes a single app to lose your secrets stored in memory.