I am really an amateur when it comes to Spectre-like attacks, but do you strictly need a valid pointer pointing to the address? I thought you "just" need to mispredict into code that would use it as a pointer, even if that code is never actually reached?

The paper demonstrates this by the C PoC using a system call as a gadget. Any value can be passed into the system call before it gets checked for validity on the other side of the kernel boundary. In their example, they use the "buffer" and "buflen" arguments to the keyctl system call, which results the values passed into the system call being in the registers r12 and r13. Then, they mispredict into a disclosure gadget that uses r12 and r13 for dereferencing pointers:

    movzx edx, byte ptr [r12]
    mov rbx, qword ptr [r13 + rdx*8]

Note how "buflen" isn't even a pointer (for keyctl) to begin with, but the (as far as I understand) unrelated disclosure gadget code dereferences r13 (because it treats it as a pointer), and they managed to mispredict into it through keyctl's call to the "read" function pointer (this is the part where it's still a bit fuzzy to me, as I unfortunately don't fully understand the misprediction itself and how they control for arbitrary destinations).

Now, obviously you can't directly make system calls through JS. But I don't understand yet what, if anything, is in place to absolutely make sure that there are no indirect ways that result in a system call (or another path!) where benign, but arbitrary values get passed as arguments in registers, executing benign code, but being mispredicted into a different kernel code path where those registers would be used as pointers.

And then, once you can do that, you can affect almost arbitrary physical memory, since typically almost all physical memory is mapped in the kernel's address space.

Sure, this is much harder because of the layers in between, but I still don't quite understand why it's impossible, and why a sufficiently motivated attacker might not eventually find a workable solution?

Spectre just seems so fundamentally catastrophic for me, that anything but proper hardware fixes to how privilege boundaries are honored by speculative execution seems to merely make things harder to me, but how hard is a very non-trivial question for me. Is it hard enough?

(As for ASLR, in their paper they break that as their first step using their own methods.)