Also like sleep sort, something else -- some other real resource -- still has to bear the asymptotic cost of the algorithm. In this case, AIUI, you have to pay in "light distance traveled", which goes up exponentially with the problem size.
There is the Computational Complexity-Theoretic Church–Turing Thesis, which hypothesis that any physical model of computation can be efficiently (eg. in polynomial time) be simulated by a turing machine. This has never been proven, but it has led to correct physical predictions so far.
We also have have demonstrated polynomial improvement that can be had by moving away of a turing machine. For example, the use of random access memory offers an asymptotic improvement for many algorithms relative to the linear access memory used by a turing machine. Additionally, Grover's algorithm allows an exponential speed up in a wide range of problems on a quantom computer; and neither of these approaches requires shifting the savingns onto some other resource.
(It's true that you can pack the memory into a 3D volume and thus have a cubic improvement in access time, but you can do the same thing with a multi-dimensional TM.)
RAM machines are an abstraction, there's no such thing as constant-time access to a random address in any physical store. At best you're looking at O(n^0.5) where n is the number of bits in the store. Still a speedup, but not nearly as much as predicted by using a RAM machine.
Grover's algorithm doesn't offer exponential speedup over classical algorithms, its complexity is O(n^0.5) queries, where n the size of the list being searched, versus O(n) queries classically.
You should be able to get memory down to O(n^1/3) if you move into 3D.
For Grover's algorithm, it depends on how you paramiterize the problem. For example, consider brute forcing a key. We generally parameterize this problem by the size in bits of the key, in which case we are looking at O(2^n) queries. Grovers algorithm lets us reduce this to O(2^.5n) querires, which is an exponential speedup.