This breaks the standard GUI interaction model. Buttons are typically triggered on mouse-up events, not mouse-down, and there's probably good reasons for that.
Any client-side hacks that attempt to improve the perceived page speed are somehow bound to break something. If there really is a simple way, it's already been implemented at the browser level, for all websites. Link prefetching is already a thing. Do you want fast pages? Make your server-side rendering fast.