It would be useful to understand the flow of an SMS from a source to a Google voice number. While you can't port a Google voice number, it seems like if you can intercept an SMS from a source before it gets to Google then this technique will work.

A useful strategy to help against this in any case is to use a different email address for every online service. Hackers generally can't initiate an account password reset if they don't know the account.

Also if you use a different phone number for account security than your public one then it's a lot harder for them to know what SMS to intercept. Security by obscurity sucks but in this world it may be your only practical choice.