The lack of a physical SIM does not give more safety. "SIM Swap" means "convincing a system or human to transfer a phone number." A GV number is just as easy to transfer as any other phone number.
Porting attacks are definitely possible against Google Voice, but these require confirming the port in the target account first, no?
And the Google Voice equivalent to a SIM swap would just be a compromise of the Google account itself. Definitely not impossible, and I know I’m tying my availability to a company not exactly known for being the best custodian for that – but I’ll take my chances with them over any phone provider.
There are "fingerprint" cookie marketplaces that sell tokens from malware-compromised computers and allow you to make HTTP requests from a victim's connection, this could be one approach. There are also scammer call centers that will call unsuspecting people pretending to be Google, Coinbase, AT&T, or whomever, and have them click buttons in user interfaces.
I've seen entire Google accounts deleted with no recourse due to this "suspicious activity" that victims had no control over. Computer says no, and it's near-impossible to get in touch with a human at Google.
(I agree with you on terminology but media reports tend to group number porting attacks in with "SIM swaps")
There is nobody to social engineer (it's Google, they hate customer service) and the system rejects all port-out requests until you unlock the number by paying a few dollars which requires breaking into the Google Account to begin with. It is absolutely not the same as compromising an employee of a carrier.
To be clear I'm describing Google Voice which is purely a VOIP service, not Google Fi which is a MVNO.