This seems like it would depend heavily on your threat model, especially keeping in mind that we're talking about second factors here.
For example, one threat is that bad guys gain access to the authentication data of the Relying Party. For example, maybe they find the daily backups are in backups.tgz on the web server for convenient downloading. Or maybe you never changed the password on the MySQL server. This is of course one way bad guys might have everybody's passwords, the first factor...
For TOTP the stored credentials include a "seed" value used to generate those six digit codes, and so by the relying party to confirm your code is correct. So for that credential access threat, the bad guys also have your TOTP codes and you're no better off with TOTP.
Whereas for SMS the stored credentials just include a phone number to send the one use codes to, bad guys having that isn't great news necessarily, but it doesn't actually give them the codes. Even if the one-use codes are stored in the same place as permanent credentials (which they may not be) and thus accessible to bad guys, the bad guys can't necessarily arrange to see them before you use them, and in any case can't arrange for you not to wonder why you're getting all these one-use SMS codes suddenly.
In contrast notably Security Keys don't end up with the Relying Party having any secrets at all, and so bad guys do not learn how to impersonate your users even if they somehow have access to the same means you use to authenticate those users.