TOTP is almost purpose made to be phished. There are just six digits, most people are getting them from a separate device, and you can type them into the box on phishing-site.example just as easily as real-site.example, there's no indication this might not be a good idea, and hey, phishing-site.example can even give you a "Thumbs up" indication that you got the digits right, hooray.
All the bad guys need is one of dozens of ready-made tools to eat the input on phishing-site.example and feed it into real-site.example, stall the duped user and give them access. So this is something you "don't have to worry" about only in the same sense as duplicating mag stripe cards, or somebody cutting off your bike lock, or a dozen other things we know crooks do all the time with low effort and little risk of being caught.